Another great way to understand how vllm works is to read the code of nano-vllm[1]. It's basically "vllm but cut down to size. It's ~5kloc, supports just one model, disposes of some of the abstraction layers that vllm needs due to its codebase size, but contains all the major pieces that make an inference engine fast.
vLLM is originally marketed as paged attention, but in hindsight, separating the web server and GPU process, continuous batching, kv caching / chunking, and a huge model library including low precision mattered more.
I wonder how much it would cost to vibe code the whole thing from scatch?
I wonder how much better models need to get before such a thing wouldn't look like code vomit?
I've been working on a fresh, AI assisted port of DiffusionGemma from scratch and it takes a significant amount of time to deslop. I've spend a nonzero amount of time on refactoring and comment-vomit cleanup.
Any plans to support smaller models? I have a M4 Mac Mini with 16GB unified memory and an RTX 3060 (Laptop) with 6GB VRAM. My own product experiments all revolve around small models and harness around them. Happy to contribute.
[1] https://github.com/GeeeekExplorer/nano-vllm
[0] https://github.com/jmaczan/tiny-vllm
[1] https://sgl-project-sglang-93.mintlify.app/concepts/radix-at...
I wonder how much it would cost to vibe code the whole thing from scatch?
I wonder how much better models need to get before such a thing wouldn't look like code vomit?
https://old.reddit.com/r/LocalLLaMA/comments/1vh9lx4/i_porte...
https://github.com/mmastrac/diffgemma