Working on something similar targeting macOS on Apple Silicon, Unsloth split GGUF, compressed partial residency in unified memory (would make more sense on 128GB instead of my 64GB...), native Metal kernels, and RAM-only native compressed KV. Happy to put on GitHub when it's ready.
My main question is whether when put into practical use, this can be measured in tokens/second, or more like 1 token per minute... I have seen locally hosted LLM that are as slow as 1 tok/second still be very useful if you give it a project to do something overnight and metaphorically walk away from it, check back with what it has done in 6 or 8 hours.
0.05 to 0.1 tok/s on the other hand, as reported in the URL for the lowest class of hardware, isn't really usable for much.
edit: I think this is a fantastic project in general concept, and look forward to seeing more efforts towards the general idea of being able to run a 350B to 900B size model locally, even if as slow as 1 tok/s, on hardware that ordinary people can afford. Anything along the general concept of "we have fast read NVME SSD storage, we have a big ass model on local disk, we'll read it at 11GB/tok as we need it, not try to load the whole thing".
I’ve been wondering if chat is the wrong interface for slower local models (and some projects) and maybe something like a ticket system is a better fit. I just decided how I would test this idea on my available hardware before I go drop money on a Mac Studio or GPUs. I’ll probably have a POC this week. There is nothing novel here, just need to spend the time to get it working for me.
In the readme you can see benchmark which everyone with different hardware is running Colibrì, and I have to say I've seen great times! I'm always doing more to improve!
I have a 16-core system with 256GB RAM here I could try it with but regretfully it's so old the CPUs aren't AVX2 capable. Otherwise it makes a fairly good llama-server test system for CPU only stuff. Oh well. Time to upgrade (painful to the wallet these days).
If you get good at extracting remarkable performance from the most lesser of instruments enough to pull their own weight regardless, just imagine what it can be like when such a practitioner gets behind the keyboard of a world-class Steinway. And just does what they do best. Without ever having touched such a capable instrument themself.
On a level playing field the expression of virtuosity can outshine those who have never known any instrumental limitations at all :)
When pulling way more than your own weight happens like for few others.
There should be an award for getting the most out of the electronics rather than trying to reach orbit by building the tallest pile of e-waste.
First Prize right before your eyes !
Grande praise !
And just starting to ascend toward an unconquered summit that others find forbidding ;) Or they find uninteresting since the limit naturally lies on firm earth somewhere below the stratosphere.
For most projects the more practical solution is to use clouds offering GLM 5.2 for free. 1 token per minute is minuscule compared to their rate limits for free usage.
Maybe that's a measure of the self-fulfilling dollar incentive toward "renting" someone else's RAM in the future rather than trying to actually own such an outlandishly luxury item :\
Ideally this engineer's approach will yield better performance on lesser equipment in the future, if they keep up the good work after they get more-capable gear to experiment with as time goes by :)
I was actually just working on the same thing as this, but I went down the route of mmapping the entire model into memory to avoid the extra ram usage. I also had Claude implement Medusa[1] on the model to try and avoid loading an additional model into memory but still get the benefits of MTP. Currently at a stop light so I can't list everything and I didn't get to read your full post either yet.
To expand since I just got home, I'm making all of my modifications to llama.cpp, the goal was to eventually put this on a SBC of some kind with an nvme to handle the mmapped files. I think the theoretical limit of my current setup is about 1.8 tok/s based on prior testing but that is also with the additional medusa heads not fully trained (I honestly don't know if the counting it's generated tokens or not.)
In the end it seems like the idea we had is similar, I just don't know how to write an llm parser/runner from scratch yet and instead of specifying what needed to stay in memory I just let the linux kernel handle it.
Oh last note, I also capped llama.cpp usage to 16GB of my 32GB, so it might be possible to get it down even lower.
Yeah I'll see what I can transfer over from my llama.cpp work. As before I'm not too experienced with llm work, but I have a lot of experiments I'm trying out. So I'll make a PR if I get any interesting results.
I've been going smaller.. I have a custom-quantized Rust port of DiffusionGemma (26B) that seems to perform better (in responses) than benchmarks seemed to indicate and reasonably fast for its model size. Works really well on a 36GB mac as well for both prefill and generation.
It's been interesting learning about the balance of factors for performant metal kernels on unified memory.
Should have a repo up on github in the next few weeks.
The page has an SSD wear warning [0] I use desktop PCs that I build from components so I can replace the SSD, but what do users with soldered SSD do? Just avoid these applications or forge ahead disregarding the possible early burnout of their storage? They must use external storage as the burner SSD.
Even under AppleCare this is a $400 service which for an older macbook costs almost as much as the whole thing. And without Applecare it's not worth fixing at all.
Pretty cool! I've also been playing around with GLM 5.2 this week and was equally impressed. At work we're running it locally on some crazy expensive hardware as a test before starting another project so it's great to see people taking this massive FOSS model release and running it on an average machine, even if it's not terribly practical at this point.
Basically I kept needing an inference engine that could stream weights in and out as needed in an LRU manner. So I ended up vibe coding this thing that accepts a `--vram-budget` and stays under it (mostly). It turns out moving mmap'd bytes in and out of VRAM is way cheap compared to compute. Coupled with some pipelining/double-buffering, I almost always end up compute bound not memory bound. Granted I use way smaller models heh.
Wow, I see you managed to fit in so many models (krea, wan, hunyan, etc.). Did you get to build a common harness to run all of them? Which ones stay under your VRAM budget more consistently?
All stay under because I had Claude build the workflow to respect it (text encoding, denoising, vae, etc), there's just a tiny bit of untracked pieces. While there are common interfaces to invoke them (CLI and API/webpage) and they share ops and some pieces, lots of model logic is unique. This is all vibe coded and surely has inaccuracies.
I just learned about Gemma4.pas at the beginning of this week. Now this. This make me wonder how can inference engines could be built that easy. I'm not knowledgeable in this, but I thought it would take very deep Mathematic and system level knowledge, ... and a lot of patience.
I'm not fully understanding this business of MoE so please forgive me if this is a dumb question, but would it be possible to use MPI with a small cluster to distribute the load?
I am curious if it's possible to adjust this to use more RAM, as i've got a machine with 64GB RAM and 24GB VRAM. Or perhaps I could run Gemma/Qwen on the GPU and have GLM-5.2 delegate smaller tasks to it. It might take some retraining of GLM-5.2
I'm also curious if you can speed this up by using many disks in parallel to increase bandwidth.
>SSD Wear Warning
> Cold starts are heavy on random reads (~11 GB/token). Reads themselves are safe, but the OS page cache can generate writes. Heavy use may accelerate wear on cheaper SSDs. Use with caution and monitor your drive health.
Hmm, maybe a safe way to do this would be to make a separate partition for the model weights, and set them to read-only?
Not sure how the page cache works, if it's like per partition or per disk. If it's per disk, maybe you could have a read-only data.iso formatted as a partition and mount it as a disk?
I have a small laptop.
If you have more disks available, you could really do some testing.
When you have some benchmarks, submit a pull request or issue so we can maybe work on them.
We are really happy for contribute!
I have epyc 9654 ES and a 7900 XTX. I was running the numbers, and even if I maxxed out the ram to like 12x32 gig sticks, it would cost me thousands more and I could only run GLM-5.2 at a couple tokens per second at q3. So this project is very promising because it suggests I could get pretty high speed and this CPU/motherboard combination suggests I have a lot of pci bandwidth that is unused.
I think another route might be looking at holding an even larger chunk of model weights in ram, and taking advantage of RAM<->GPU bandwidth, perhaps using a PCIe 5 GPU. This was my first thought since I have dedicated GPU.
If you are using Laptop, you're looking at shared memory between the iGPU and CPU. I've also tried that route, but I have always been skeptical of killing flash with too many reads, it essentially uses SSD like it's a consumable item.
I'm going to benchmark this right now with what I have and I'll get back to you on github.
> Is this a hallucination? What am I missing? Why would heavy reads generate writes?
I take it heavy reads means more stuff goes into RAM, meaning other stuff has to be cached?
I've got same question as GP: e.g. is there a way to set moderately fast consumer NVMe SSDs (I've got both a Samsung 990 Pro and a WD SN850X) in a complete read-only mode to prevent "wear"?
I think if you had something like a theoretical used/refurb 2U rackmount server with two older multi core CPUs, 768GB of RAM, you would see faster performance loading a Q6 or Q8 GGUF of GLM5.2 into a freshly-compiled latest copy of llama-server, with the "no-mmap" option turned on to intentionally load the whole thing into RAM at the time the llama-server daemon launches.
If you want a CPU-only machine with 512GB to 1024GB of RAM, despite extreme cost rises, there are still some great options out there from companies selling ex-lease stuff that's 3, 4, 5 years old. It'll be loud as hell under full CPU load when running inference, so if you plan to use it at home, put it in your garage or basement or laundry room or somewhere similar on the far end of a network cable.
The software that OP has published appears to be specifically designed to hold only the active parameters in RAM (<100GB) and read content off local NVME SSD as needed on the fly. All that NVME SSD read wouldn't be necessary if you can hold the model in RAM, even in the absence of any GPUs.
Another recent project that runs a huge model on a 48gb Mac is https://github.com/danveloper/flash-moe - it gets over 5 tokens/sec on an M3 Max compared to this projects very impressive 1 token/sec on an M5 Max. So for anyone wanting to tackle a Mac only version that targets lower spec machines this looks like a good candidate with plenty of room for speedups [edit: because it doesn't use the gpu].
Not hijacking anything as this project is amazing.
What causes problems is the rewriting in this case are only read while writing is the cache! However, I'm working to improve more and more and make some parts lighter!
With so many people implementing their own SSD streaming for specific combinations of model+hardware, maybe we should look into upstreaming to antirez/ds4 or llama.cpp...
0.05 to 0.1 tok/s on the other hand, as reported in the URL for the lowest class of hardware, isn't really usable for much.
edit: I think this is a fantastic project in general concept, and look forward to seeing more efforts towards the general idea of being able to run a 350B to 900B size model locally, even if as slow as 1 tok/s, on hardware that ordinary people can afford. Anything along the general concept of "we have fast read NVME SSD storage, we have a big ass model on local disk, we'll read it at 11GB/tok as we need it, not try to load the whole thing".
On a level playing field the expression of virtuosity can outshine those who have never known any instrumental limitations at all :)
When pulling way more than your own weight happens like for few others.
There should be an award for getting the most out of the electronics rather than trying to reach orbit by building the tallest pile of e-waste.
First Prize right before your eyes !
Grande praise !
And just starting to ascend toward an unconquered summit that others find forbidding ;) Or they find uninteresting since the limit naturally lies on firm earth somewhere below the stratosphere.
These days, can "ordinary people" afford 24GB of ram and half a TB of NVME ssd?
sigh
https://github.com/ztxz16/fastllm
To expand since I just got home, I'm making all of my modifications to llama.cpp, the goal was to eventually put this on a SBC of some kind with an nvme to handle the mmapped files. I think the theoretical limit of my current setup is about 1.8 tok/s based on prior testing but that is also with the additional medusa heads not fully trained (I honestly don't know if the counting it's generated tokens or not.)
In the end it seems like the idea we had is similar, I just don't know how to write an llm parser/runner from scratch yet and instead of specifying what needed to stay in memory I just let the linux kernel handle it.
Oh last note, I also capped llama.cpp usage to 16GB of my 32GB, so it might be possible to get it down even lower.
[1] https://arxiv.org/abs/2401.10774
Also in case your CPU is old enough, did you try disabling CPU bug mitigations?
I've been going smaller.. I have a custom-quantized Rust port of DiffusionGemma (26B) that seems to perform better (in responses) than benchmarks seemed to indicate and reasonably fast for its model size. Works really well on a 36GB mac as well for both prefill and generation.
It's been interesting learning about the balance of factors for performant metal kernels on unified memory.
Should have a repo up on github in the next few weeks.
[0] https://github.com/JustVugg/colibri#ssd-wear-warning
Laptops with soldered in SSDs should definitely monitor their usage and take care with this.
This project seems more of an experiment than something everyone should run, but pretty cool nonetheless
You don't need to be superstitious here: disk activity, including writes in particular, can be measured. E.g. `iostat` or `vmstat` on Linux.
Nice work!
Basically I kept needing an inference engine that could stream weights in and out as needed in an LRU manner. So I ended up vibe coding this thing that accepts a `--vram-budget` and stays under it (mostly). It turns out moving mmap'd bytes in and out of VRAM is way cheap compared to compute. Coupled with some pipelining/double-buffering, I almost always end up compute bound not memory bound. Granted I use way smaller models heh.
In theory MPI could distribute experts across nodes. In practice, for small clusters the added network latency usually hurts more than it helps.
Better suited for big clusters with fast interconnects. For now we're focusing on single-machine speed (caching, GPU hybrid, etc.).
I'm also curious if you can speed this up by using many disks in parallel to increase bandwidth.
>SSD Wear Warning
> Cold starts are heavy on random reads (~11 GB/token). Reads themselves are safe, but the OS page cache can generate writes. Heavy use may accelerate wear on cheaper SSDs. Use with caution and monitor your drive health.
Hmm, maybe a safe way to do this would be to make a separate partition for the model weights, and set them to read-only? Not sure how the page cache works, if it's like per partition or per disk. If it's per disk, maybe you could have a read-only data.iso formatted as a partition and mount it as a disk?
I think another route might be looking at holding an even larger chunk of model weights in ram, and taking advantage of RAM<->GPU bandwidth, perhaps using a PCIe 5 GPU. This was my first thought since I have dedicated GPU.
If you are using Laptop, you're looking at shared memory between the iGPU and CPU. I've also tried that route, but I have always been skeptical of killing flash with too many reads, it essentially uses SSD like it's a consumable item.
I'm going to benchmark this right now with what I have and I'll get back to you on github.
Is this a hallucination? What am I missing? Why would heavy reads generate writes?
I take it heavy reads means more stuff goes into RAM, meaning other stuff has to be cached?
I've got same question as GP: e.g. is there a way to set moderately fast consumer NVMe SSDs (I've got both a Samsung 990 Pro and a WD SN850X) in a complete read-only mode to prevent "wear"?
https://askubuntu.com/questions/103915/how-do-i-configure-sw...
If you want a CPU-only machine with 512GB to 1024GB of RAM, despite extreme cost rises, there are still some great options out there from companies selling ex-lease stuff that's 3, 4, 5 years old. It'll be loud as hell under full CPU load when running inference, so if you plan to use it at home, put it in your garage or basement or laundry room or somewhere similar on the far end of a network cable.
The software that OP has published appears to be specifically designed to hold only the active parameters in RAM (<100GB) and read content off local NVME SSD as needed on the fly. All that NVME SSD read wouldn't be necessary if you can hold the model in RAM, even in the absence of any GPUs.
Not hijacking anything as this project is amazing.
Amazing job!
https://github.com/JustVugg/colibri#ssd-wear-warning