This looks really interesting, and if I understand what this does properly: it was high time someone built this!
Without diving into an experiment myself, it would be amazing if you could add some stats or experiment logs, if it's not too much problem and you have them.
For example:
Given model XYZ, every assuming 5 donors with a uniform 32GB each, each forward pass shunts xGB over the link. Each pass takes nMS, etc. etc. Resulting in n T/s, assuming latency of n ms.
Do you have such stats? Or perhaps I missed them in the repo?
I am sorry I did not understand all of it. But, would this allow running large MoE LLMs on a local network with experts spread out over multiple cheaper GPUs (or even CPUs)? This would perhaps be more useful than over the Internet, within offices for example.
That’s one of the strongest use-cases.
On a local network (office, lab, home cluster) the RTT is a few milliseconds instead of 20-50 ms, so the expert-offloading becomes much more practical. You can spread the experts across several cheaper GPUs or even CPUs, keep only the dense parts + router on the machine you’re chatting from, and the whole thing stays private inside your LAN.
No internet required, no cloud, just the machines you already have.
Because everything stays local and latency is tiny, even modest always-on devices can contribute. A few Raspberry Pi 5s, old mini-PCs, or stronger IoT-style boards can each hold and run a handful of experts. The protocol doesn’t care if the peer is a big GPU or a small ARM box, as long as it can load the expert weights and do the matmul.
Pure busybox-class sensors are usually too limited in RAM and compute for current MoE experts, but the broader “every half-decent always-on box in the house joins the swarm” vision works well and keeps everything private inside your LAN.
P2P inference sounds really nice and it would bring us back to the 2000s culture if not for the fact inference is already so cheap. Even so it is an insurance policy if the cloud providers or governments have ideas of further censoring and monitoring chats.
Ho thanks for the comment.
Verification does not depend on temperature.
Expert execution is deterministic (pure matmul).
LUMABRI_VERIFY=N re-runs N% of the calls on a second replica and requires byte-identical output.
Temperature (and sampling) happens only on the chatter, after the experts return their activations.
So it can be any value (0, 0.7, 1.2…) without affecting the verification contract.
Without diving into an experiment myself, it would be amazing if you could add some stats or experiment logs, if it's not too much problem and you have them.
For example:
Given model XYZ, every assuming 5 donors with a uniform 32GB each, each forward pass shunts xGB over the link. Each pass takes nMS, etc. etc. Resulting in n T/s, assuming latency of n ms.
Do you have such stats? Or perhaps I missed them in the repo?
All the iot devices contributing to matmul but within a LAN?
Isn't that only true in theory but wrong in practice due to floating points?
J/k, this looks cool :)