Mixture-of-experts (MoE) inference on consumer hardware is limited by weight memory: a 35 B model quantized to 4 bit occupies about 19.5 GB, and sparsity only reduces compute per token without shrinking the required bytes. Naively offloading weights to SSD does not help because the experts for layer N+1 must be selected before the output of layer N is available, so reads cannot be started early enough to hide behind computation.
Edge0 is a streaming MoE inference engine that closes this gap with a prerouter. For each layer a head predicts the routing of the next layer one token ahead. The prediction is consumed as the actual routing, so the staged expert set exactly matches the routed set and no expert is dropped.
To recover the quality loss caused by int4 quantization and routing replacement, Edge0 uses an unmerged recovery LoRA trained on the student path, restoring performance close to the fp16 teacher model.
On a single machine with 24 GB of RAM, Edge0 serves a 35 B MoE at roughly 20 tokens per second while keeping peak active memory at only 3 GiB. Across five public benchmarks the overall performance is within a few points of the fp16 teacher. The same framework also runs an 8 B tier, and all code, checkpoints, and adapters are open source.
Review: Edge0 demonstrates that pre‑routing prediction combined with LoRA recovery can run very large MoE models with minimal memory, offering a practical path for SSD‑assisted inference on consumer devices.