NeFut Logo NeFut
中 Admin Login

[CS.AI] Cascadia: Resident 975B MoE Inference on Eleven AI PCs

Published at: 2026-10-07 22:00 Last updated: 2026-10-08 01:25
#AI #Machine Learning #LLM

Mixture‑of‑experts (MoE) models make nearly trillion‑parameter capacity accessible with sparse per‑token computation, assuming the serving system can distribute weights and coordinate execution. We resident‑run the Inkling model—975 B total, 41 B active parameters—on eleven Intel Core Ultra X7 358H AI PCs, each equipped with 64 GB RAM, Arc B390 iGPU and gigabit Ethernet.\ \ We built a custom resident MoE engine that preserves Inkling's routing rules, constructs compressed graphs for OpenVINO fused iGPU primitives, and restores FP32 output after FP16 expert computation. The engine fits six consecutive decoder layers per machine, representing dense feed‑forward blocks as all‑active expert slices, cutting dense‑layer call time from roughly 8.1 ms to 4.5 ms.\ \ A streaming pipeline coordinates concurrent generation, while captured‑state draft evaluation measures agreement with the deployed numerical path. Paired measurements across fifteen concurrency levels from 1 to 176 streams show an aggregate decode throughput of 60.29 tokens/s at 88 streams and 46.87 tokens/s over the full serving phases. At fifteen streams the median first‑token latency is 6.05 s.\ \ Increasing the context budget from the default 1 024 positions up to 64 k tokens, real prompts of 1 k‑64 k recover the embedded code in all 19 measured answers. First‑token time grows as $aN+bN^2$, decode latency grows approximately linearly; both are bounded by a single‑threaded CPU attention loop rather than memory, which holds 512 k positions per stream.\ \ Evaluation on captured fleet states separates the effects of vocabulary selection and weight quantization on draft agreement.\ \ Together these contributions establish an execution and evaluation approach for large sparse models on distributed client systems with shared CPU‑GPU memory.\ \ Review

Original Source: https://arxiv.org/abs/2610.07219

[h] Back to Home