The Offline AI Modules workstream aims to deliver practical, low‑power, community‑accessible voice‑first AI systems, targeting African language communities where speech dominates interaction and internet connectivity is unreliable. The workstream comprises three reinforcing components: a modular voice‑first offline architecture, a low‑cost hardware reference bill of materials, and a reproducible quantization and benchmarking pipeline for instruction‑tuned language models in the 2‑5B parameter range.
This paper presents the first end‑to‑end benchmark across two hardware tiers: NVIDIA Jetson Orin NX (TierB) and Raspberry Pi 5 (TierA). Three instruction‑tuned models are evaluated under four quantization formats, measuring decode throughput, chat latency, memory usage, power consumption, and multilingual quality—topic classification accuracy on MasakhaNEWS covering English, Hausa, Igbo, Nigerian Pidgin, and Yoruba, as well as per‑language perplexity drift. Speech recognition is assessed with Ethio‑ASR on Amharic and Oromo for both tiers.
The principal finding is that Q4_K_M quantization offers the best size‑to‑quality trade‑off: on TierB, gemma‑4‑E2B‑it achieves 28.8 t/s decode throughput and 89.2% topic classification accuracy at Q4_K_M, while on TierA all three models run within the 16 GB memory budget.
Review: The study demonstrates a viable path for deploying multilingual instruction‑tuned models on constrained hardware, providing a practical foundation for expanding offline voice AI in underserved regions.