On‑premise assistants can provide factory workers with conversational access to machine documentation, yet models capable of this task often cannot fit shop‑floor hardware. We first apply structural compression and then adapt the model with retrieval‑grounded fine‑tuning. Experiments show that after compression the general capability drops almost linearly with parameter count, while the quality of retrieval‑augmented answers remains largely independent of model size. Consequently, we treat deployment as a post‑adaptation sub‑network selection problem: under a configurable general‑capability floor and memory budget, we assign each device a single sub‑network based on measured on‑device throughput and human‑judged answer quality. Rules that optimize size, speed, or quality alone each sacrifice capability or throughput. A weight‑shared supernetwork trained with sandwich‑style in‑place distillation keeps the selection inexpensive.
In a manufacturing‑manual case study, structural compression reduces judged quality to 13.7% of the unpruned model’s score. Retrieval‑grounded distillation restores quality to within 4.6% of the original, recovering roughly two‑thirds of the loss. The same assistant runs across three heterogeneous edge tiers with standby power ranging from 1.3 W to 5 W.
Review