As frontier large language models (LLMs) become more capable, their running cost rises sharply. Small language models (SLMs) meanwhile gain capabilities while keeping inference cost stable, offering 5‑40× cheaper processing for the narrow, repetitive subtasks that agents decompose work into. The remaining challenge is to obtain correct behavior from a small model quickly, efficiently, and economically. In most cases zero‑shot SLMs fail narrow production tasks, and retraining each time the behavior changes is too slow for iteration.
We propose a two‑step workflow: first develop the desired behavior in prompt space on a frontier model, using live traffic for immediate feedback; once the behavior converges, distill the recorded production behavior (not synthetic imitations) into an SLM via supervised fine‑tuning (SFT). We demonstrate this with a scheduling agent for home‑care agencies. The teacher model, GPT‑5.4‑nano, generates a yes/no safety decision for each caregiver‑shift pair. A LoRA‑tuned Qwen3‑8B student, trained on 11.6k recorded production decisions, achieves 0.913 agreement with the teacher’s recorded prompts, while the teacher re‑run on its own prompts agrees with itself only 72.7% of the time, indicating the SLM can be more consistent. When the two disagree, a blinded professional scheduler sided with the student 48‑29. The SLM also shows roughly 2.5× lower median latency and a fraction of the cost.
We believe this lifecycle of rapid building, live feedback, and cheap inference provides a blueprint for others building similar systems.
Review: The paper presents a practical pipeline that leverages a powerful LLM for fast prompt iteration and then distills stable behavior into a cheaper SLM, achieving higher consistency, lower latency, and significant cost savings, offering a valuable template for real‑world deployments.