The most accurate task‑oriented agents are usually powered by large cloud‑hosted models that read users' private data to solve problems. Small models that can run on edge devices keep data local but suffer a severe drop in performance. A third alternative is to let a cloud model write a program that runs on the device; the data stays local, yet because the program is written before any data is seen, it fails whenever a step requires interpreting content that only appears at runtime.
We propose an intermediate architecture: the cloud model first generates a program skeleton, and whenever a bounded question about user data arises, it delegates the query to a small local model (SLM) via a normal function call. We evaluate this design on the AppWorld benchmark across six combinations of backbone model and test partition. The results show that deferring such queries to the local model improves the success rate of a code‑only agent by up to 14.9 percentage points, recovering a substantial portion of the gap to a fully cloud‑hosted ReAct loop while keeping user data on‑device.
Key benefits of the approach include:
- Data remains on the device, reducing privacy concerns;
- The local model provides on‑the‑fly reasoning to cover the blind spots of the pre‑written program;
- Overall success rates increase without a large computational overhead.
Our experiments demonstrate that using local models as interpreters can achieve a practical trade‑off between performance and privacy in real‑world deployments. Review