Small vision-language models (VLMs) can read external evidence but often fail to obtain it. We introduce Harness Compilation (HC), an offline procedure that reshapes the division of labor between a frozen small VLM and an external harness. A large teacher model leverages student execution traces to revise reusable content and control, and a separate validation set selects the harness for deployment. Deployment requires no weight updates nor teacher calls.
Across seven visual question‑answering settings with students up to 9B parameters, HC improves scores by 9.9–23.9 points, averaged over three independent builds per setting. Interventions on five runtime decision types—invocation, selection, argument generation, evidence integration, and abstention—show that allocation matters: requesting evidence and generating open queries are costly, while bounded choices and reading supplied text remain useful student work. Fact cards benefit all ten evaluated students, though decision policies transfer unevenly. Recompilation helps when a transferred interface no longer fits a new student. With 100 practice items, HC exceeds answer‑only LoRA on three tasks. Larger training budgets can match or surpass a fixed harness, and combining both improves SlideVQA beyond either alone. These findings support allocating work based on measured student behavior rather than uniformly removing decisions.
Review