LLM agents fuse large models with execution harnesses, so benchmark scores reflect their joint behavior. We argue that harnesses should be judged through their interaction with the underlying models. To this end we introduce five complementary measures:
- Valid Completion Rate (VCR), the likelihood that an agent finishes normally and yields an evaluable output regardless of task correctness;
- Model‑Harness Fit, how well a harness enables a given model to achieve tasks via valid completions;
- Cross‑Model Fit, the overall suitability of a harness across all evaluated models;
- Interaction Residuals, pair‑specific execution strengths and weaknesses hidden by component‑level summaries;
- Token Load, the token demand of agent execution, including failed attempts.
We evaluate four general‑purpose harnesses (OpenCode, Codex CLI, Pi CLI, DeepAgents) with four core models, plus two fixed native model‑harness systems, on six benchmarks covering context management, tool use, long‑context planning, multimodal interaction, and artifact handoff. Including native references and stress conditions yields 111 evaluation settings and 52,168 trajectories.
Findings show that harness rankings vary across models and benchmarks; task performance and valid completion may favor different pairings, and similar task scores can mask distinct execution profiles. OpenCode leads in cross‑model performance, while DeepSeek V4 Flash Preview maintains the highest mean VCR across the shared four‑benchmark panel. These results support evaluating model‑harness interactions, training transferable harness‑use capabilities in models, and designing harnesses adapted to specific models and tasks.
Review