NeFut Logo NeFut
中 Admin Login

[CS.AI] kw-bench: Probing Frontier Model Harnesses on Real-World Tasks

Published at: 2026-10-28 22:00 Last updated: 2026-10-29 00:58
#AI #LLM #DeepSeek

Each new generation of frontier models pushes the edge of what is possible. When placed inside a harness—a loop, a shell, a filesystem that lets the model write and run its own code—the model can tackle increasingly difficult work. Recent research treats the coding‑agent harness as infrastructure for knowledge work rather than a mere coding tool, and a wave of benchmarks now evaluates it. Most benchmarks are generative: given a brief, the model builds a financial model, a memo, or a spreadsheet, and the output is judged by a rubric or another model. The costly part of knowledge work is often investigative—answers already exist hidden in incomplete, inconsistent evidence spread across systems, and must be recovered and defended to the cent in audit, tax, due‑diligence, regulatory reporting, claims, or numeric disputes.

We introduce kw-bench, a benchmark built from real engagements described by specialists. Its first task family reconstructs five years of VAT across three generations of SAP—legacy ECC, a partial S/4HANA migration, and a consultant’s redesign. The data are raw system extracts: page headers, local number formats, partial link tables, with no documentation of how entities connect. Grading is deterministic and does not involve an LLM judge.

We ran four harnesses—Claude Code, Codex, OpenCode, and micro‑cc—on the same model, DeepSeek v4.1 Flash, for two hours. The two model‑agnostic harnesses (OpenCode, micro‑cc) scored 0.542 and 0.528 respectively; the two lab‑specific harnesses (Claude Code, Codex) delivered zero when detached from their home model. The failure source was inertia—the inability to keep many weak signals alive over a long session.

On a true frontier model the task collapses. micro‑cc and Claude Code on Claude Opus 5.5 achieved 0.9986 and 1.000 in 22 and 25 minutes. A weeks‑long engagement now costs minutes and a few dollars; below the frontier, the harness decides how far a model gets.

Review: kw-bench exposes the tight coupling between model capability and harness design on real‑world investigative tasks, showing that with a frontier model the speed and precision of VAT reconstruction rival human auditors, while traditional harnesses remain the primary bottleneck.

Original Source: https://arxiv.org/abs/2610.36405

[h] Back to Home