Evaluating language agents on open‑ended, artifact‑producing tasks is notoriously hard: model‑based graders can be inconsistent, and exhaustive programmatic checks are often impractical. Textual rubrics make evaluation criteria explicit, but authoring them requires domain expertise, and rubrics derived solely from task descriptions tend to miss requirements and failure modes that only become apparent in actual solution attempts.
RubricCraft treats rubric construction as the generation of natural‑language specifications guided by candidate solutions. Given a task description and independently sampled solver trajectories, the system compares how each attempt satisfies or violates the task requirements and uses these differences to automatically create rubric criteria.
The generated criteria are then applied to the collected solutions, and the system revises checks that overlook errors or incorrectly penalize valid alternatives. This enables the capture of partial correctness without needing a single complete reference solution.
Experiments on SpreadsheetBench v2 show that RubricCraft rubrics pass fewer wrong outputs than a prompt‑only rubric written and applied by the same model (29.5% vs 42.6%), do not reject any outputs accepted by the benchmark’s own checker, and rank outputs by correctness more often while preserving model separation.
A repair study finds that fixing a single attempt leaves grading unchanged, but repairs driven by attempts from three model families affect different tasks and are judged by an evidence audit to improve grading in most cases, even on tasks where the reference solution is itself incorrect.
Review