Abstract
Large language models (LLMs) often generate code that appears correct on typical inputs but fails on edge cases, invalid inputs, and other specification-defined corner conditions. A popular fix involves having the model write its own tests and repair until they pass, but the source of the gain remains unclear: does it come from the mere existence of tests, or from their grounding in a specification of what the code should do? We isolate this factor.
Holding the tester, test budget, and repair loop fixed, we modify a single prompt line controlling whether the tester receives the spec as a checklist of rules. The baseline performance is strong: it is already instructed to probe invalid inputs and edge cases. Grounding the tests in the spec produces correct code 38 percentage points more often than this baseline across three Claude tiers (Haiku 4.5, Sonnet 4.6, Opus 4.8), and 36 points on a held-out set. Grounding, not test quantity, is the primary driver: doubling the test budget barely helps, and combining eight independent ungrounded suites plateaus far below grounding. An ablation isolates the spec's content, not its format: given the spec as a plain paragraph, the tester recovers 27 of 30 bugs, but when asked to plan tests without the spec, it recovers only 2 of 30.
This effect persists across stronger baselines: a property-based generator catches 28 of 30 bugs but invents out-of-spec requirements, and an AlphaCodium-style loop only matches the baseline. It replicates across vendors (GPT-5.3-codex +28, Gemini 3.5 Flash +19), with a task-level sign test over 18 tasks significant at p=0.002. Grounding improves both sensitivity and precision: it catches more real bugs and wrongly rejects far less correct code, cutting the false-alarm rate from 33% (68% against a Python standard-library oracle) to 0%. On well-specified algorithmic problems, it neither helps nor hurts.
Blogger's Review: This research underscores the importance of specification grounding in code testing, revealing that merely increasing test quantity does not effectively enhance code quality. Instead, ensuring tests are aligned with clear functional specifications significantly improves the ability to capture real bugs while reducing false positives, providing developers with more reliable tools. This approach not only enhances code correctness but also offers crucial directions for future model optimization.