RTL repair benchmarks aim to measure whether a model can fix broken RTL when a specification is provided.
To this end we built RTLRepairDataset with 5,784 (broken, fixed) training pairs and RTLRepairBench with 459 held‑out cases, of which 374 are syntactic and 85 are behavioral mutations. An evaluation harness supporting simulation and formal equivalence is also provided.
Using this infrastructure we identified two measurement confounds. First, simulation and formal scores show opposite signs for the effect of providing a specification; reset‑aware rescoring attributes almost all of the disagreement to reset and first‑cycle comparison semantics. Second, across 274 mined model failures the specification‑only versus specification‑conditioned recovery gap ranges from -8.5 to +18.3 percentage points, indicating that a conventional repair score does not isolate whether the model actually used the faulty RTL.
From these findings we propose evaluation controls: report initialization, reset, first‑cycle and unknown‑value semantics; disclose oracle coverage and undecided cases; and include a specification‑only baseline. Together the resources and audit provide both a benchmark for behavioral repair and a protocol for diagnosing what contributes to the measured score.
The benchmark, training set, construction code, sanitized aggregate evidence, and reproduction materials are released at https://github.com/Evaaaz/RTLRepairBench.
Review