When completing tasks or filtering synthetic training data, LLM agents need trustworthy ways to validate their outputs. This work studies verifier generation: given a task instruction and the available environment, an agent writes an executable verifier that must accept every valid solution and reject invalid ones. We introduce VerifierBench, a benchmark containing 222 tasks drawn from six agent suites covering tool‑call, state‑based, and code verification scenarios, together with 4,993 hidden test cases that include alternative valid solutions and adversarial attacks. To screen source verifiers and test suites we propose Verifier4Verifier (V4V), a six‑check audit pipeline.
Across 19 models evaluated in a common agent harness, normalized balanced accuracy ranges from 10.6% to 62.1%. Generated verifiers tend to reject invalid solutions more reliably than they accept valid ones, but this apparent strength can hide overly restrictive behavior: most false rejections stem from verifiers that accept no tested solution at all. Case studies reveal unsupported assumptions about solution formats, tool usage, and code changes that exclude legitimate answers. These errors survive self‑testing because agents evaluate solutions that share the verifier's mistaken assumptions, or submit verifiers without confirming that any correct solution passes.
The findings suggest that testing should verify both that a verifier accepts a known correct solution and that it does not reject other solutions permitted by the task specification. The project page and dataset are publicly available.
Review