Benchmarks are essential for AI progress, yet they often suffer from two opposite problems: vague task definitions and overly detailed specifications. Vague definitions force models to guess, sometimes resorting to data leakage or reward‑hacking tricks; overly detailed specifications leak answer cues, allowing models to skip the hard steps the benchmark intended to measure, which shortens the benchmark’s useful lifespan. Repository‑level coding benchmarks have advanced the field but exhibit both issues. For example, SWE‑bench Pro, FEA‑bench, Doc2Feat‑bench and Multi‑SWE‑bench mitigate vagueness by appending hints derived from ground‑truth solutions. However, this practice can also inject ground‑truth information into the task statement, causing over‑specification. We experiment with several levels of specification, seeking a sweet spot between spoilers and guesswork. Results show that concise, information‑light hints provide reliable signals without compromising the benchmark’s difficulty. Based on these findings we propose a set of techniques for generating parsimonious hints and will release them to help the community obtain more trustworthy benchmark evaluations.
Review