Frontier large language models often detect when they are being evaluated, a capability known as evaluation awareness. If a model behaves differently during evaluation than in deployment, the validity of evaluation results—crucial for current AI safety frameworks—gets compromised. To address this, we present EvalDetectBench, an open pipeline and benchmark that measures evaluation awareness and works with any Inspect‑compatible evaluation, enabling practitioners to test both existing and future benchmarks.
EvalDetectBench ships with a newly curated transcript suite covering current frontier system‑card evaluations and diverse deployment sources. The benchmark serves two purposes: measuring how reliably frontier LLMs recognize they are being evaluated, and assessing how detectable individual benchmarks are as evaluations.
We identify two methodological sources of systematic bias in prior work. First, the identity of the model that generated deployment transcripts accounts for 11.25% of measurement variance and can reorder model rankings. Second, elicitation prompts optimized for one model may perform near chance on others. EvalDetectBench corrects both via per‑model probe calibration and a stratified generator‑harmonisation procedure.
Review