Rapid advances in AI capabilities are pushing benchmark scores toward saturation. When many models achieve near‑perfect results, a benchmark loses its discriminative power and its usefulness for AI safety cases and frontier policy decisions. Yet each benchmark item also records a full transcript of how the model solved the task. We investigate whether these transcripts can still be used to assess model capabilities after scores have saturated.
We retrospectively simulate saturation using historical benchmarks and model releases, and restrict ourselves to correctly solved transcripts to predict capabilities on held‑out, unsaturated items. The evaluated extraction protocols are: pairwise judging, single‑transcript absolute scoring, predictions from deterministic transcript features, and predictions from transcript embeddings.
Results show that pairwise judging achieves the lowest prediction error, reducing it by a factor of 5.3 compared to a baseline that uses only saturated benchmark scores. Capability‑relevant information is spread across transcript sections but is especially concentrated in channels such as reasoning steps and tool‑calls.
The predictive power of these protocols does not always generalize across distribution shifts; for example, they tend to underestimate the capabilities of the first reasoning models introduced in early 2025. Thus, while saturated benchmarks retain useful capability signals in their transcripts, extending their lifetime requires methods that can robustly extract these signals.
Review