With AI development accelerating, benchmarks calibrated against humans or current AI quickly become circular or saturated. This raises a fundamental question: how to evaluate machines that may surpass both existing systems and the test designers?
One answer treats evaluation as an inverse problem—given observations and prior knowledge, difficulty depends on the complexity of the underlying mechanism. Levin's $K_t$, a computable variant of Kolmogorov complexity, provides a theoretical upper bound, balancing the length of the simplest representation with the logarithm of the runtime.
Earlier C‑tests relied on minimal Universal Turing Machines, capable of only very limited representations and no prior knowledge. This paper treats large language models (LLMs) as UTMs that embed a vast compendium of human culture, using prompts as natural‑language programs.
We present the first C‑test based on $K_t$ that uses LLMs as the reference machine. A lightweight generation and certification pipeline can produce items at difficulties higher than any human or frontier model can solve, measuring difficulty on a ratio scale in bits.
A battery of certified items was administered to twelve LLMs. The resulting agent characteristic curves decline monotonically with difficulty, reaching the uncharted “supra‑intelligence” region—items we can certify but that remain beyond the reach of current frontier intelligence.
Review