Evaluating frontier large language models (LLMs) is challenging due to static benchmarks suffering from contamination and saturation, preventing users from distinguishing top models and leaving developers blind to specific failure modes, while human preferences are subjective. This paper poses the question: Do LLMs know what other LLMs don't? Can we leverage this dynamic for evaluation?
We present LivingArena, an automated, contamination-resistant evaluation framework. In this framework, models take turns proposing questions, aiming to pose items that opponents cannot answer correctly. Questioners are encouraged to actively identify and exploit opponents' knowledge boundaries, receiving rewards when the answerer fails, while the answerer is rewarded otherwise. To ensure questions contain objectively verifiable answers, a judge panel of strong models validates them, penalizing questioners if validation fails.
Evaluating ten frontier LLMs, LivingArena yields a stable Elo leaderboard. Our behavioral analyses show that models identify and exploit their peers' cognitive boundaries: self-play and tournament logs indicate that they localize and double down on opponents' weak dimensions.
Beyond static knowledge recall, peer probing measures factual rigor and the higher-order ability to probe an opponent's weaknesses, correlating only weakly with human preference and offering a scalable, low-cost approach to continuous evaluation.
Blogger's Review: LivingArena's evaluation framework provides a fresh perspective on comparing large language models, particularly in dynamic assessment and knowledge boundary probing. This method emphasizes not just the amount of knowledge but also the interactions between models, showcasing the complexity and potential among LLMs. It opens new avenues for future model evaluation, warranting deeper exploration.