Abstract
The rapid advancement of Large Language Models (LLMs) necessitates effective evaluation methods to guide downstream applications and future improvements. The Item Response Theory (IRT) model, combined with Computerized Adaptive Testing, has emerged as a promising framework for evaluating LLMs based on their response accuracy. Beyond mere response accuracy, the chain of thought (CoT) length serves as a crucial indicator of LLMs' reasoning abilities.
To leverage CoT length information, we propose the Latency-Response Theory (LaRT) model, which jointly models response accuracy and CoT length by introducing a key correlation parameter between latent ability and latent speed.
We derive an efficient stochastic approximation Expectation-Maximization algorithm for parameter estimation. We establish rigorous identifiability results for the latent ability and latent speed parameters to ensure the statistical validity of their estimation.
Through both theoretical asymptotic analyses and simulation studies, we demonstrate LaRT’s advantages over IRT in terms of superior estimation accuracy and shorter confidence intervals for latent trait estimation.
To evaluate LaRT with real data, we collect responses from various LLMs on popular benchmark datasets. We find that LaRT yields different LLM rankings than IRT and outperforms IRT across multiple key evaluation metrics including predictive power, item efficiency, ranking validity, and LLM evaluation efficiency. Code and data are available at GitHub.
Blogger's Review: The LaRT model innovatively incorporates latent speed parameters to combine response accuracy with reasoning depth, providing a more comprehensive perspective for LLM evaluation. This approach not only enhances evaluation accuracy but also offers crucial guidance for future model improvements, making it a valuable avenue for researchers to explore.