Current benchmark evaluation of large language models (LLMs) usually aggregates per‑example performance into a single scalar score, which is too coarse and ignores the diverse skill set covered by the datasets. To overcome this limitation and better exploit benchmark data, we introduce SkillEval, a cross‑model, skill‑level evaluation framework.
SkillEval combines two recent advances: (1) it leverages the meta‑cognitive abilities of LLMs to automatically discover interpretable skills from questions, and (2) it applies cognitive diagnosis models (CDMs) from psychometrics to estimate per‑skill abilities.
We evaluate 3,811 LLMs on 9,523 items across five benchmarks (MATH, BBH, GPQA, MuSR, IFEval). Compared with the traditional average‑based paradigm, SkillEval yields more reliable, better‑calibrated, and concise model profiles that predict behavior on unseen items more accurately.
SkillEval also proves useful in two downstream tasks: (1) LLM routing, which dynamically assigns each query to a capable yet efficient model, cutting inference cost by over 60% while improving performance; (2) Computerized Adaptive Testing (CAT), where an unseen model can be efficiently evaluated using a trained CDM.
Furthermore, the skill profiles reveal insights such as how benchmarks share skills and how instruction‑tuned models differ from their base counterparts.
Review