NeFut Logo NeFut
中 Admin Login

[CS.AI] TCSExam: Evaluating Frontier LLMs on the Full Range of TCS Research

Published at: 2026-11-06 22:00 Last updated: 2026-11-07 12:32
#AI #Machine Learning #LLM

We introduce TCSExam, a unified benchmark that spans the entire spectrum of theoretical computer science (TCS) research. It evaluates four tracks— theorem proving, solver engineering, constructive proofs, and direct attempts at open conjectures— thereby capturing concrete research outputs as well as progress toward long‑standing goals.\n\nTCSExam‑Full comprises 100 expert‑reviewed tasks whose submissions are checked by the Lean kernel or an executable verifier. TCSExam‑Lite selects five tasks per track, totaling 20 tasks, reducing the overall attempt count by 80% while keeping the per‑attempt budget unchanged.\n\nEight agents are evaluated on Full and nine on Lite; their scores are aggregated into the TCSExam Index across all tracks. GPT‑6 Astra leads Full with a score of 71.9, while Opus 5.5 tops Lite at 72.3. For the eight agents that appear in both settings, the Spearman correlation between Full and Lite rankings is $\rho=0.93$, confirming that Lite remains a compact yet highly discriminative benchmark.\n\nThe evaluations yield improved solver outputs and mathematical constructions, along with independently verified special cases and bounds derived from open‑conjecture attempts, highlighting the models’ potential to contribute to unresolved problems. TCSExam also supplies task sources, Docker build recipes, and evaluation tools to ensure reproducibility and extensibility for the community.\n\nReview: TCSExam offers a systematic, reproducible way to measure large language models’ real contributions to theoretical computer science, and the strong correlation of the Lite version makes it a practical tool for rapid model screening.

Original Source: https://arxiv.org/abs/2611.07962

[h] Back to Home