Recent breakthroughs in artificial intelligence are enabling scientists to make significant progress in mathematics, medicine, and materials science. To accelerate this momentum, researchers continuously develop new evaluation datasets for AI models. In the STEM domain, however, frontier models have already consumed most publicly available online data, creating a demand for human‑crafted datasets that capture the expertise of leading scholars. Existing STEM datasets, while valuable to the community, suffer from several shortcomings:
- Model performance on these benchmarks has saturated, leaving little headroom for meaningful differentiation;
- Taxonomy distributions are skewed, biasing evaluation toward certain sub‑fields;
- The prevalent multiple‑choice format does not align with how scientists actually employ AI;
- Answers and rationales often contain errors due to contest‑style data collection and time‑constrained review processes.
In this work we introduce Expert‑validated STEM QA, a high‑quality dataset created by 241 domain experts, covering Physics, Chemistry, Biology, and Mathematics with a total of 398 question‑answer pairs. Our construction pipeline incorporates four core principles:
- A carefully balanced taxonomy ensuring roughly equal representation across sub‑domains;
- Rigorous vetting of question contributors coupled with quality‑driven incentives;
- Multiple review rounds, where revisions are validated by expert consensus;
- A verifiable Q&A format that facilitates reproducibility and downstream evaluation.
Empirical evaluation reveals that state‑of‑the‑art models still perform poorly on this benchmark, $\text{performance}\approx 30\%$, indicating ample room for future improvements.
Blogger's Review: This dataset addresses several gaps in current STEM evaluation resources, particularly through expert validation and balanced taxonomy design. Expanding its scale and incorporating more interdisciplinary questions could further strengthen its utility for assessing general‑purpose AI capabilities.