Large language models (LLMs) are increasingly embedded into first‑principles computational pipelines, yet their ability to configure scientific calculations remains poorly quantified. To address this gap we introduce INCARBench, a benchmark targeting the INCAR input file of the Vienna Ab initio Simulation Package (VASP). The benchmark comprises two tasks:
- Configuration generation – the model generates a full set of INCAR parameters from a textual description of the calculation.
- Configuration repair – given an INCAR with errors, the model must identify and correct the faulty settings while preserving already‑correct entries.
We evaluated 19 state‑of‑the‑art models (including GPT‑4, Claude, Llama‑2, etc.) across three hierarchical metrics:
- Semantic accuracy – does the output match the intent expressed in natural language?
- Policy accuracy – does the output follow VASP‑recommended parameter policies?
- Task‑critical correctness – are the physically important parameters (e.g.,
ENCUT,EDIFF,LDAUseries) scientifically valid?
While several models achieve >90% semantic and policy accuracy, their task‑critical correctness stays below 60%, indicating that high-level correctness does not guarantee a scientifically sound configuration.
Error concentration: most mistakes occur in physically coupled settings that require multiple constraints to be satisfied simultaneously, such as DFT+U, magnetism (ISPIN, MAGMOM), and strongly correlated materials. These scenarios demand concurrent handling of electron correlation, magnetic ordering, and convergence criteria, which current models struggle to reconcile.
Repair findings: models can often spot and fix individual errors, but preserving already‑correct parameters proves difficult. Even when a critical error is corrected, new inconsistencies are frequently introduced, highlighting configuration preservation as a distinct and harder capability.
In summary, INCARBench quantifies scientific configuration as a measurable LLM capability and lays a foundation for developing more reliable AI assistants for computational materials science.
Review