ERPBench introduces a state‑grounded evaluation paradigm that assesses screenshot‑only computer‑use agents operating within Enterprise Resource Planning (ERP) systems. ERP interfaces are dense, require multi‑step coordinated interactions, and errors often persist in business records rather than merely appearing on the screen, posing challenges for traditional desktop or web benchmarks.
To address these issues, the authors built the ERPBench benchmark: agents run on a reproducible live ERP environment, can only observe screen captures and submit simulated actions, and each task’s outcome is compared against ground‑truth values stored in the backend database, yielding precise success metrics. A production‑grade harness gates every agent action behind human approval, ensuring safe deployment.
Evaluating six closed‑source and open‑source agents revealed that strong performance on generic GUI tasks does not translate to enterprise reliability. Even when an agent reaches the correct form and saves it, the persisted record is frequently wrong: save rates can reach 85% while correct value insertion drops to as low as 3%. The authors further characterize failure modes specific to enterprise workflows, such as mismatched form field mappings, out‑of‑order transaction commits, and unhandled rollback exceptions.
The study highlights that enterprise‑oriented evaluation must go beyond surface interaction success and focus on data integrity and end‑to‑end workflow correctness, providing a crucial benchmark for the safe adoption of computer‑use agents in business settings.
Review