Large language models are increasingly used as scientific assistants, yet reliable ways to assess whether they can actively investigate a system are still missing. A proper evaluation must let agents choose experiments, interpret limited evidence, and make causal decisions while keeping the environment consistent across repeated observations and interventions and preserving objective ground truth. We introduce ACED-BENCH (Active Causal Evidence and Decision Benchmark), which separates the readable scientific surface from the formal causal world: LLMs generate natural‑language scenarios and task descriptions, while hidden probabilistic graphical models define the causal graph, data‑generating process, intervention semantics, validation checks, and oracle answers. This yields scientific worlds that are natural for agents to explore yet precise enough for scalable programmatic evaluation. The main leaderboard ACED-CORE contains 180 tasks: 60 set‑valued causal ancestry/descendancy queries and 120 constrained causal‑decision tasks. We also provide ACED-QUAL, a 150‑question yes/no qualification check using the same hidden‑PGM interface. Seven contemporary LLMs and two earlier models are evaluated across three active‑agent designs. ACED-QUAL tests whether agents can use the interface to draw conclusions from evidence, while ACED-CORE tests whether they can decide what evidence to collect and when to commit to an answer. Evidence‑ledger analysis pinpoints failures: testing the wrong intervention, losing relevant evidence, or continuing after sufficient evidence appears. These findings suggest that reliable AI co‑scientists need explicit support for experiment design, evidence retention, answer commitment, and stopping.
Review