We introduce CivBench, an open‑source benchmark for evaluating language‑model agents in long‑horizon, tool‑mediated environments, using the Model Context Protocol (MCP).
A single episode spans over 300 turns, generating thousands of tool calls across a large action space, demanding sustained planning, state monitoring, and execution under partial observability.
The environment exposes 76 MCP tools and a narration layer that converts visual game state into structured text for the model.
We characterize agent behavior across four model families in 23 admissible runs. This study is a pilot, not a model ranking; aggregate outcomes at this scale do not reliably discriminate models.
To measure interface‑level performance we propose two metrics:
- Proactive Monitoring Rate (PMR): captures whether agents actively query latent strategic state.
- RAG@10: captures whether commitments stated in structured planning reflections are executed within the next 10 turns.
Two consistent patterns emerge:
- Under‑monitoring: despite a playbook instruction to query victory progress every 20 turns, agents query only every 30‑75 turns; in 7 out of 20 detectable defeats they failed to query within the 20‑turn warning window.
- Low commitment execution: agents frequently do not carry out near‑term commitments, with RAG@10 ranging from 48.2% to 65.8% across models.
These patterns persist even with tool access and explicit guidance, which we interpret as instructional deviation rather than a lack of capability.
The environment, scenarios, logs, metrics, and analysis pipeline are released on GitHub: https://github.com/lmwilki/civ6-mcp
Review