Introduction
We present AgentLens, a production-assessed benchmark for interactive code agents. Most code-agent benchmarks reduce a run to a single bit -- did the task pass? -- but the people who actually use these agents experience the entire trajectory: how the agent follows instructions, uses its tools, verifies its own work, recovers from mistakes, and interacts with them along the way. AgentLens evaluates that whole trajectory.
Evaluation Method
AgentLens pairs formal verification, where an objective check exists, with LLM-written trajectory reviews and side-by-side comparisons, so that each run yields a readable explanation of why the score is what it is. This makes AgentLens useful for more than ranking models: we use it to diagnose model behavior, compare successive versions of our own agent, and catch product regressions in a nightly evaluation pipeline.
Open Source Release
We release the benchmark as open source at GitHub.
Blogger's Review: The introduction of AgentLens fills a gap in the evaluation of code agents, emphasizing the entire interactive process rather than a single outcome. This significantly enhances the understanding and debugging capabilities of model behavior. The open-source nature also facilitates further research within the community, making it a noteworthy tool.