As AI models rapidly improve, their autonomous coding capabilities are becoming increasingly evident, particularly with demonstrations like AI implementing a C compiler. However, existing coding benchmarks typically focus on shorter tasks and often involve human guidance, making systematic comparisons difficult.
To address these challenges, we introduce MirrorCode, a long-horizon coding benchmark based on reimplementing entire software projects. In MirrorCode, AI agents must replicate the functionalities of an existing program without access to its source code. AI solutions must match the original program's output exactly in end-to-end tests, including held-out tests.
MirrorCode includes 25 target programs across various computing domains: Unix utilities, data serialization and query tools, bioinformatics, interpreters, static analysis, cryptography, and compression. Existing AI models can already reimplement complex software, with the strongest model scoring 56% across the benchmark. For instance, AI can reimplement gotree, a 16,000-line bioinformatics toolkit, a task we believe would take weeks for a human engineer.
However, exploring the frontier of performance requires a larger inference budget than typical benchmarks, e.g., $2,600 over 19 days for a single attempt on a large task. We demonstrate that AI agents can already tackle long-horizon software engineering tasks, especially when requirements are precisely specified.
More broadly, our work suggests that AI will have transformative effects on software engineering as autonomous agents continue to improve.
Blogger's Review: The introduction of MirrorCode marks a significant milestone in AI's role in software development, showcasing its potential in complex tasks and highlighting the trend towards automation in software engineering. As AI technology evolves, developers may increasingly turn to these tools to enhance efficiency.