Abstract
Recent papers on autonomous penetration testing report high benchmark scores while adding multi-component security harnesses around frontier LLMs. Since these systems often alter both architecture and backbone models, it becomes challenging to discern how much performance improvement stems from the harness versus the underlying model. This paper presents a controlled study on the 104-task XBOW benchmark using default coding CLI agents as plain-agent baselines.
We first run Codex, OpenCode, and Pi with the same GPT-5 model, budget, target interface, and scoring rule. This phase identifies the strongest same-model baseline and tests whether security-specific prompt variants improve its observed score. We then compare the default Codex scaffold with published MAPTA and PentestGPT V2 results under the closest available model matches. Finally, we repeat the plain-agent experiment with GPT-5.2 and GPT-5.5 to measure model scaling within the same scaffold.
The results show a mixed but practical picture. Specialized harnesses can provide measurable benchmark lift and may improve cost efficiency, but plain coding agents already solve a large share of the benchmark; repeated plain-agent runs can match or exceed some published architecture scores in union coverage, and newer models substantially improve the same scaffold. Future evaluations should report model-matched plain-agent baselines before attributing benchmark gains to architecture design alone.
Blogger's Review: This study underscores the importance of baselines in assessing technical performance. By comparing the performance of different models, researchers can more clearly identify which improvements are due to architectural design rather than optimizations of the model itself, advancing the understanding of autonomous penetration testing. The potential of plain coding agents, in particular, deserves further exploration.