Context compaction enables long‑horizon coding agents to keep working beyond their context windows, but it can erase the state needed to finish a task. Prior evaluations have focused on efficiency, retained information, or end‑to‑end success, making it hard to isolate the impact of a specific history replacement. Compact‑Bench pairs full‑history continuations with summary‑only continuations from the same repository state, measuring how recurring details and successive production compactions are preserved.
The benchmark covers 35 SWE‑bench repairs, 146 agentic sessions, and 106 production chains. Across 14 summarizer backbones, the loss of full‑history success ranges from 5%‑64%: Claude Opus 4.8 preserves the most repairs, while DeepSeek‑Flash incurs the greatest damage. With Kimi‑K3 as summarizer, the Kimi Code prompt retains 19/22 task continuations, whereas Claude Code retains the highest proportion of recurring details (15.0%). LLM summaries keep at most 16.0% of recurring details compared with 79.1% for mechanical extraction; production summaries retain 15.7% at the first boundary.
Failures recur across methods: one repair is damaged by 13/14 backbones, another by all 5 prompts. These findings show that reliable compaction must preserve the progress needed to complete a task, and that merely retaining relevant information does not guarantee successful continuation. Code and data are publicly available on GitHub and HuggingFace.
Review