Language‑model agents compact their context for the current question, but the compacted context must also serve questions that appear later. We evaluate this on four corpora that annotate evidence for several questions per text and record agent trajectories, comparing every method with random deletion under the same budget.
On the MultiRC benchmark, keeping the sentences that gpt‑6‑luna (via the OpenAI Decisions API) deems necessary for the current question raises its accuracy by 11.4 points when only $25\%$ of the tokens are retained.
For later questions, however, no method that selects sentences for the current question outperforms random deletion. Deleting words instead of sentences works better: LLMLingua‑2, which ignores the question, beats random sentence deletion at the same token count by 12.5 points on the later MultiRC questions. It retains about twice the share of evidence content words compared to random deletion, and when content‑word retention is equal, its gain shrinks to 7.3 points (exploratory).
On FEVER, label accuracy can overstate what the kept text actually supports: when answers are scored against the retained evidence, the gain from sentence selection disappears, and the 21.0‑point gain of LLMLingua‑2 is not shown to rest on the kept text. Rankings also depend on the setting: on Evidence Inference, sentence selection beats LLMLingua‑2 at a nominal $25\%$ budget, while on $\tau$‑bench trajectories—where later tool calls need exact identifiers—sentence selection beats random deletion and LLMLingua‑2 shows no detectable gain.
Scores on the question for which a compaction was made do not reliably indicate how well it serves later questions.
Review