Many long‑horizon LLM deployments are constrained by prompt budgets—latency, cost, and context length limits make full‑context prompting impractical as interaction length grows. The core issue thus shifts from raw recall to finding the memory design that offers the best quality‑token trade‑off in the compact‑memory regime.
We introduce RSM‑full, an online clustered‑memory pipeline engineered for a strong quality‑token Pareto point. RSM‑full combines two key design choices:
- A cosine‑gated max‑member merge write rule—when a new token arrives, it is merged into the cluster whose centroid has the highest cosine similarity, provided the similarity exceeds a threshold, preserving cluster representativeness.
- An atom‑aware grouped context packer—during context packing, tokens are grouped by atomic units, keeping tokens from the same atom contiguous and improving retrieval and concatenation efficiency.
On our primary compact‑memory benchmark AMA‑Bench, RSM‑full achieves $83\%$ of full‑context quality at only $32\%$ of the token cost under a $4$k budget. Averaged over four random seeds, it outperforms the closest streaming‑clustered baseline Online K‑Means by $+3.5$–$6.0$ percentage points ($p<0.05$).
These results demonstrate that, within tight token budgets, RSM‑full can dramatically shrink memory usage while preserving near‑full‑context reasoning quality.
Review