NeFut Logo NeFut
Admin Login

[CS.AI] LSREP: A Longitudinal State‑Replay Protocol for Evaluating Conversational Memory, with ICE v2 as an Audited Local‑First Architecture

Published at: 2026-09-16 22:00 Last updated: 2026-09-18 00:46
#AI #Machine Learning #LLM

LSREP (Longitudinal State‑Replay Evaluation Protocol) combines ordered replay, explicit lifecycle schedules, repeated probes, evolving reference answers, and mechanism‑fidelity checks to evaluate how conversational memory accumulates, ages, and incorporates revisions.\ \ ICE v2 serves as the architectural case study: a local‑first memory middleware featuring typed stores, retrieval fusion, and dynamic context budgets. The private single‑user instance comprises 1,985 turns, 219 distinct probes, and 1,211 probe‑checkpoint observations across 52 checkpoints.\ \ On three ordinary‑density datasets, ICE v2 shows a near‑zero mean quality difference from vector‑RAG while selecting 32% fewer fragments and using 6.6% more estimated prompt tokens. A fourth dense dataset reveals catastrophic failures of the unbudgeted baseline. The fidelity audit limits attribution: procedural retrieval is defective, several mechanisms remain unexercised, and graph utility is not established.\ \ In a complementary public diagnostic (LongMemEval), ICE v2 loses decisively to pure vector‑RAG: 50.8% versus 72.8% in the evidence‑only oracle and 43.0% versus 69.5% in the full‑S setting, with paired differences of -22.0 points (95% CI [-26.6, -17.4]) and -26.5 points ([-31.3, -21.8]). Conservative abstention accompanies severe multi‑session and temporal failures. ICE uses less context in this diagnostic, highlighting a quality‑cost trade‑off rather than superior efficiency.\ \ Together, replay, fidelity auditing, and public endpoint testing expose distinct failure modes that neither architectural descriptions nor aggregate scores identify alone.\ \ Review

Original Source: https://arxiv.org/abs/2609.16730

[h] Back to Home