Large language models (LLM) are increasingly deployed as long‑horizon conversational agents, driving interest in memory systems. Existing benchmarks mainly evaluate memory via QA‑style probing rather than in‑situ dialogue usage. To address this, we introduce LOCOMO‑CONV, a conversational memory benchmark derived from LoCoMo that features four query styles: dialog, implicit, counterfactual, and composed. We evaluate five representative memory systems, measuring both retrieval recall and end‑to‑end response quality. Experiments reveal that conversational framing exposes substantial retrieval gaps missed by QA benchmarks, especially for implicit and composed queries; multi‑facet query rewriting narrows the gap for raw‑turn memory but not for abstractive memory. Moreover, strong retrieval does not automatically translate into better response quality, and implicit queries exhibit silent grounding, where memory improves contextual understanding without explicitly surfacing the gold fact. These findings point to reasoning‑based memory elaboration as a promising direction, and we release auxiliary supportive_memory annotations that capture conversationally useful context beyond the original evidence.
Review