Modern voice assistants are often shared among several users and must answer questions about past conversations, e.g., “When did I originally plan to leave?”. This goes beyond retrieving topically similar passages; the system must identify the current speaker, recover the relevant past state, and separate it from later revisions. PERSIST is a persistent memory system for multi‑session, multi‑speaker spoken dialogue that explicitly models Who, What, and When. It structures cross‑session histories into readable event records and retrieves them with a 3W joint scoring mechanism that combines semantic content, acoustic speaker identity, and temporal state. For real‑time full‑duplex interaction, PERSIST reuses intermediate representations from the dialogue backbone, avoiding query‑audio re‑encoding and cutting retrieval latency from 578.42 ms to 7.03 ms. We also introduce SpokenTrace, a diagnostic benchmark that factorizes evaluation into memory tasks and speaker‑query types, exposing failures in recall, speaker attribution, and temporal‑state tracking. On SpokenTrace, PERSIST achieves 85.08% end‑to‑end task accuracy and raises all‑support EM@3 from 49.01% with BGE‑large to 82.10%.
Review