Conversational multimodal large language models (MLLMs) are increasingly expected to assist in professional workflows such as AI research, engineering design, product management, and business operations. Existing benchmarks, however, focus on informal daily interactions and personal‑life scenarios, using natural photos or isolated static artifacts and emphasizing recall, which does not reflect the structured, information‑dense, and frequently revised documents typical of professional settings.
To fill this gap we introduce DSV-Mem (Dense Stateful Visual Memory). It contains expert‑reviewed real‑world scenarios and 1,000 questions grouped into five user‑oriented categories: current state, past state, derived state, change history, and conflict/refusal. Each question requires the model to inspect extensive visual evidence and reconcile multiple document versions while tracking state precisely.
DSV-Mem adopts a Hartley‑inspired criterion that favors questions demanding broader visual‑evidence inspection, thereby raising evaluation rigor. We also provide a generation harness that decouples state‑transition synthesis from conversation filling, enabling automatic creation of evaluation suites.
We evaluated 27 configurations spanning frontier and open‑weight models together with various memory‑management strategies. The strongest baseline achieved less than $45\%$ accuracy on DSV-Mem. Analysis reveals three key findings: 1) multimodality and information density increase difficulty, yet the dominant factor is the number of governing state updates; 2) raw conversation length, OCR capability, and arithmetic are not the primary bottlenecks, and models often answer without verifying user premises against prior state updates; 3) increasing reasoning effort and improving memory management yield limited gains, whereas designs that are explicitly state‑aware prove more effective.
The benchmark and code will be released publicly for the community.
Review