Abstract
Recognition tells an agent what is in an image; personal memory affects what is worth looking up next. In a camera-first setting, the user can send only an image, so the agent must form the lookups. We study whether personal visual memory improves agent-side tool choice and tool arguments, thereby enabling more user-aligned multi-tool lookups.
The design uses a three-layer personal visual memory (profile, short-term focus, observations) that is loaded on each turn to condition an LLM tool-calling loop under camera-first intake, including conflict-aware write-back intended to refresh the user model for later captures.
Experiments on 800 images paired with synthetic memory blocks show that removing the full three-layer memory block reduces tool-query relevance by 0.47 points absolute (4.21 - 3.74 on a 5-point scale; 11.2% relative) and end-to-end utility by 0.082 absolute (0.842 - 0.760; 9.7% relative). These results measure memory conditioning of tool policy under image-only intake with fixed synthetic blocks, not multi-session write-back from live user histories.
Blogger's Review: This paper delves into the application of personal visual memory in camera-first visual agents, highlighting the impact of memory on tool selection. The design of the three-layer memory architecture provides crucial insights for enhancing user experience and validates the key role of memory in optimizing tool calls. Future research could explore better integration of dynamic user histories to further elevate the intelligence of the system.