AI agents are increasingly adopted in enterprise and personal settings, gaining access to emails, databases, documents, and other tools to read, update, and disseminate sensitive information. Previous research on data leakage risks in agents has primarily focused on adversarial data exfiltration through prompt injections and jailbreaks. However, sensitive information can also be exposed during non-adversarial use, posing leakage risks even when users issue benign requests.
A joint evaluation by the Singapore AI Safety Institute and the Korea AI Safety Institute investigated agent data leakage across 12 realistic, non-adversarial tasks, including customer support, DevOps, web automation, and enterprise/personal productivity. The evaluation covered five risk types: lack of data awareness, audience awareness, policy compliance, data minimization, and access-boundary awareness. Both institutes tested a common set of scenarios mirroring real-world deployments using independent testing environments and task-specific LLM-judge rubrics.
Among the three tested agents, none achieved fully correct and safe execution across all scenarios. Successful task completion often coincided with data-handling failures, such as accessing unnecessary information or disclosing information to inappropriate recipients, indicating that capability and data-handling safety should be evaluated separately. Qualitative reviews revealed claim-action mismatches, simulation-aware behavior, user-simulator role reversals, and interpretation gaps in automated judging. Overall, the results indicate that operational data leakage is a first-order agent-safety concern distinct from adversarial exfiltration and provide a methodology for future evaluations of agent data-handling safety.
Blogger's Review: This article highlights the potential risks of LLM agents in data handling, particularly in non-adversarial scenarios, emphasizing the need for clear distinctions between capability and data-processing safety. It offers valuable insights for future AI system designs, advocating for a focus on non-adversarial data leakage risks to enhance the security of AI agents.