In evaluating smartphone personal assistants, longitudinal personal data is essential, yet acquiring context-rich evaluation data poses privacy-sensitive issues. To address this challenge, we present SenWorld, a physically grounded, deterministic, event-sourced digital-twin simulation capable of generating data with known ground truth.
In SenWorld, personas live through a full day in a world built from real map, weather, holiday, and network data; every observable signal is archived in full-system snapshots; and each evaluation case is labeled by a pointer to an existing record rather than by post-hoc annotation or a large language model (LLM) judge.
We evaluated this method with 16 personas in Beijing. The generated data closely matches the held-out real-user benchmark in category distribution (Jensen–Shannon divergence (JSD) 0.070) and in the daily rhythm of communication records (JSD below 0.1), although the generated records remain shorter than real ones.
Without scripted interaction, personas form a fully reciprocated dialogue subgraph and differentiated behavioral repertoires. Projected into 717 evaluation cases, the generated data exposes 78 failures in a production smartphone assistant, focusing on call and Short Message Service (SMS) records while contacts, schedules, and alarms never fail. The snapshot pointer confirms each failure as an assistant-side retrieval error, with no LLM judge involved.
Overall, SenWorld offers a privacy-safe, reproducible, and distribution-checked path to evaluation data whose labels are fixed by construction.
Blogger's Review: The innovation of SenWorld lies in its use of digital twin simulation technology to address privacy concerns while providing reliable evaluation data. This offers new insights for optimizing smart assistants' performance, especially in a landscape where data sensitivity is increasingly paramount, making it highly applicable.