Digital Adoption Platforms (DAPs) often embed overlays in web systems to guide users through unfamiliar interfaces quickly. Completing real tasks, however, involves a sequence of actions across changing page states, rather than just clicking buttons on a single page.
Previous studies have treated automated web agent action generation and guide text generation as separate problems, often relying on textual page representations like the DOM or accessibility trees instead of the rendered screens used by humans.
In this work, we introduce MAG, the first benchmark that unifies task execution and guide writing into a single Multimodal Action and Guide task, utilizing two grounding schemes based on screenshots: Set-of-Mark element selection and raw pixel coordinates.
We also build a complete harness for this compound task, covering annotation assisted by LLMs, human verification, training, evaluation in live environments, and joint metrics for actions and guides.
With this harness, we evaluate frontier API models and open multimodal models, providing detailed analyses. Finally, we design a GRPO training method augmented with expert trajectories, which nearly doubles the success rate of a supervised 9B agent (from 6.9% to 13.2%) while improving guide quality. Even the strongest model completes fewer than 40% of the tasks, indicating significant room for future research.
Blogger's Review: The introduction of MAG effectively addresses the separation between multimodal task execution and guide text generation, offering new insights for complex web operations. By leveraging LLM assistance and expert trajectories, it significantly enhances model performance, showcasing vast potential for development in this field. Future research should focus on optimizing models to improve task completion rates.