Recent multimodal foundation models have dramatically improved the ability to operate graphical user interfaces (GUIs) using visual observations and natural‑language instructions, yet maintaining strong performance on unseen applications remains difficult because real tasks often require long‑horizon procedural knowledge and application‑specific operation patterns. Building executable tasks and collecting corresponding trajectories still relies heavily on human‑annotated data, incurring high cost. To address this, we introduce MAGE (Manual‑augmented computer use AGent Execution and learning), a framework that leverages user manuals as a scalable source of procedural knowledge to enhance both inference and training. During execution, MAGE retrieves task‑relevant sections from manuals and employs a planner to generate explicit step‑by‑step guidance for computer use agents (CUAs). To bridge gaps between manuals and actual GUI environments—such as version differences or user settings—MAGE further adapts the policy through interaction with the environment. In training, MAGE converts manuals into executable task‑plan pairs, using the plans to steer environment rollouts toward task‑relevant states; the resulting trajectories are then used to optimize the policy without any human‑authored demonstrations. This unified design enables manuals to serve both execution and learning, markedly reducing reliance on human‑generated training data. Experiments across six applications show higher task success rates compared with strong baselines, confirming the effectiveness of manual‑guided planning and policy adaptation in new software environments. The code and reproduction scripts are publicly available at https://github.com/ku-autoai/MAGE.
Review