Computer‑using agents are becoming increasingly capable of handling complex tasks in the digital realm, yet high‑quality interaction trajectories remain scarce for specialized desktop applications. The scarcity stems from limited application knowledge, cumbersome GUI workflows, and incomplete exploration.
We introduce OS‑Wizard, an automated trajectory‑collection framework that tackles these challenges. It combines tutorial‑guided exploration with a hybrid GUI‑Code action space. The GUI layer supplies application‑specific knowledge, while code actions allow direct execution of native commands, dramatically simplifying click‑heavy workflows. During tree‑structured exploration, the framework checkpoints pivotal states and revisits them later to try alternative workflows, complementing linear rollouts and expanding coverage.
In our experiments we gathered 10.9K trajectories, amounting to 54.9K state‑action samples. Fine‑tuning open‑source base models on this corpus produced a best‑performing model that improves the specialized desktop‑workflow benchmark by 58.2 percentage points to 64.1%, surpassing the reported human performance of 59.8% and setting a new state‑of‑the‑art. The fine‑tuned model also shows gains on general desktop workflows that rely solely on GUI actions, indicating that the collected supervision benefits broader tasks. Comparative and ablation studies confirm the efficiency of the trajectory collection and the critical role of code actions.
Review: OS‑Wizard’s integration of tutorial knowledge with direct code execution offers a scalable solution for gathering high‑quality trajectories, substantially advancing the capabilities of agents in specialized software environments.