NeFut Logo NeFut
Admin Login

[CS.AI] JarvisGUI: Towards Cross-Device GUI Agents with Dynamic Task Composition

Published at: 2026-09-11 22:00 Last updated: 2026-09-12 06:35
#algorithm #AI #Open Source

Real‑world GUI usage often spans multiple devices and platforms such as Android, Windows, and Ubuntu, requiring the transfer of intermediate results, shared‑state maintenance, and coordinated interaction across heterogeneous environments. Existing GUI benchmarks mainly focus on single‑device, statically defined tasks, which fails to probe agents' capabilities in cross‑device scenarios and leads to an overly optimistic assessment of their real‑world readiness.\ \ JarvisGUI abstracts GUI tasks as input‑output transformations and builds a lightweight type system on top of this abstraction. The type system records the input type, output type, and required device for each step, enabling automatic composition of multi‑step, cross‑device workflows at runtime without manual scripting.\ \ In experiments, JarvisGUI deploys virtual environments covering Android, Windows, and Ubuntu simultaneously, providing a unified evaluation of open‑source GUI agents across platforms. The results reveal that state‑of‑the‑art open‑source agents still struggle with three critical abilities:\

Original Source: https://arxiv.org/abs/2609.10451

[h] Back to Home