NeFut Logo NeFut
中 Admin Login

[CS.AI] From Tapping to Hopping: Augmenting Mobile GUI Agents with App‑Native Deeplinks

Published at: 2026-09-28 22:00 Last updated: 2026-09-30 01:41
#algorithm #AI #Machine Learning

Mobile GUI agents accomplish tasks by performing generic actions such as taps and swipes, which work across many apps but become inefficient when a task requires navigating through several screens. An app‑native deeplink can replace a whole sequence of screen‑by‑screen GUI actions by opening the target screen directly. We therefore propose a hybrid interaction model: use deeplinks for fast navigation and fall back to conventional GUI actions for on‑screen operations and recovery.\ \ To enable this, we first run static analysis on the app binaries to extract candidate deeplink URLs. Each candidate is then exercised on real devices, its landing screen is observed, and invalid links are discarded. This yields a verified deeplink catalog where every working deeplink is paired with a description of its landing page.\ \ Using the catalog, we build GUI‑Hopper. During task planning GUI‑Hopper queries the catalog and prefers a usable deeplink to jump directly to the desired interface; if no deeplink is available or further on‑screen manipulation is needed, it falls back to a sequence of taps and swipes. Evaluations on several commercial apps running on real devices show that hybrid interaction markedly improves task success rates and execution speed, demonstrating the practical benefit of deeplinks for mobile GUI agents.\ \ Review

Original Source: https://arxiv.org/abs/2609.30887

[h] Back to Home