Abstract
Computer-use agents are evolving into capable software operators, yet their interface to desktop applications remains fragile: they rely on screenshots, predict coordinates, click, and hope for the intended visible state change. This merges target grounding, action execution, and outcome verification into a single ambiguous operation.
We present Tactile, an open-source tool layer that provides agents with more reliable 'hands and feet' for desktop use. Tactile transforms heterogeneous UI evidence—operating-system accessibility semantics, OCR-grounded text, and visual fallback regions—into action-grounded interface states: compact target candidates with source labels, roles or text, state, geometry, executable affordances, and verification cues.
Agents operate through an observe-act-verify loop that prefers native semantic actions when available, falls back to OCR-grounded coordinates when visible text is the best evidence, and maintains full provenance for replay and failure attribution. In macOSWorld-style tasks, incorporating Tactile improves Codex Success@100 from 41.1% to 50.0% overall, and from 45.2% to 55.3% on accessibility-adapted tasks; a 96-task cross-agent subset shows consistent gains across Codex, Claude Code, OpenCode, and Goose. These results indicate that reliable computer use requires not only stronger models but also a reusable execution substrate that presents software actions as semantic, verifiable, and auditable objects rather than anonymous screen coordinates.
Blogger's Review: The introduction of Tactile opens new avenues for enhancing the operational capabilities of computer agents. By transforming UI evidence into semantic actions, agents can execute tasks more reliably, thereby improving overall success rates. This is not just a technical advancement but a revolution in how computers interact with users. Looking forward to the widespread adoption of this tool in practical applications.