Vercel Labs releases agent-desktop for AI agent computer use via accessibility trees

Vercel Labs has introduced agent-desktop, a tool designed to enable AI agents to observe and operate desktop applications across macOS, Windows, and Linux. Rather than relying on fragile pixel-matching techniques, the tool leverages native operating system accessibility trees to identify and interact with UI elements.

The project provides a Command Line Interface (CLI) and a C library (FFI) to facilitate seamless integration into agentic workflows. It supports various interaction types, including clicking, typing, and window management, while maintaining a headless-by-default approach that avoids moving the physical mouse pointer or stealing foreground focus.

Technical implementation and platform support

• Interaction Model: The tool uses a snapshot-and-ref system where the OS accessibility tree is converted into a JSON structure containing unique reference IDs for every interactive element.
• Cross-Platform Accessibility: It utilizes AT-SPI on Linux, native accessibility APIs on macOS, and standard accessibility interfaces on Windows to map UI components.
• Headless Execution: By default, actions are performed through accessibility APIs without physical input simulation, though a --headed flag allows for physical pointer and keyboard movement.
• Developer Integration: Beyond the CLI, the release includes libagent_desktop_ffi, allowing developers to call the tool in-process via C bindings rather than spawning separate CLI processes.
• Linux Environment Requirements: Linux support requires a graphical session equipped with at-spi2-core, and Wayland users may need wl-clipboard for clipboard functionality.

Why it matters

This release provides a standardized way for developers to build autonomous agents that can navigate complex desktop software without custom computer vision models. By utilizing existing accessibility standards like AT-SPI, it lowers the barrier for integrating AI into Linux-based desktop automation and professional workflows.