agent-desktop is a free, open-source computer automation tool developed by Vercel Labs that enables AI agents to interact with desktop applications on Windows, macOS, and Linux. Built with Rust, the software uses operating system accessibility APIs to identify interface elements and perform actions without relying on screenshots, image recognition, or pixel-based navigation. Agent Desktop provides structured JSON representations of application interfaces, allowing AI agents to locate buttons, menus, text fields, and other interactive elements using unique references. The platform supports automated clicking, typing, window navigation, clipboard operations, and interface inspection, enabling AI agents to execute complex desktop workflows through command-line instructions. Developers can integrate agent-desktop with AI coding assistants and automation frameworks through its command-line interface, native C library, and Model Context Protocol (MCP) server.
Features
- AI-Powered Desktop Automation: Enables AI agents to interact with desktop applications by clicking buttons, entering text, navigating menus, managing windows, and performing automated workflows through operating system accessibility APIs.
- Accessibility-Based Interface Recognition: Reads application accessibility trees and returns structured JSON data containing interactive elements and unique references, eliminating the need for screenshot-based element detection or pixel matching.
- Cross-Platform Computer Control: Supports Windows, macOS, and Linux using native accessibility frameworks, including Windows UI Automation, macOS Accessibility APIs, and Linux AT-SPI.
- Command-Line Interface and Native Library: Provides a CLI and C library for executing desktop actions, inspecting application interfaces, managing sessions, and integrating automation capabilities into custom software.
- Model Context Protocol Integration: Includes an MCP server that exposes desktop automation commands as tools, allowing compatible AI agents and development environments to interact with desktop applications.
- - Reliable Element Targeting and Verification: Uses snapshot-based element references, structured error reporting, interface state inspection, and post-action verification to help agents detect changes, recover from stale references, and execute workflows reliably.