vercel-labs/agent-desktop lets agents read and operate the desktop through the OS accessibility tree instead of screenshot pixels (@SimonObserves' hands-on notes)
Agents usually drive a desktop by staring at screenshot pixels, or by reading the accessibility tree screen readers already use. Vercel Labs' agent-desktop (Apache-2.0, npm 1.2.0) is a model-free CLI/C library: `snapshot` returns JSON with @refs, then `click`/`type`; headless by default so it won't steal focus. Overseas writer @SimonObserves documented a Linux run: almost nothing works without AT-SPI; with it, listing windows and menu clicks work, and terminal typing needs `--headed`. On our headless Linux box after `npm install -g agent-desktop`, `version` was 1.2.0 and `status` reported accessibility=denied (no D-Bus session) with empty supported_surfaces—matching that write-up. We did not wire Claude/Codex MCP to drive a real desktop.
Visit site agent-desktop (vercel-labs) ↗- Pricing
- Free
- Region
- Global
- Where it's from
- Overseas
- Added
- Nov 6, 2026
- Tested on
- Nov 6, 2026
- Version we used
- agent-desktop npm 1.2.0; protocol envelope version 2.6; repo HEAD b43ac4f
- Categories
- CodingAutomation & AgentsProductivity
- Original author
- SimonObserves @SimonObserves ↗
- Our test
- Partly tested
- Cost / time
- Free/open source (Apache-2.0). Driving it still needs a Claude Code / Codex (etc.) subscription; Linux may need at-spi2-core and a real desktop session.
The task
Let a coding agent click buttons, fill forms and read control state in real desktop apps—without guessing from screenshot coordinates.
Steps
- Install: `npm install -g agent-desktop` (Node ≥18, ships native binaries) or `brew install agent-desktop`. Run `agent-desktop version` and `agent-desktop status` first.
- Permissions: on macOS use `agent-desktop permissions --guide`; on Linux you need AT-SPI / a D-Bus desktop session—cloud boxes and bare SSH often show accessibility denied.
- Load built-in skills: `agent-desktop skills get desktop` and `… get platform` (compiled into the binary). Or `npx skills add https://github.com/vercel-labs/agent-desktop --skill agent-desktop`.
- Loop: `agent-desktop snapshot --app "AppName" -i` → `click`/`type` a `@snapshot:ref` → snapshot again to verify. Stale UI → STALE_REF; retake.
- Controls that lack semantic typing (e.g. some terminals) may POLICY_DENIED in headless mode—add `--headed` for real input. Snapshots/clipboard can hold private data; don't upload them.
Prompt
Use agent-desktop on this machine: run status to confirm accessibility, snapshot -i the target app, act only on refs from the current snapshot, and re-snapshot after every action—never guess screen coordinates.
What the author says
@SimonObserves / vercel-labs: among computer-use paths, agent-desktop uses the accessibility tree; no model—just hands and eyes. Without AT-SPI on Linux almost nothing works; with it, list apps/windows and click menus; terminal typing is denied headless until --headed. Failures return reasoned JSON (STALE_REF, POLICY_DENIED) rather than fake screenshots.
Our test
npm install -g [email protected]; ran version, status, --help. Read vercel-labs/agent-desktop README and @SimonObserves' article. No AT-SPI, no desktop session, no real GUI snapshot/click, no MCP wiring.
What we found when we tried it:version: ok, os=linux, target=x86_64, version=1.2.0. status: permissions.accessibility.state=denied with a no D-Bus session suggestion; supported_surfaces=[]. Matches the author's Linux note that permissions are the first wall.
How to get in
npm / Homebrew / GitHub Releases; needs a local desktop and a11y permission. CN users may want an npm mirror. Hands-on write-up and docs are in English.
Who it suits
People who need an agent on local GUIs (settings, IDEs, office apps) that expose solid accessibility trees; headless cloud boxes are a poor fit.
Pitfalls
- Without a desktop session / AT-SPI, status stays denied and list/snapshot barely work—fix local a11y before blaming the tool.
- Games, custom-drawn UIs and remote-desktop views often expose no usable tree.
- Refs belong to one snapshot; UI changes → retake or you get STALE_REF.
- Our run: global npm install 1.2.0, version OK; this box has no D-Bus session so status=denied. No real desktop clicks.
More in this category
- A demo video per docs page that flags itself when the UI changesVideoScreenplayClaude CodeTool free; uses your agent's quota◐ Partly testedOverseas
- AI backend code that guards against races, leaks and bad migrationsCodingbackendskillsClaude CodePostgreSQLFree; about 2× tokens per task◐ Partly testedOverseas
- An afternoon of AI work, a 3D action game you play in the browserCodingClaude CodeTripoJimengSkill free; Claude quota plus asset tools◐ Partly testedChina
- Keep a long-running AI-built project from turning into a messCodingClaudeCodexoil-delegateNeeds Claude and Codex plans◐ Partly testedChina
- Use a web app once and get an AI write-up of how it worksResearchREAClaude CodeChromeFree tools; needs Claude or Codex◐ Partly testedOverseas