agent-desktop (vercel-labs)Partly tested🌏 World

vercel-labs/agent-desktop lets agents read and operate the desktop through the OS accessibility tree instead of screenshot pixels (@SimonObserves' hands-on notes)

Agents usually drive a desktop by staring at screenshot pixels, or by reading the accessibility tree screen readers already use. Vercel Labs' agent-desktop (Apache-2.0, npm 1.2.0) is a model-free CLI/C library: `snapshot` returns JSON with @refs, then `click`/`type`; headless by default so it won't steal focus. Overseas writer @SimonObserves documented a Linux run: almost nothing works without AT-SPI; with it, listing windows and menu clicks work, and terminal typing needs `--headed`. On our headless Linux box after `npm install -g agent-desktop`, `version` was 1.2.0 and `status` reported accessibility=denied (no D-Bus session) with empty supported_surfaces—matching that write-up. We did not wire Claude/Codex MCP to drive a real desktop.

Visit site agent-desktop (vercel-labs) ↗
Pricing
Free
Region
Global
Where it's from
Overseas
Added
Nov 6, 2026
Tested on
Nov 6, 2026
Version we used
agent-desktop npm 1.2.0; protocol envelope version 2.6; repo HEAD b43ac4f
Categories
CodingAutomation & AgentsProductivity
Original author
SimonObserves @SimonObserves ↗
Our test
Partly tested
Cost / time
Free/open source (Apache-2.0). Driving it still needs a Claude Code / Codex (etc.) subscription; Linux may need at-spi2-core and a real desktop session.
Source
X @SimonObserves ↗GitHub vercel-labs/agent-desktop ↗

The task

Let a coding agent click buttons, fill forms and read control state in real desktop apps—without guessing from screenshot coordinates.

Steps

  1. Install: `npm install -g agent-desktop` (Node ≥18, ships native binaries) or `brew install agent-desktop`. Run `agent-desktop version` and `agent-desktop status` first.
  2. Permissions: on macOS use `agent-desktop permissions --guide`; on Linux you need AT-SPI / a D-Bus desktop session—cloud boxes and bare SSH often show accessibility denied.
  3. Load built-in skills: `agent-desktop skills get desktop` and `… get platform` (compiled into the binary). Or `npx skills add https://github.com/vercel-labs/agent-desktop --skill agent-desktop`.
  4. Loop: `agent-desktop snapshot --app "AppName" -i` → `click`/`type` a `@snapshot:ref` → snapshot again to verify. Stale UI → STALE_REF; retake.
  5. Controls that lack semantic typing (e.g. some terminals) may POLICY_DENIED in headless mode—add `--headed` for real input. Snapshots/clipboard can hold private data; don't upload them.

Prompt

Use agent-desktop on this machine: run status to confirm accessibility, snapshot -i the target app, act only on refs from the current snapshot, and re-snapshot after every action—never guess screen coordinates.

What the author says

@SimonObserves / vercel-labs: among computer-use paths, agent-desktop uses the accessibility tree; no model—just hands and eyes. Without AT-SPI on Linux almost nothing works; with it, list apps/windows and click menus; terminal typing is denied headless until --headed. Failures return reasoned JSON (STALE_REF, POLICY_DENIED) rather than fake screenshots.

Our test

npm install -g [email protected]; ran version, status, --help. Read vercel-labs/agent-desktop README and @SimonObserves' article. No AT-SPI, no desktop session, no real GUI snapshot/click, no MCP wiring.

What we found when we tried it:version: ok, os=linux, target=x86_64, version=1.2.0. status: permissions.accessibility.state=denied with a no D-Bus session suggestion; supported_surfaces=[]. Matches the author's Linux note that permissions are the first wall.

How to get in

npm / Homebrew / GitHub Releases; needs a local desktop and a11y permission. CN users may want an npm mirror. Hands-on write-up and docs are in English.

Who it suits

People who need an agent on local GUIs (settings, IDEs, office apps) that expose solid accessibility trees; headless cloud boxes are a poor fit.

Pitfalls

  • Without a desktop session / AT-SPI, status stays denied and list/snapshot barely work—fix local a11y before blaming the tool.
  • Games, custom-drawn UIs and remote-desktop views often expose no usable tree.
  • Refs belong to one snapshot; UI changes → retake or you get STALE_REF.
  • Our run: global npm install 1.2.0, version OK; this box has no D-Bus session so status=denied. No real desktop clicks.

More in this category