annotated screenshots (#503)

* screenshot annotation

* fixes

* fix CI checks

* fixes

* fixes

* fixes

* fixes

* fixes
This commit is contained in:
Chris Tate
2026-02-18 22:20:01 -06:00
committed by GitHub
parent 06a32f4191
commit e2e259f1e2
12 changed files with 445 additions and 14 deletions
+23
View File
@@ -112,6 +112,7 @@ agent-browser scrollintoview <sel> # Scroll element into view (alias: scrolli
agent-browser drag <src> <tgt> # Drag and drop
agent-browser upload <sel> <files> # Upload files
agent-browser screenshot [path] # Take screenshot (--full for full page, saves to a temporary directory if no path)
agent-browser screenshot --annotate # Annotated screenshot with numbered element labels
agent-browser pdf <path> # Save as PDF
agent-browser snapshot # Accessibility tree with refs (best for AI)
agent-browser eval <js> # Run JavaScript (-b for base64, --stdin for piped input)
@@ -401,6 +402,27 @@ agent-browser snapshot -i -c -d 5 # Combine options
The `-C` flag is useful for modern web apps that use custom clickable elements (divs, spans) instead of standard buttons/links.
## Annotated Screenshots
The `--annotate` flag overlays numbered labels on interactive elements in the screenshot. Each label `[N]` corresponds to ref `@eN`, so the same refs work for both visual and text-based workflows.
```bash
agent-browser screenshot --annotate
# -> Screenshot saved to /tmp/screenshot-2026-02-17T12-00-00-abc123.png
# [1] @e1 button "Submit"
# [2] @e2 link "Home"
# [3] @e3 textbox "Email"
```
After an annotated screenshot, refs are cached so you can immediately interact with elements:
```bash
agent-browser screenshot --annotate ./page.png
agent-browser click @e2 # Click the "Home" link labeled [2]
```
This is useful for multimodal AI models that can reason about visual layout, unlabeled icon buttons, canvas elements, or visual state that the text accessibility tree cannot capture.
## Options
| Option | Description |
@@ -422,6 +444,7 @@ The `-C` flag is useful for modern web apps that use custom clickable elements (
| `--device <name>` | iOS device name, e.g. "iPhone 15 Pro" (or `AGENT_BROWSER_IOS_DEVICE` env) |
| `--json` | JSON output (for agents) |
| `--full, -f` | Full page screenshot |
| `--annotate` | Annotated screenshot with numbered element labels (or `AGENT_BROWSER_ANNOTATE` env) |
| `--headed` | Show browser window (not headless) |
| `--cdp <port\|url>` | Connect via Chrome DevTools Protocol (port or WebSocket URL) |
| `--auto-connect` | Auto-discover and connect to running Chrome (or `AGENT_BROWSER_AUTO_CONNECT` env) |