annotated screenshots (#503)
* screenshot annotation * fixes * fix CI checks * fixes * fixes * fixes * fixes * fixes
This commit is contained in:
@@ -112,6 +112,7 @@ agent-browser scrollintoview <sel> # Scroll element into view (alias: scrolli
|
||||
agent-browser drag <src> <tgt> # Drag and drop
|
||||
agent-browser upload <sel> <files> # Upload files
|
||||
agent-browser screenshot [path] # Take screenshot (--full for full page, saves to a temporary directory if no path)
|
||||
agent-browser screenshot --annotate # Annotated screenshot with numbered element labels
|
||||
agent-browser pdf <path> # Save as PDF
|
||||
agent-browser snapshot # Accessibility tree with refs (best for AI)
|
||||
agent-browser eval <js> # Run JavaScript (-b for base64, --stdin for piped input)
|
||||
@@ -401,6 +402,27 @@ agent-browser snapshot -i -c -d 5 # Combine options
|
||||
|
||||
The `-C` flag is useful for modern web apps that use custom clickable elements (divs, spans) instead of standard buttons/links.
|
||||
|
||||
## Annotated Screenshots
|
||||
|
||||
The `--annotate` flag overlays numbered labels on interactive elements in the screenshot. Each label `[N]` corresponds to ref `@eN`, so the same refs work for both visual and text-based workflows.
|
||||
|
||||
```bash
|
||||
agent-browser screenshot --annotate
|
||||
# -> Screenshot saved to /tmp/screenshot-2026-02-17T12-00-00-abc123.png
|
||||
# [1] @e1 button "Submit"
|
||||
# [2] @e2 link "Home"
|
||||
# [3] @e3 textbox "Email"
|
||||
```
|
||||
|
||||
After an annotated screenshot, refs are cached so you can immediately interact with elements:
|
||||
|
||||
```bash
|
||||
agent-browser screenshot --annotate ./page.png
|
||||
agent-browser click @e2 # Click the "Home" link labeled [2]
|
||||
```
|
||||
|
||||
This is useful for multimodal AI models that can reason about visual layout, unlabeled icon buttons, canvas elements, or visual state that the text accessibility tree cannot capture.
|
||||
|
||||
## Options
|
||||
|
||||
| Option | Description |
|
||||
@@ -422,6 +444,7 @@ The `-C` flag is useful for modern web apps that use custom clickable elements (
|
||||
| `--device <name>` | iOS device name, e.g. "iPhone 15 Pro" (or `AGENT_BROWSER_IOS_DEVICE` env) |
|
||||
| `--json` | JSON output (for agents) |
|
||||
| `--full, -f` | Full page screenshot |
|
||||
| `--annotate` | Annotated screenshot with numbered element labels (or `AGENT_BROWSER_ANNOTATE` env) |
|
||||
| `--headed` | Show browser window (not headless) |
|
||||
| `--cdp <port\|url>` | Connect via Chrome DevTools Protocol (port or WebSocket URL) |
|
||||
| `--auto-connect` | Auto-discover and connect to running Chrome (or `AGENT_BROWSER_AUTO_CONNECT` env) |
|
||||
|
||||
Reference in New Issue
Block a user