annotated screenshots (#503)

* screenshot annotation

* fixes

* fix CI checks

* fixes

* fixes

* fixes

* fixes

* fixes
This commit is contained in:
Chris Tate
2026-02-18 22:20:01 -06:00
committed by GitHub
parent 06a32f4191
commit e2e259f1e2
12 changed files with 445 additions and 14 deletions
+2
View File
@@ -23,6 +23,7 @@ agent-browser scrollintoview <sel> # Scroll element into view
agent-browser drag <src> <dst> # Drag and drop
agent-browser upload <sel> <files> # Upload files
agent-browser screenshot [path] # Screenshot (--full for full page)
agent-browser screenshot --annotate # Annotated screenshot with numbered element labels
agent-browser pdf <path> # Save page as PDF
agent-browser snapshot # Accessibility tree with refs
agent-browser eval <js> # Run JavaScript
@@ -236,6 +237,7 @@ agent-browser reload # Reload page
--device <name> # iOS device name (e.g., "iPhone 15 Pro")
--json # JSON output (for scripts)
--full, -f # Full page screenshot
--annotate # Annotated screenshot with numbered element labels
--headed # Show browser window (not headless)
--cdp <port|url> # Connect via Chrome DevTools Protocol (port or WebSocket URL)
--auto-connect # Auto-discover and connect to running Chrome
+16
View File
@@ -78,12 +78,28 @@ agent-browser snapshot -i # Get fresh refs
agent-browser click @e1 # Use new refs
```
## Annotated screenshots
For visual context alongside text snapshots, use `screenshot --annotate` to overlay numbered labels on interactive elements. Each label `[N]` maps to ref `@eN`:
```bash
agent-browser screenshot --annotate ./page.png
# -> Screenshot saved to ./page.png
# [1] @e1 button "Submit"
# [2] @e2 link "Home"
# [3] @e3 textbox "Email"
agent-browser click @e2
```
Annotated screenshots also cache refs, so you can interact with elements immediately. This is useful when the text snapshot is insufficient -- unlabeled icons, canvas content, or visual layout verification.
## Best practices
1. Use `-i` to reduce output to actionable elements
2. Re-snapshot after page changes to get updated refs
3. Scope with `-s` for specific page sections
4. Use `-d` to limit depth on complex pages
5. Use `screenshot --annotate` when visual context is needed alongside refs
## JSON output