annotated screenshots (#503)

* screenshot annotation

* fixes

* fix CI checks

* fixes

* fixes

* fixes

* fixes

* fixes
This commit is contained in:
Chris Tate
2026-02-18 22:20:01 -06:00
committed by GitHub
parent 06a32f4191
commit e2e259f1e2
12 changed files with 445 additions and 14 deletions
+20
View File
@@ -80,6 +80,7 @@ agent-browser wait 2000 # Wait milliseconds
# Capture
agent-browser screenshot # Screenshot to temp dir
agent-browser screenshot --full # Full page screenshot
agent-browser screenshot --annotate # Annotated screenshot with numbered element labels
agent-browser pdf output.pdf # Save as PDF
```
@@ -278,6 +279,25 @@ agent-browser snapshot -i # MUST re-snapshot
agent-browser click @e1 # Use new refs
```
## Annotated Screenshots (Vision Mode)
Use `--annotate` to take a screenshot with numbered labels overlaid on interactive elements. Each label `[N]` maps to ref `@eN`. This also caches refs, so you can interact with elements immediately without a separate snapshot.
```bash
agent-browser screenshot --annotate
# Output includes the image path and a legend:
# [1] @e1 button "Submit"
# [2] @e2 link "Home"
# [3] @e3 textbox "Email"
agent-browser click @e2 # Click using ref from annotated screenshot
```
Use annotated screenshots when:
- The page has unlabeled icon buttons or visual-only elements
- You need to verify visual layout or styling
- Canvas or chart elements are present (invisible to text snapshots)
- You need spatial reasoning about element positions
## Semantic Locators (Alternative to Refs)
When refs are unavailable or unreliable, use semantic locators: