Commit Graph
9 Commits
Author SHA1 Message Date
Chris Tateandctate 60f3afcf61 Add iframe support for CLI interactions and snapshots (#869)
* Add iframe support for CLI interactions and snapshots

This PR adds comprehensive iframe support to the agent browser CLI, allowing users to interact with elements inside iframes seamlessly.

## Problem
Users couldn't interact with elements inside iframes via the command line. The existing `frame` command was non-functional as it set `active_frame_id` but no other code read this value.

## Changes Made

### Enhanced Frame Context Tracking
- Added `frame_id` field to `RefEntry` to track which frame each element reference belongs to
- Updated `RefMap::add` and related methods to accept and store frame context
- Modified element resolution functions to use frame context from ref entries

### Improved Frame Command
- Fixed the existing `frame` command to actually work by threading `active_frame_id` through snapshot operations
- Added support for iframe element references (e.g., `frame @e2`) in addition to CSS selectors
- Enhanced frame detection to work with both named frames and iframe elements

### Updated Snapshot Behavior
- Modified `take_snapshot` to accept optional frame context parameter
- Updated all snapshot call sites to pass appropriate frame context
- Maintained backward compatibility while enabling frame-scoped operations

### Element Resolution Updates
- Updated `resolve_element_center` and `resolve_element_object_id` to use frame context from ref entries
- Modified `find_node_id_by_role_name` to support frame-specific element lookup
- Ensured all interaction functions work correctly within iframe contexts

## Implementation Details
- Frame context is now properly propagated through the entire element interaction pipeline
- The `frame` command can accept both CSS selectors and element references
- All existing functionality remains intact while adding iframe capabilities
- Added `Iframe` to interactive roles for better element discovery

Fixes #863

* docs: add iframe support documentation

Document the new iframe capabilities across all documentation surfaces:
- Auto-inlining of iframe content in snapshots
- Direct interaction with iframe element refs
- frame command support for element refs (@e3)
- Scoped snapshots via frame switching

* fix: pass active frame context to diff snapshots and fix nameless iframe lookup

- handle_diff_snapshot now respects active_frame_id instead of always
  passing None, so diff snapshots work correctly inside iframes
- Nameless/id-less iframes now fall back to src URL (or null) instead of
  the literal string 'frame' which never matched any frame in the tree

* fix: resolve iframe frame ID via DOM.describeNode and reduce code duplication

- handle_frame: Use DOM.describeNode + contentDocument.frameId to resolve
  iframe frame IDs directly, fixing failures for nameless iframes that
  lack name/id/src attributes
- element.rs: Deduplicate add() by delegating to add_with_frame()
- snapshot.rs: Guard against out-of-bounds insert_str when iframe marker
  is on the last line without a trailing newline

---------

Co-authored-by: ctate <366502+ctate@users.noreply.github.com>
2026-03-17 10:51:17 -05:00
Chris Tate 315d191606 inspect (#736)
* inspect

* fixes

* improvements

* fixes

* fixes

* improvements

* fix null cdp url

* fix rust reader loop

* improvements

* improvements

* fixes
2026-03-12 12:01:40 -05:00
Chris Tate f2d4089284 auth docs (#730)
* add docs

* note

* format
2026-03-12 00:54:19 -05:00
Chris Tate 644a4f5b63 add scale factor to set viewport for retina screenshots (#691)
* device scale

* fix node.js daemon

* fix cargo fmt formatting for scale factor code

* fixes
2026-03-09 11:10:22 -05:00
Andrew ImmandChris Tate 59fa36b6e2 feat: Enable capture of profiling data (#290)
* feat: Enable capture of profiling data

Adding a new set of commands:
```
agent-browser profiler start

agent-browser profiler stop trace.json
```

With this, agents can start a profiling trace, perform a set of actions, and then extract the profiling data for analysis.

**Note:** I was originally going to call it `agent-browser profile` but I realized that might cause confusion with the `--profile` flag

CDP supports a couple commands for starting/stopping a trace.
When a trace is running, it emits events that need to be picked up.
We store these locally in the daemon until the trace is completed.
When the final event is received, we dump all of them into an output file.

That file can be loaded directly into chrome devtools or another analysis tool to visualize what happened during the agentic run.

Added some basic rust tests for parsing the commands (since they have some optional / required args)

TS daemon adds ~6 tests to make sure the profiling lifecycle (including saving the output file) works as intended

* add docs

* fixes

* fixes

---------

Co-authored-by: Chris Tate <chris@ctate.dev>
2026-02-17 23:11:11 -06:00
Chris Tate 76df589aea update docs (#493) 2026-02-17 21:37:41 -06:00
Chris Tate 0dc36f2cff Add --stdin flag for eval command (#348)
Adds --stdin flag to read JavaScript from stdin, enabling heredoc usage
for multiline scripts without shell escaping issues.
2026-02-02 20:29:43 -06:00
Chris Tate e52aa49706 Add skill-creator and improve agent-browser skill (#341)
* add skills-creator

* update skill

* better docs

* minor fixes
2026-02-02 19:18:34 -06:00
Yonatan f74924cd0c feat(skills): Add hierarchical structure with references and templates (#157)
* feat(skills): Add hierarchical structure with references and templates

Adds modular documentation and executable templates to the agent-browser skill
for better AI agent consumption and progressive disclosure.

## Added

### References (deep-dive documentation)
- `references/snapshot-refs.md` - Ref lifecycle, invalidation, troubleshooting
- `references/session-management.md` - Parallel sessions, state persistence
- `references/authentication.md` - Login flows, OAuth, 2FA patterns
- `references/video-recording.md` - Recording for debugging/docs
- `references/proxy-support.md` - Proxy configuration, geo-testing

### Templates (ready-to-use workflows)
- `templates/form-automation.sh` - Form filling with validation
- `templates/authenticated-session.sh` - Login once, reuse state
- `templates/capture-workflow.sh` - Content extraction with screenshots

## Modified
- `SKILL.md` - Added reference tables linking to new documentation

## Benefits
- Progressive disclosure: Load overview first, deep dives on demand
- Reduced context: Smaller chunks for better LLM token efficiency
- Ready workflows: Copy-paste templates for common patterns

* fix(templates): Make authenticated-session.sh runnable out-of-box

Addresses review feedback: login actions were commented but verification
wasn't, causing script to fail when run as-is.

New approach:
- DISCOVERY MODE runs first (shows form structure)
- LOGIN FLOW section is fully commented as a unit
- User runs once to see refs, then customizes

┌─────────────────────────────────────────────────────────────┐
│ LOGIN FORM STRUCTURE                                        │
├─────────────────────────────────────────────────────────────┤
│ @e1 [input type="email"]                                    │
│ @e2 [input type="password"]                                 │
│ @e3 [button] "Sign In"                                      │
└─────────────────────────────────────────────────────────────┘
2026-01-22 10:26:31 -06:00