Commit Graph
4 Commits
Author SHA1 Message Date
Chris Tateandctate 60f3afcf61 Add iframe support for CLI interactions and snapshots (#869)
* Add iframe support for CLI interactions and snapshots

This PR adds comprehensive iframe support to the agent browser CLI, allowing users to interact with elements inside iframes seamlessly.

## Problem
Users couldn't interact with elements inside iframes via the command line. The existing `frame` command was non-functional as it set `active_frame_id` but no other code read this value.

## Changes Made

### Enhanced Frame Context Tracking
- Added `frame_id` field to `RefEntry` to track which frame each element reference belongs to
- Updated `RefMap::add` and related methods to accept and store frame context
- Modified element resolution functions to use frame context from ref entries

### Improved Frame Command
- Fixed the existing `frame` command to actually work by threading `active_frame_id` through snapshot operations
- Added support for iframe element references (e.g., `frame @e2`) in addition to CSS selectors
- Enhanced frame detection to work with both named frames and iframe elements

### Updated Snapshot Behavior
- Modified `take_snapshot` to accept optional frame context parameter
- Updated all snapshot call sites to pass appropriate frame context
- Maintained backward compatibility while enabling frame-scoped operations

### Element Resolution Updates
- Updated `resolve_element_center` and `resolve_element_object_id` to use frame context from ref entries
- Modified `find_node_id_by_role_name` to support frame-specific element lookup
- Ensured all interaction functions work correctly within iframe contexts

## Implementation Details
- Frame context is now properly propagated through the entire element interaction pipeline
- The `frame` command can accept both CSS selectors and element references
- All existing functionality remains intact while adding iframe capabilities
- Added `Iframe` to interactive roles for better element discovery

Fixes #863

* docs: add iframe support documentation

Document the new iframe capabilities across all documentation surfaces:
- Auto-inlining of iframe content in snapshots
- Direct interaction with iframe element refs
- frame command support for element refs (@e3)
- Scoped snapshots via frame switching

* fix: pass active frame context to diff snapshots and fix nameless iframe lookup

- handle_diff_snapshot now respects active_frame_id instead of always
  passing None, so diff snapshots work correctly inside iframes
- Nameless/id-less iframes now fall back to src URL (or null) instead of
  the literal string 'frame' which never matched any frame in the tree

* fix: resolve iframe frame ID via DOM.describeNode and reduce code duplication

- handle_frame: Use DOM.describeNode + contentDocument.frameId to resolve
  iframe frame IDs directly, fixing failures for nameless iframes that
  lack name/id/src attributes
- element.rs: Deduplicate add() by delegating to add_with_frame()
- snapshot.rs: Guard against out-of-bounds insert_str when iframe marker
  is on the last line without a trailing newline

---------

Co-authored-by: ctate <366502+ctate@users.noreply.github.com>
2026-03-17 10:51:17 -05:00
Chris Tate 76df589aea update docs (#493) 2026-02-17 21:37:41 -06:00
Chris Tate e52aa49706 Add skill-creator and improve agent-browser skill (#341)
* add skills-creator

* update skill

* better docs

* minor fixes
2026-02-02 19:18:34 -06:00
Yonatan f74924cd0c feat(skills): Add hierarchical structure with references and templates (#157)
* feat(skills): Add hierarchical structure with references and templates

Adds modular documentation and executable templates to the agent-browser skill
for better AI agent consumption and progressive disclosure.

## Added

### References (deep-dive documentation)
- `references/snapshot-refs.md` - Ref lifecycle, invalidation, troubleshooting
- `references/session-management.md` - Parallel sessions, state persistence
- `references/authentication.md` - Login flows, OAuth, 2FA patterns
- `references/video-recording.md` - Recording for debugging/docs
- `references/proxy-support.md` - Proxy configuration, geo-testing

### Templates (ready-to-use workflows)
- `templates/form-automation.sh` - Form filling with validation
- `templates/authenticated-session.sh` - Login once, reuse state
- `templates/capture-workflow.sh` - Content extraction with screenshots

## Modified
- `SKILL.md` - Added reference tables linking to new documentation

## Benefits
- Progressive disclosure: Load overview first, deep dives on demand
- Reduced context: Smaller chunks for better LLM token efficiency
- Ready workflows: Copy-paste templates for common patterns

* fix(templates): Make authenticated-session.sh runnable out-of-box

Addresses review feedback: login actions were commented but verification
wasn't, causing script to fail when run as-is.

New approach:
- DISCOVERY MODE runs first (shows form structure)
- LOGIN FLOW section is fully commented as a unit
- User runs once to see refs, then customizes

┌─────────────────────────────────────────────────────────────┐
│ LOGIN FORM STRUCTURE                                        │
├─────────────────────────────────────────────────────────────┤
│ @e1 [input type="email"]                                    │
│ @e2 [input type="password"]                                 │
│ @e3 [button] "Sign In"                                      │
└─────────────────────────────────────────────────────────────┘
2026-01-22 10:26:31 -06:00