Add iframe support for CLI interactions and snapshots (#869)
* Add iframe support for CLI interactions and snapshots This PR adds comprehensive iframe support to the agent browser CLI, allowing users to interact with elements inside iframes seamlessly. ## Problem Users couldn't interact with elements inside iframes via the command line. The existing `frame` command was non-functional as it set `active_frame_id` but no other code read this value. ## Changes Made ### Enhanced Frame Context Tracking - Added `frame_id` field to `RefEntry` to track which frame each element reference belongs to - Updated `RefMap::add` and related methods to accept and store frame context - Modified element resolution functions to use frame context from ref entries ### Improved Frame Command - Fixed the existing `frame` command to actually work by threading `active_frame_id` through snapshot operations - Added support for iframe element references (e.g., `frame @e2`) in addition to CSS selectors - Enhanced frame detection to work with both named frames and iframe elements ### Updated Snapshot Behavior - Modified `take_snapshot` to accept optional frame context parameter - Updated all snapshot call sites to pass appropriate frame context - Maintained backward compatibility while enabling frame-scoped operations ### Element Resolution Updates - Updated `resolve_element_center` and `resolve_element_object_id` to use frame context from ref entries - Modified `find_node_id_by_role_name` to support frame-specific element lookup - Ensured all interaction functions work correctly within iframe contexts ## Implementation Details - Frame context is now properly propagated through the entire element interaction pipeline - The `frame` command can accept both CSS selectors and element references - All existing functionality remains intact while adding iframe capabilities - Added `Iframe` to interactive roles for better element discovery Fixes #863 * docs: add iframe support documentation Document the new iframe capabilities across all documentation surfaces: - Auto-inlining of iframe content in snapshots - Direct interaction with iframe element refs - frame command support for element refs (@e3) - Scoped snapshots via frame switching * fix: pass active frame context to diff snapshots and fix nameless iframe lookup - handle_diff_snapshot now respects active_frame_id instead of always passing None, so diff snapshots work correctly inside iframes - Nameless/id-less iframes now fall back to src URL (or null) instead of the literal string 'frame' which never matched any frame in the tree * fix: resolve iframe frame ID via DOM.describeNode and reduce code duplication - handle_frame: Use DOM.describeNode + contentDocument.frameId to resolve iframe frame IDs directly, fixing failures for nameless iframes that lack name/id/src attributes - element.rs: Deduplicate add() by delegating to add_with_frame() - snapshot.rs: Guard against out-of-bounds insert_str when iframe marker is on the last line without a trailing newline --------- Co-authored-by: ctate <366502+ctate@users.noreply.github.com>
This commit is contained in:
@@ -251,6 +251,30 @@ agent-browser state clear myapp
|
||||
agent-browser state clean --older-than 7
|
||||
```
|
||||
|
||||
### Working with Iframes
|
||||
|
||||
Iframe content is automatically inlined in snapshots. Refs inside iframes carry frame context, so you can interact with them directly.
|
||||
|
||||
```bash
|
||||
agent-browser open https://example.com/checkout
|
||||
agent-browser snapshot -i
|
||||
# @e1 [heading] "Checkout"
|
||||
# @e2 [Iframe] "payment-frame"
|
||||
# @e3 [input] "Card number"
|
||||
# @e4 [input] "Expiry"
|
||||
# @e5 [button] "Pay"
|
||||
|
||||
# Interact directly — no frame switch needed
|
||||
agent-browser fill @e3 "4111111111111111"
|
||||
agent-browser fill @e4 "12/28"
|
||||
agent-browser click @e5
|
||||
|
||||
# To scope a snapshot to one iframe:
|
||||
agent-browser frame @e2
|
||||
agent-browser snapshot -i # Only iframe content
|
||||
agent-browser frame main # Return to main frame
|
||||
```
|
||||
|
||||
### Data Extraction
|
||||
|
||||
```bash
|
||||
|
||||
Reference in New Issue
Block a user