Add iframe support for CLI interactions and snapshots (#869)
* Add iframe support for CLI interactions and snapshots This PR adds comprehensive iframe support to the agent browser CLI, allowing users to interact with elements inside iframes seamlessly. ## Problem Users couldn't interact with elements inside iframes via the command line. The existing `frame` command was non-functional as it set `active_frame_id` but no other code read this value. ## Changes Made ### Enhanced Frame Context Tracking - Added `frame_id` field to `RefEntry` to track which frame each element reference belongs to - Updated `RefMap::add` and related methods to accept and store frame context - Modified element resolution functions to use frame context from ref entries ### Improved Frame Command - Fixed the existing `frame` command to actually work by threading `active_frame_id` through snapshot operations - Added support for iframe element references (e.g., `frame @e2`) in addition to CSS selectors - Enhanced frame detection to work with both named frames and iframe elements ### Updated Snapshot Behavior - Modified `take_snapshot` to accept optional frame context parameter - Updated all snapshot call sites to pass appropriate frame context - Maintained backward compatibility while enabling frame-scoped operations ### Element Resolution Updates - Updated `resolve_element_center` and `resolve_element_object_id` to use frame context from ref entries - Modified `find_node_id_by_role_name` to support frame-specific element lookup - Ensured all interaction functions work correctly within iframe contexts ## Implementation Details - Frame context is now properly propagated through the entire element interaction pipeline - The `frame` command can accept both CSS selectors and element references - All existing functionality remains intact while adding iframe capabilities - Added `Iframe` to interactive roles for better element discovery Fixes #863 * docs: add iframe support documentation Document the new iframe capabilities across all documentation surfaces: - Auto-inlining of iframe content in snapshots - Direct interaction with iframe element refs - frame command support for element refs (@e3) - Scoped snapshots via frame switching * fix: pass active frame context to diff snapshots and fix nameless iframe lookup - handle_diff_snapshot now respects active_frame_id instead of always passing None, so diff snapshots work correctly inside iframes - Nameless/id-less iframes now fall back to src URL (or null) instead of the literal string 'frame' which never matched any frame in the tree * fix: resolve iframe frame ID via DOM.describeNode and reduce code duplication - handle_frame: Use DOM.describeNode + contentDocument.frameId to resolve iframe frame IDs directly, fixing failures for nameless iframes that lack name/id/src attributes - element.rs: Deduplicate add() by delegating to add_with_frame() - snapshot.rs: Guard against out-of-bounds insert_str when iframe marker is on the last line without a trailing newline --------- Co-authored-by: ctate <366502+ctate@users.noreply.github.com>
This commit is contained in:
@@ -189,10 +189,35 @@ agent-browser tab new [url] # New tab
|
||||
agent-browser tab <n> # Switch to tab
|
||||
agent-browser tab close [n] # Close tab
|
||||
agent-browser window new # Open new browser window
|
||||
agent-browser frame <sel> # Switch to iframe
|
||||
agent-browser frame <sel> # Switch to iframe by CSS selector
|
||||
agent-browser frame @e3 # Switch to iframe by element ref
|
||||
agent-browser frame main # Back to main frame
|
||||
```
|
||||
|
||||
### Iframe support
|
||||
|
||||
Iframes are detected automatically during snapshots. `Iframe` nodes are resolved and their content is
|
||||
inlined beneath the iframe element in the snapshot output. Refs assigned to elements inside iframes carry
|
||||
frame context, so `click`, `fill`, and other interactions work without manually switching frames.
|
||||
|
||||
```bash
|
||||
agent-browser snapshot -i
|
||||
# @e3 [Iframe] "payment-frame"
|
||||
# @e4 [input] "Card number"
|
||||
# @e5 [button] "Pay"
|
||||
|
||||
# Interact directly using refs — no frame switch needed
|
||||
agent-browser fill @e4 "4111111111111111"
|
||||
agent-browser click @e5
|
||||
|
||||
# Or switch frame context for scoped snapshots
|
||||
agent-browser frame @e3
|
||||
agent-browser snapshot -i # Only elements inside that iframe
|
||||
agent-browser frame main # Return to main frame
|
||||
```
|
||||
|
||||
The `frame` command accepts element refs (`@e3`), CSS selectors (`"#my-iframe"`), or frame name/URL.
|
||||
|
||||
## Dialogs
|
||||
|
||||
```bash
|
||||
|
||||
@@ -102,6 +102,31 @@ agent-browser click @e2
|
||||
|
||||
Annotated screenshots also cache refs, so you can interact with elements immediately. This is useful when the text snapshot is insufficient -- unlabeled icons, canvas content, or visual layout verification.
|
||||
|
||||
## Iframes
|
||||
|
||||
Snapshots automatically detect and inline iframe content. Each `Iframe` node in the main frame is resolved and its child accessibility tree is included directly beneath it. Refs assigned to elements inside iframes carry frame context, so interactions work without switching frames first.
|
||||
|
||||
```bash
|
||||
agent-browser snapshot -i
|
||||
# @e1 [heading] "Checkout"
|
||||
# @e2 [Iframe] "payment-frame"
|
||||
# @e3 [input] "Card number"
|
||||
# @e4 [button] "Pay"
|
||||
|
||||
agent-browser fill @e3 "4111111111111111"
|
||||
agent-browser click @e4
|
||||
```
|
||||
|
||||
Only one level of iframe nesting is expanded. Cross-origin iframes that block accessibility tree access and empty iframes are silently omitted.
|
||||
|
||||
To scope a snapshot to a single iframe, switch into it first:
|
||||
|
||||
```bash
|
||||
agent-browser frame @e2
|
||||
agent-browser snapshot -i # Only elements inside that iframe
|
||||
agent-browser frame main # Return to main frame
|
||||
```
|
||||
|
||||
## Best practices
|
||||
|
||||
1. Use `-i` to reduce output to actionable elements
|
||||
|
||||
Reference in New Issue
Block a user