Add iframe support for CLI interactions and snapshots (#869)
* Add iframe support for CLI interactions and snapshots This PR adds comprehensive iframe support to the agent browser CLI, allowing users to interact with elements inside iframes seamlessly. ## Problem Users couldn't interact with elements inside iframes via the command line. The existing `frame` command was non-functional as it set `active_frame_id` but no other code read this value. ## Changes Made ### Enhanced Frame Context Tracking - Added `frame_id` field to `RefEntry` to track which frame each element reference belongs to - Updated `RefMap::add` and related methods to accept and store frame context - Modified element resolution functions to use frame context from ref entries ### Improved Frame Command - Fixed the existing `frame` command to actually work by threading `active_frame_id` through snapshot operations - Added support for iframe element references (e.g., `frame @e2`) in addition to CSS selectors - Enhanced frame detection to work with both named frames and iframe elements ### Updated Snapshot Behavior - Modified `take_snapshot` to accept optional frame context parameter - Updated all snapshot call sites to pass appropriate frame context - Maintained backward compatibility while enabling frame-scoped operations ### Element Resolution Updates - Updated `resolve_element_center` and `resolve_element_object_id` to use frame context from ref entries - Modified `find_node_id_by_role_name` to support frame-specific element lookup - Ensured all interaction functions work correctly within iframe contexts ## Implementation Details - Frame context is now properly propagated through the entire element interaction pipeline - The `frame` command can accept both CSS selectors and element references - All existing functionality remains intact while adding iframe capabilities - Added `Iframe` to interactive roles for better element discovery Fixes #863 * docs: add iframe support documentation Document the new iframe capabilities across all documentation surfaces: - Auto-inlining of iframe content in snapshots - Direct interaction with iframe element refs - frame command support for element refs (@e3) - Scoped snapshots via frame switching * fix: pass active frame context to diff snapshots and fix nameless iframe lookup - handle_diff_snapshot now respects active_frame_id instead of always passing None, so diff snapshots work correctly inside iframes - Nameless/id-less iframes now fall back to src URL (or null) instead of the literal string 'frame' which never matched any frame in the tree * fix: resolve iframe frame ID via DOM.describeNode and reduce code duplication - handle_frame: Use DOM.describeNode + contentDocument.frameId to resolve iframe frame IDs directly, fixing failures for nameless iframes that lack name/id/src attributes - element.rs: Deduplicate add() by delegating to add_with_frame() - snapshot.rs: Guard against out-of-bounds insert_str when iframe marker is on the last line without a trailing newline --------- Co-authored-by: ctate <366502+ctate@users.noreply.github.com>
This commit is contained in:
@@ -189,10 +189,35 @@ agent-browser tab new [url] # New tab
|
||||
agent-browser tab <n> # Switch to tab
|
||||
agent-browser tab close [n] # Close tab
|
||||
agent-browser window new # Open new browser window
|
||||
agent-browser frame <sel> # Switch to iframe
|
||||
agent-browser frame <sel> # Switch to iframe by CSS selector
|
||||
agent-browser frame @e3 # Switch to iframe by element ref
|
||||
agent-browser frame main # Back to main frame
|
||||
```
|
||||
|
||||
### Iframe support
|
||||
|
||||
Iframes are detected automatically during snapshots. `Iframe` nodes are resolved and their content is
|
||||
inlined beneath the iframe element in the snapshot output. Refs assigned to elements inside iframes carry
|
||||
frame context, so `click`, `fill`, and other interactions work without manually switching frames.
|
||||
|
||||
```bash
|
||||
agent-browser snapshot -i
|
||||
# @e3 [Iframe] "payment-frame"
|
||||
# @e4 [input] "Card number"
|
||||
# @e5 [button] "Pay"
|
||||
|
||||
# Interact directly using refs — no frame switch needed
|
||||
agent-browser fill @e4 "4111111111111111"
|
||||
agent-browser click @e5
|
||||
|
||||
# Or switch frame context for scoped snapshots
|
||||
agent-browser frame @e3
|
||||
agent-browser snapshot -i # Only elements inside that iframe
|
||||
agent-browser frame main # Return to main frame
|
||||
```
|
||||
|
||||
The `frame` command accepts element refs (`@e3`), CSS selectors (`"#my-iframe"`), or frame name/URL.
|
||||
|
||||
## Dialogs
|
||||
|
||||
```bash
|
||||
|
||||
Reference in New Issue
Block a user