oimwiodev/chrome-use
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Add iframe support for CLI interactions and snapshots (#869)

Browse Source 
* Add iframe support for CLI interactions and snapshots

This PR adds comprehensive iframe support to the agent browser CLI, allowing users to interact with elements inside iframes seamlessly.

## Problem
Users couldn't interact with elements inside iframes via the command line. The existing `frame` command was non-functional as it set `active_frame_id` but no other code read this value.

## Changes Made

### Enhanced Frame Context Tracking
- Added `frame_id` field to `RefEntry` to track which frame each element reference belongs to
- Updated `RefMap::add` and related methods to accept and store frame context
- Modified element resolution functions to use frame context from ref entries

### Improved Frame Command
- Fixed the existing `frame` command to actually work by threading `active_frame_id` through snapshot operations
- Added support for iframe element references (e.g., `frame @e2`) in addition to CSS selectors
- Enhanced frame detection to work with both named frames and iframe elements

### Updated Snapshot Behavior
- Modified `take_snapshot` to accept optional frame context parameter
- Updated all snapshot call sites to pass appropriate frame context
- Maintained backward compatibility while enabling frame-scoped operations

### Element Resolution Updates
- Updated `resolve_element_center` and `resolve_element_object_id` to use frame context from ref entries
- Modified `find_node_id_by_role_name` to support frame-specific element lookup
- Ensured all interaction functions work correctly within iframe contexts

## Implementation Details
- Frame context is now properly propagated through the entire element interaction pipeline
- The `frame` command can accept both CSS selectors and element references
- All existing functionality remains intact while adding iframe capabilities
- Added `Iframe` to interactive roles for better element discovery

Fixes #863

* docs: add iframe support documentation

Document the new iframe capabilities across all documentation surfaces:
- Auto-inlining of iframe content in snapshots
- Direct interaction with iframe element refs
- frame command support for element refs (@e3)
- Scoped snapshots via frame switching

* fix: pass active frame context to diff snapshots and fix nameless iframe lookup

- handle_diff_snapshot now respects active_frame_id instead of always
  passing None, so diff snapshots work correctly inside iframes
- Nameless/id-less iframes now fall back to src URL (or null) instead of
  the literal string 'frame' which never matched any frame in the tree

* fix: resolve iframe frame ID via DOM.describeNode and reduce code duplication

- handle_frame: Use DOM.describeNode + contentDocument.frameId to resolve
  iframe frame IDs directly, fixing failures for nameless iframes that
  lack name/id/src attributes
- element.rs: Deduplicate add() by delegating to add_with_frame()
- snapshot.rs: Guard against out-of-bounds insert_str when iframe marker
  is on the last line without a trailing newline

---------

Co-authored-by: ctate <366502+ctate@users.noreply.github.com>
This commit is contained in:
Chris Tate
2026-03-17 10:51:17 -05:00
committed by GitHub
co-authored by ctate
parent f51e955d99
commit 60f3afcf61
9 changed files with 370 additions and 27 deletions
Download Patch File Download Diff File Expand all files Collapse all files
cli/src/native/actions.rs
+88 -8
View File
@@ -1500,8 +1500,14 @@ async fn handle_snapshot(cmd: &Value, state: &mut DaemonState) -> Result<Value,
};
state.ref_map.clear();
let tree =
snapshot::take_snapshot(&mgr.client, &session_id, &options, &mut state.ref_map).await?;
let tree = snapshot::take_snapshot(
&mgr.client,
&session_id,
&options,
&mut state.ref_map,
state.active_frame_id.as_deref(),
)
.await?;
let url = mgr.get_url().await.unwrap_or_default();
@@ -1605,6 +1611,7 @@ async fn handle_screenshot(cmd: &Value, state: &mut DaemonState) -> Result<Value
..SnapshotOptions::default()
},
&mut state.ref_map,
state.active_frame_id.as_deref(),
)
.await?;
}
@@ -2395,8 +2402,14 @@ async fn handle_diff_snapshot(cmd: &Value, state: &mut DaemonState) -> Result<Va
selector,
..SnapshotOptions::default()
};
let current =
snapshot::take_snapshot(&mgr.client, &session_id, &options, &mut state.ref_map).await?;
let current = snapshot::take_snapshot(
&mgr.client,
&session_id,
&options,
&mut state.ref_map,
state.active_frame_id.as_deref(),
)
.await?;
let baseline = cmd.get("baseline").and_then(|v| v.as_str());
@@ -2441,13 +2454,15 @@ async fn handle_diff_url(cmd: &Value, state: &mut DaemonState) -> Result<Value,
let session_id = mgr.active_session_id()?.to_string();
let options = SnapshotOptions::default();
let snap1 =
snapshot::take_snapshot(&mgr.client, &session_id, &options, &mut state.ref_map).await?;
snapshot::take_snapshot(&mgr.client, &session_id, &options, &mut state.ref_map, None)
.await?;
// Navigate to URL2 and snapshot
mgr.navigate(url2, wait_until).await?;
state.ref_map.clear();
let snap2 =
snapshot::take_snapshot(&mgr.client, &session_id, &options, &mut state.ref_map).await?;
snapshot::take_snapshot(&mgr.client, &session_id, &options, &mut state.ref_map, None)
.await?;
let result = diff::diff_text(&snap1, &snap2);
Ok(json!({
@@ -3558,14 +3573,79 @@ async fn handle_frame(cmd: &Value, state: &mut DaemonState) -> Result<Value, Str
let frame_tree = &tree_result["frameTree"];
// If selector, resolve via JS to find the iframe's contentWindow
// If selector is a ref (@e1), resolve the iframe element from the ref map
if let Some(sel) = selector {
if let Some(ref_id) = super::element::parse_ref(sel) {
let entry = state
.ref_map
.get(&ref_id)
.ok_or_else(|| format!("Unknown ref: {}", ref_id))?;
let backend_node_id = entry
.backend_node_id
.ok_or_else(|| format!("Ref {} has no backend node id", ref_id))?;
// Use DOM.describeNode to resolve the child frame ID directly.
// This works reliably for all iframes, including those without
// name, id, or src attributes.
let describe: Value = mgr
.client
.send_command(
"DOM.describeNode",
Some(json!({ "backendNodeId": backend_node_id, "depth": 1 })),
Some(&session_id),
)
.await?;
// Verify this is an iframe/frame element
let node_name = describe
.get("node")
.and_then(|n| n.get("nodeName"))
.and_then(|v| v.as_str())
.unwrap_or("");
if node_name != "IFRAME" && node_name != "FRAME" {
return Err("Ref does not point to an iframe element".to_string());
}
// Try contentDocument.frameId first (standard for iframes)
let frame_id = describe
.get("node")
.and_then(|n| n.get("contentDocument"))
.and_then(|cd| cd.get("frameId"))
.and_then(|v| v.as_str())
// Fallback: the node itself may carry a frameId
.or_else(|| {
describe
.get("node")
.and_then(|n| n.get("frameId"))
.and_then(|v| v.as_str())
})
.ok_or("Could not resolve frame ID for iframe element")?;
let label = describe
.get("node")
.and_then(|n| n.get("attributes"))
.and_then(|a| a.as_array())
.and_then(|attrs| {
attrs
.iter()
.enumerate()
.find(|(_, v)| v.as_str() == Some("name"))
.and_then(|(i, _)| attrs.get(i + 1))
.and_then(|v| v.as_str())
})
.unwrap_or(&ref_id);
state.active_frame_id = Some(frame_id.to_string());
return Ok(json!({ "frame": label }));
}
// CSS selector path
let js = format!(
r#"(() => {{
const el = document.querySelector({});
if (!el) return null;
if (el.tagName === 'IFRAME' || el.tagName === 'FRAME') {{
return el.name || el.id || 'frame';
return el.name || el.id || el.src || null;
}}
return null;
}})()"#,