Compare commits

...
14 Commits
Author SHA1 Message Date
leeguooooo fd2cdcde77 chore(release): 1.5.18 — issue-reporting guidance + clear data:-over-relay error
Release binaries / Build macOS ARM64 (push) Has been cancelled
Release binaries / Build macOS x64 (push) Has been cancelled
Release binaries / Build Linux ARM64 (push) Has been cancelled
Release binaries / Build Linux musl ARM64 (push) Has been cancelled
Release binaries / Build Linux musl x64 (push) Has been cancelled
Release binaries / Build Linux x64 (push) Has been cancelled
Release binaries / Build Windows x64 (push) Has been cancelled
Release binaries / Attach binaries to GitHub Release (push) Has been cancelled
2026-06-17 14:10:07 +09:00
leeguooooo 8e001e3d88 feat(dx): guide users to file issues; clear error for data: URLs over the relay
#5: the CLI never pointed users at the issue tracker. `--help` and `--version`
now print the issues URL (https://github.com/leeguooooo/chrome-use/issues), so
agents hitting a rough edge know where to report it (the skill already nudges).

#1: navigating to a `data:` URL over the extension relay fails with a cryptic
`net::ERR_ABORTED` on an about:blank tab (chrome.debugger/chrome.tabs can't drive
a top-frame data: navigation; it works fine under --launch). Detect that case and
explain it — use a real http(s)://file:// URL or --launch — instead of leaking
the raw code. (Surfaced while stress-testing 5 concurrent agents.)
2026-06-17 14:10:04 +09:00
leeguooooo eb2bc343a0 chore(release): 1.5.17 — complete multi-agent isolation (no foreign tab/pop-up adoption on click)
Release binaries / Build macOS ARM64 (push) Has been cancelled
Release binaries / Build macOS x64 (push) Has been cancelled
Release binaries / Build Linux ARM64 (push) Has been cancelled
Release binaries / Build Linux musl ARM64 (push) Has been cancelled
Release binaries / Build Linux musl x64 (push) Has been cancelled
Release binaries / Build Linux x64 (push) Has been cancelled
Release binaries / Build Windows x64 (push) Has been cancelled
Release binaries / Attach binaries to GitHub Release (push) Has been cancelled
2026-06-17 13:31:50 +09:00
leeguooooo d86c9c4be2 fix(relay): don't adopt foreign tabs/pop-ups on click — completes multi-agent isolation (#last hole)
After strict isolation (v1.5.16) a session's tracked set is only its OWN tabs, so
in `adopt_newly_opened` (run after every click to follow a pop-up) EVERY foreign
tab looks "new" relative to the session's `before` set and got adopted — a
click-heavy flow on a busy shared Chrome pulled other agents'/the user's tabs
(github, Lark, iphone-use) into the session mid-flow.

A pop-up the agent itself opened can't be told apart from a foreign tab over the
relay (the synthesized targetInfo carries no opener/window/group), so on the relay
`adopt_newly_opened` now adopts nothing: the agent drives only tabs it explicitly
created; pop-ups (OAuth/login windows) are the user's. Launched browsers (every
tab ours) still follow pop-ups. Verified live: a fresh click-heavy relay session
stays clean (only its own tabs), and 5 concurrent agents churning tabs show zero
cross-agent drift.
2026-06-17 13:31:49 +09:00
leeguooooo 32c25a6627 chore(release): 1.5.16 — strict multi-agent tab isolation on the relay
Release binaries / Build macOS ARM64 (push) Has been cancelled
Release binaries / Build macOS x64 (push) Has been cancelled
Release binaries / Build Linux ARM64 (push) Has been cancelled
Release binaries / Build Linux musl ARM64 (push) Has been cancelled
Release binaries / Build Linux musl x64 (push) Has been cancelled
Release binaries / Build Linux x64 (push) Has been cancelled
Release binaries / Build Windows x64 (push) Has been cancelled
Release binaries / Attach binaries to GitHub Release (push) Has been cancelled
2026-06-17 11:50:10 +09:00
leeguooooo a8ce3dd3f8 fix(relay): strict multi-agent isolation — a session owns only its own tabs
Several agents (and other tools opening tabs) share one real Chrome via the
relay. Previously every session adopted ALL tabs from Target.getTargets, so
another agent's tab churn polluted the list, dropped the tab being driven, and
drifted commands onto the wrong page (the W-8BEN tax tab vanished mid-flow when a
concurrent iphone-use agent opened tabs).

A tab group belongs to exactly one agent. On the relay a session now tracks and
drives ONLY the tabs it created (its own colored group) plus pop-ups its own
clicks open — it never adopts the user's or other agents' tabs:

- discover_and_attach (relay): create the session's own tab and pin it; do not
  adopt any existing foreign tab.
- resync_targets (relay): never adopt unknown targets; never prune the session's
  tabs on a single getTargets snapshot (multi-agent churn / cross-process-nav
  gaps routinely omit live tabs) — prune only after RELAY_PRUNE_MISSES
  consecutive absent snapshots (debounced), pinned active always protected.
- adopt_newly_opened: a tab that appears right after this session's action is a
  pop-up we opened — record it as owned.

Launched browsers (every tab ours) keep adopting all tabs. Adds debounced_prune_ids
+ unit tests for the churn tolerance.
2026-06-17 11:50:09 +09:00
leeguooooo 997373fd57 chore(release): 1.5.15 — trusted activation for in-iframe buttons (#39)
Release binaries / Build macOS ARM64 (push) Has been cancelled
Release binaries / Build macOS x64 (push) Has been cancelled
Release binaries / Build Linux ARM64 (push) Has been cancelled
Release binaries / Build Linux musl ARM64 (push) Has been cancelled
Release binaries / Build Linux musl x64 (push) Has been cancelled
Release binaries / Build Linux x64 (push) Has been cancelled
Release binaries / Build Windows x64 (push) Has been cancelled
Release binaries / Attach binaries to GitHub Release (push) Has been cancelled
2026-06-17 10:16:17 +09:00
leeguooooo 5a858af93f fix(iframe): trusted activation for in-iframe buttons — keyboard, not synthetic click (#39)
A DOM `.click()` is isTrusted:false, which security-sensitive embedded forms
reject — Google Payments' enabled `保存` button silently no-op'd, so a
cross-origin payment/checkout/KYC form could be read, scrolled, and typed into
but never submitted. A coordinate click can't help either: getBoxModel for a
sub-frame node returns frame-local coords that don't compose the iframe offset,
so it lands wrong (verified — the same-origin probe came back isTrusted:false
via the coordinate fallback).

Fix: click on an in-iframe ref now focuses the element in its own frame session
and dispatches a real Enter (Space for checkbox-like roles) on the page session.
Chrome routes the key to the focused element across frames (same mechanism as
`type --focused`), and Enter/Space on a focused button/link/checkbox fires a
trusted click. Non-activatable roles fall back to DOM .click().

Adds e2e_iframe_button_click_is_trusted (+ fixture): an in-iframe button records
event.isTrusted into its own text; the test asserts the ref-click delivers
isTrusted:true.
2026-06-17 10:16:15 +09:00
leeguooooo 58dc02bfdc chore(release): 1.5.14 — fix eval await regression (replMode) + default scroll; green CI (#36, #38)
Release binaries / Build macOS ARM64 (push) Has been cancelled
Release binaries / Build macOS x64 (push) Has been cancelled
Release binaries / Build Linux ARM64 (push) Has been cancelled
Release binaries / Build Linux musl ARM64 (push) Has been cancelled
Release binaries / Build Linux musl x64 (push) Has been cancelled
Release binaries / Build Linux x64 (push) Has been cancelled
Release binaries / Build Windows x64 (push) Has been cancelled
Release binaries / Attach binaries to GitHub Release (push) Has been cancelled
2026-06-17 02:34:55 +09:00
leeguooooo c47601bd7b fix(eval): replMode only for sync let/const decls, keep awaitPromise for async (#38)
replMode and awaitPromise are mutually exclusive in Chrome — under replMode a
returned promise serialises to {} instead of being awaited, which broke every
fetch/async eval (e2e_domain_filter, e2e_headers, e2e_react_tree all regressed).
Enable replMode only for synchronous scripts that declare a top-level let/const
(the #38 case); promise-returning scripts keep awaitPromise — restoring the
pre-#38 await behaviour while still fixing the let-redeclaration collision.
2026-06-17 02:08:11 +09:00
leeguooooo 0296bc7a88 fix(scroll): keep default scroll on window.scrollBy; wheel only for --at/--frame (#36)
The centered-wheel default no-op'd on some pages (headless e2e_hover_scroll_press
regressed). Restore window.scrollBy for plain page scroll; the coordinate wheel
stays opt-in via --at/--frame for cross-origin iframe content.
2026-06-17 02:01:23 +09:00
leeguooooo 32e203b908 style: cargo fmt (fixes the CI format-check failure) 2026-06-17 01:33:22 +09:00
leeguooooo fc51cd63ba chore(release): 1.5.13 — eval replMode (re-declarable let/const) + snapshot-first skill rule (#37, #38)
Release binaries / Build macOS ARM64 (push) Has been cancelled
Release binaries / Build macOS x64 (push) Has been cancelled
Release binaries / Build Linux ARM64 (push) Has been cancelled
Release binaries / Build Linux musl ARM64 (push) Has been cancelled
Release binaries / Build Linux musl x64 (push) Has been cancelled
Release binaries / Build Linux x64 (push) Has been cancelled
Release binaries / Build Windows x64 (push) Has been cancelled
Release binaries / Attach binaries to GitHub Release (push) Has been cancelled
2026-06-17 01:15:31 +09:00
leeguooooo f714c7920b fix(eval): replMode so successive evals can re-declare let/const; snapshot-first skill rule (#37, #38)
#38: `chrome-use eval` now runs with Runtime.evaluate replMode (like the DevTools
console) — top-level `let`/`const` no longer throw "already been declared" across
successive evals (independent `eval` steps in a `test` suite collided in the
page's shared lexical scope), and top-level await is allowed. Main-world and
completion-value semantics are unchanged.

#37: core skill gains a hard rule — snapshot-first, never screenshot+coordinates
to locate form fields/buttons; `snapshot -i` now pierces cross-origin iframes and
lists their elements by @ref; screenshots are for visual checks only, and a
full-page retina screenshot often exceeds an image reader's limits.
2026-06-17 01:15:31 +09:00
13 changed files with 615 additions and 110 deletions
+1 -1
View File
@@ -290,7 +290,7 @@ checksum = "613afe47fcd5fac7ccf1db93babcb082c5994d996f20b8b159f2ad1658eb5724"
[[package]] [[package]]
name = "chrome-use" name = "chrome-use"
version = "1.5.12" version = "1.5.18"
dependencies = [ dependencies = [
"aes", "aes",
"aes-gcm", "aes-gcm",
+1 -1
View File
@@ -1,6 +1,6 @@
[package] [package]
name = "chrome-use" name = "chrome-use"
version = "1.5.12" version = "1.5.18"
edition = "2021" edition = "2021"
description = "Fast browser automation CLI for AI agents" description = "Fast browser automation CLI for AI agents"
license = "Apache-2.0" license = "Apache-2.0"
+72 -16
View File
@@ -34,11 +34,50 @@ pub enum ParseError {
/// suggestions on an unknown command (issue #29). Not exhaustive — just the /// suggestions on an unknown command (issue #29). Not exhaustive — just the
/// common verbs plus a few known wrong-guesses mapped to the real command. /// common verbs plus a few known wrong-guesses mapped to the real command.
const KNOWN_COMMANDS: &[&str] = &[ const KNOWN_COMMANDS: &[&str] = &[
"open", "navigate", "click", "fill", "type", "press", "snapshot", "screenshot", "eval", "get", "open",
"text", "html", "frames", "find", "wait", "scroll", "hover", "select", "check", "uncheck", "navigate",
"tab", "tabs", "close", "back", "forward", "reload", "sessions", "status", "daemon", "doctor", "click",
"upgrade", "connect", "cookies", "mouse", "keyboard", "stream", "frame", "profiles", "title", "fill",
"url", "is", "drag", "dialog", "upload", "type",
"press",
"snapshot",
"screenshot",
"eval",
"get",
"text",
"html",
"frames",
"find",
"wait",
"scroll",
"hover",
"select",
"check",
"uncheck",
"tab",
"tabs",
"close",
"back",
"forward",
"reload",
"sessions",
"status",
"daemon",
"doctor",
"upgrade",
"connect",
"cookies",
"mouse",
"keyboard",
"stream",
"frame",
"profiles",
"title",
"url",
"is",
"drag",
"dialog",
"upload",
]; ];
/// Levenshtein distance, capped — small inputs only (command names). /// Levenshtein distance, capped — small inputs only (command names).
@@ -547,7 +586,9 @@ fn parse_command_inner(args: &[String], flags: &Flags) -> Result<Value, ParseErr
context: "type".to_string(), context: "type".to_string(),
usage: "type <selector> <text> (or: type --focused <text>) [--key-events]", usage: "type <selector> <text> (or: type --focused <text>) [--key-events]",
})?; })?;
Ok(json!({ "id": id, "action": "type", "selector": sel, "text": rest[1..].join(" "), "keyEvents": key_events })) Ok(
json!({ "id": id, "action": "type", "selector": sel, "text": rest[1..].join(" "), "keyEvents": key_events }),
)
} }
"pick" => { "pick" => {
// pick <selector|@ref> --option "<text>" — atomic combobox select: // pick <selector|@ref> --option "<text>" — atomic combobox select:
@@ -761,7 +802,8 @@ fn parse_command_inner(args: &[String], flags: &Flags) -> Result<Value, ParseErr
_ => { _ => {
return Err(ParseError::InvalidValue { return Err(ParseError::InvalidValue {
message: format!("scroll --at: invalid coordinate `{}`", val), message: format!("scroll --at: invalid coordinate `{}`", val),
usage: "scroll [direction] [amount] --at <x,y> (e.g. --at 640,400)", usage:
"scroll [direction] [amount] --at <x,y> (e.g. --at 640,400)",
}) })
} }
} }
@@ -970,17 +1012,21 @@ fn parse_command_inner(args: &[String], flags: &Flags) -> Result<Value, ParseErr
"--full" | "-f" => full_page = true, "--full" | "-f" => full_page = true,
// `--clip x,y,w,h` captures a pixel region (issue #34). // `--clip x,y,w,h` captures a pixel region (issue #34).
"--clip" => { "--clip" => {
let raw = rest.get(i + 1).ok_or_else(|| ParseError::MissingArguments { let raw = rest
context: "screenshot --clip".to_string(), .get(i + 1)
usage: "screenshot --clip <x,y,w,h> [path]", .ok_or_else(|| ParseError::MissingArguments {
})?; context: "screenshot --clip".to_string(),
usage: "screenshot --clip <x,y,w,h> [path]",
})?;
let nums: Vec<f64> = raw let nums: Vec<f64> = raw
.split(',') .split(',')
.filter_map(|n| n.trim().parse::<f64>().ok()) .filter_map(|n| n.trim().parse::<f64>().ok())
.collect(); .collect();
if nums.len() != 4 { if nums.len() != 4 {
return Err(ParseError::InvalidValue { return Err(ParseError::InvalidValue {
message: format!("--clip expects 'x,y,w,h' (4 numbers), got '{raw}'"), message: format!(
"--clip expects 'x,y,w,h' (4 numbers), got '{raw}'"
),
usage: "screenshot --clip <x,y,w,h> [path]", usage: "screenshot --clip <x,y,w,h> [path]",
}); });
} }
@@ -4128,7 +4174,11 @@ mod tests {
fn test_type_key_events() { fn test_type_key_events() {
// --key-events sends real keystrokes (for autocomplete/combobox) and must // --key-events sends real keystrokes (for autocomplete/combobox) and must
// not be swallowed into the typed text. // not be swallowed into the typed text.
let cmd = parse_command(&args("type #postal 201-0001 --key-events"), &default_flags()).unwrap(); let cmd = parse_command(
&args("type #postal 201-0001 --key-events"),
&default_flags(),
)
.unwrap();
assert_eq!(cmd["action"], "type"); assert_eq!(cmd["action"], "type");
assert_eq!(cmd["selector"], "#postal"); assert_eq!(cmd["selector"], "#postal");
assert_eq!(cmd["text"], "201-0001"); assert_eq!(cmd["text"], "201-0001");
@@ -4426,8 +4476,11 @@ mod tests {
#[test] #[test]
fn test_screenshot_clip() { fn test_screenshot_clip() {
// `--clip x,y,w,h` captures a pixel region (issue #34); the path still parses. // `--clip x,y,w,h` captures a pixel region (issue #34); the path still parses.
let cmd = parse_command(&args("screenshot --clip 10,20,200,40 out.png"), &default_flags()) let cmd = parse_command(
.unwrap(); &args("screenshot --clip 10,20,200,40 out.png"),
&default_flags(),
)
.unwrap();
assert_eq!(cmd["action"], "screenshot"); assert_eq!(cmd["action"], "screenshot");
assert_eq!(cmd["clip"]["x"], 10.0); assert_eq!(cmd["clip"]["x"], 10.0);
assert_eq!(cmd["clip"]["y"], 20.0); assert_eq!(cmd["clip"]["y"], 20.0);
@@ -5005,7 +5058,10 @@ mod tests {
assert_eq!(nearest_command("sesions").as_deref(), Some("sessions")); assert_eq!(nearest_command("sesions").as_deref(), Some("sessions"));
assert_eq!(nearest_command("session").as_deref(), Some("sessions")); assert_eq!(nearest_command("session").as_deref(), Some("sessions"));
assert_eq!(nearest_command("clik").as_deref(), Some("click")); assert_eq!(nearest_command("clik").as_deref(), Some("click"));
assert_eq!(nearest_command("screenshits").as_deref(), Some("screenshot")); assert_eq!(
nearest_command("screenshits").as_deref(),
Some("screenshot")
);
// Nonsense with no close match stays silent. // Nonsense with no close match stays silent.
assert_eq!(nearest_command("xyzzy"), None); assert_eq!(nearest_command("xyzzy"), None);
// The unknown-command error embeds the suggestion. // The unknown-command error embeds the suggestion.
+48 -29
View File
@@ -3244,8 +3244,14 @@ async fn handle_type(cmd: &Value, state: &mut DaemonState) -> Result<Value, Stri
.get("text") .get("text")
.and_then(|v| v.as_str()) .and_then(|v| v.as_str())
.ok_or("Missing 'text' parameter")?; .ok_or("Missing 'text' parameter")?;
interaction::type_text_into_active_context(&mgr.client, &session_id, text, None, key_events) interaction::type_text_into_active_context(
.await?; &mgr.client,
&session_id,
text,
None,
key_events,
)
.await?;
return Ok(json!({ "typed": text, "focused": true })); return Ok(json!({ "typed": text, "focused": true }));
} }
@@ -3493,27 +3499,40 @@ async fn handle_scroll(cmd: &Value, state: &mut DaemonState) -> Result<Value, St
return Ok(json!({ "scrolled": true, "via": "selector" })); return Ok(json!({ "scrolled": true, "via": "selector" }));
} }
// No selector: dispatch a real (isTrusted) wheel at a viewport coordinate. // `--at x,y` / `--frame n`: dispatch a real (isTrusted) wheel at a viewport
// This hits the compositor and scrolls whatever scroll container is under the // coordinate. This hits the compositor and scrolls whatever scroll container
// pointer — including cross-origin iframes that `window.scrollBy` on the top // is under the pointer — including cross-origin iframes that `window.scrollBy`
// document silently no-ops on (issue #36). The coordinate is, in priority: // on the top document silently no-ops on (issue #36).
// --at x,y → that exact pixel if cmd.get("at").is_some() || cmd.get("frame").is_some() {
// --frame n → the center of frame n from `chrome-use frames` let (x, y, via) = if let Some(at) = cmd.get("at").and_then(|v| v.as_array()) {
// default → the viewport center let x = at.first().and_then(|v| v.as_f64()).unwrap_or(0.0);
let (x, y, via) = if let Some(at) = cmd.get("at").and_then(|v| v.as_array()) { let y = at.get(1).and_then(|v| v.as_f64()).unwrap_or(0.0);
let x = at.first().and_then(|v| v.as_f64()).unwrap_or(0.0); (x, y, "at")
let y = at.get(1).and_then(|v| v.as_f64()).unwrap_or(0.0); } else {
(x, y, "at") let n = cmd.get("frame").and_then(|v| v.as_u64()).unwrap_or(0);
} else if let Some(n) = cmd.get("frame").and_then(|v| v.as_u64()) { let (x, y) = frame_center(mgr, &session_id, &state.iframe_sessions, n as usize).await?;
let (x, y) = frame_center(mgr, &session_id, &state.iframe_sessions, n as usize).await?; (x, y, "frame")
(x, y, "frame") };
} else { dispatch_wheel(&mgr.client, &session_id, x, y, dx, dy).await?;
let (x, y) = viewport_center(mgr, &session_id).await?; return Ok(json!({ "scrolled": true, "via": via, "at": [x, y] }));
(x, y, "center") }
};
dispatch_wheel(&mgr.client, &session_id, x, y, dx, dy).await?; // Default (no selector/at/frame): scroll the page with `window.scrollBy`. This
Ok(json!({ "scrolled": true, "via": via, "at": [x, y] })) // is the reliable path for ordinary page scrolling; a coordinate wheel at the
// viewport centre is NOT a dependable substitute (it no-ops on some pages,
// e.g. headless), so the wheel stays opt-in via `--at`/`--frame` for the
// cross-origin-iframe case (issue #36).
interaction::scroll(
&mgr.client,
&session_id,
&state.ref_map,
None,
dx,
dy,
&state.iframe_sessions,
)
.await?;
Ok(json!({ "scrolled": true, "via": "page" }))
} }
/// Viewport center in CSS pixels, used as the default wheel landing point for /// Viewport center in CSS pixels, used as the default wheel landing point for
@@ -3836,12 +3855,9 @@ async fn handle_gettext(cmd: &Value, state: &mut DaemonState) -> Result<Value, S
async fn handle_frames(_cmd: &Value, state: &mut DaemonState) -> Result<Value, String> { async fn handle_frames(_cmd: &Value, state: &mut DaemonState) -> Result<Value, String> {
let mgr = state.browser.as_ref().ok_or("Browser not launched")?; let mgr = state.browser.as_ref().ok_or("Browser not launched")?;
let session_id = mgr.active_session_id()?.to_string(); let session_id = mgr.active_session_id()?.to_string();
let frames = super::element::collect_all_frames_text( let frames =
&mgr.client, super::element::collect_all_frames_text(&mgr.client, &session_id, &state.iframe_sessions)
&session_id, .await?;
&state.iframe_sessions,
)
.await?;
let list: Vec<Value> = frames let list: Vec<Value> = frames
.iter() .iter()
.enumerate() .enumerate()
@@ -4337,7 +4353,10 @@ async fn handle_cf_status(_cmd: &Value, state: &mut DaemonState) -> Result<Value
let url = mgr.get_url().await.unwrap_or_default(); let url = mgr.get_url().await.unwrap_or_default();
// 1. Is the page a Cloudflare challenge right now? // 1. Is the page a Cloudflare challenge right now?
let probe_raw = mgr.evaluate(CF_CHALLENGE_JS, None).await.unwrap_or(Value::Null); let probe_raw = mgr
.evaluate(CF_CHALLENGE_JS, None)
.await
.unwrap_or(Value::Null);
let probe = parse_json_string(probe_raw, "cf challenge probe").unwrap_or(Value::Null); let probe = parse_json_string(probe_raw, "cf challenge probe").unwrap_or(Value::Null);
let challenged = probe let challenged = probe
.get("challenged") .get("challenged")
+203 -38
View File
@@ -235,6 +235,43 @@ fn prunable_target_ids(
.collect() .collect()
} }
/// Consecutive missing `getTargets` snapshots before an owned relay tab is
/// pruned. >1 so a single churning/partial snapshot (other agents opening/closing
/// tabs) or a brief cross-process-nav gap can't drop the tab the agent is driving.
const RELAY_PRUNE_MISSES: u32 = 3;
/// Debounced prune for the relay: target ids to drop, mutating per-target miss
/// counters. A tab in `live_ids` resets to 0; an absent (non-pinned) tab
/// increments and is pruned only at `RELAY_PRUNE_MISSES`. Counters for
/// no-longer-tracked targets are forgotten. Pure, so the multi-agent churn
/// tolerance is unit-testable without a live browser.
fn debounced_prune_ids(
pages: &[PageInfo],
live_ids: &HashSet<String>,
pinned: Option<&str>,
misses: &mut HashMap<String, u32>,
) -> Vec<String> {
let tracked: HashSet<&str> = pages.iter().map(|p| p.target_id.as_str()).collect();
misses.retain(|tid, _| tracked.contains(tid.as_str()));
let mut prune = Vec::new();
for p in pages {
let tid = p.target_id.as_str();
if live_ids.contains(tid) {
misses.remove(tid);
continue;
}
if pinned == Some(tid) {
continue;
}
let c = misses.entry(p.target_id.clone()).or_insert(0);
*c += 1;
if *c >= RELAY_PRUNE_MISSES {
prune.push(p.target_id.clone());
}
}
prune
}
/// Whether the resolved active page is a tab the session created (its target_id /// Whether the resolved active page is a tab the session created (its target_id
/// is in `created_targets`). Pure core of [`BrowserManager::active_is_session_owned`] /// is in `created_targets`). Pure core of [`BrowserManager::active_is_session_owned`]
/// so the relay no-hijack rule is unit-testable without a live browser. /// so the relay no-hijack rule is unit-testable without a live browser.
@@ -464,6 +501,14 @@ pub struct BrowserManager {
/// the session's commands onto the wrong page — the wrong-origin-fetch hazard /// the session's commands onto the wrong page — the wrong-origin-fetch hazard
/// in the dogfood reports. Falls back to the index if the pinned tab is gone. /// in the dogfood reports. Falls back to the index if the pinned tab is gone.
active_target_id: Option<String>, active_target_id: Option<String>,
/// Per-target count of CONSECUTIVE `resync_targets` snapshots in which an
/// owned tab was missing from `Target.getTargets`. Over the relay a single
/// snapshot routinely omits live tabs (multi-agent churn, a cross-process nav
/// briefly dropping the target), so we must not prune on one miss — that lost
/// the tab the agent was driving. A tab is removed only after it's been absent
/// for `RELAY_PRUNE_MISSES` consecutive snapshots; any snapshot that includes
/// it resets the counter. Keyed by stable target_id.
relay_target_misses: HashMap<String, u32>,
next_tab_id: u32, next_tab_id: u32,
/// Whether to enable the CDP `Runtime` domain (console / error / exception capture). /// Whether to enable the CDP `Runtime` domain (console / error / exception capture).
/// OFF by default for stealth: a live `Runtime.enable` is a detectable CDP signal /// OFF by default for stealth: a live `Runtime.enable` is a detectable CDP signal
@@ -579,7 +624,10 @@ impl BrowserManager {
crate::connect::log_connect_mode( crate::connect::log_connect_mode(
&ws_url, &ws_url,
true, true,
DAEMON_SESSION.get().map(String::as_str).unwrap_or("default"), DAEMON_SESSION
.get()
.map(String::as_str)
.unwrap_or("default"),
); );
let manager = if engine == "lightpanda" { let manager = if engine == "lightpanda" {
initialize_lightpanda_manager(ws_url, process).await? initialize_lightpanda_manager(ws_url, process).await?
@@ -597,6 +645,7 @@ impl BrowserManager {
visited_origins: HashSet::new(), visited_origins: HashSet::new(),
created_targets: HashSet::new(), created_targets: HashSet::new(),
active_target_id: None, active_target_id: None,
relay_target_misses: HashMap::new(),
next_tab_id: 1, next_tab_id: 1,
capture_console: console_capture_enabled(), capture_console: console_capture_enabled(),
}; };
@@ -681,7 +730,10 @@ impl BrowserManager {
crate::connect::log_connect_mode( crate::connect::log_connect_mode(
&ws_url, &ws_url,
false, false,
DAEMON_SESSION.get().map(String::as_str).unwrap_or("default"), DAEMON_SESSION
.get()
.map(String::as_str)
.unwrap_or("default"),
); );
let client = Arc::new(CdpClient::connect_with_headers(&ws_url, headers).await?); let client = Arc::new(CdpClient::connect_with_headers(&ws_url, headers).await?);
let mut manager = Self { let mut manager = Self {
@@ -696,6 +748,7 @@ impl BrowserManager {
visited_origins: HashSet::new(), visited_origins: HashSet::new(),
created_targets: HashSet::new(), created_targets: HashSet::new(),
active_target_id: None, active_target_id: None,
relay_target_misses: HashMap::new(),
next_tab_id: 1, next_tab_id: 1,
capture_console: console_capture_enabled(), capture_console: console_capture_enabled(),
}; };
@@ -817,7 +870,19 @@ impl BrowserManager {
self.active_page_index = 0; self.active_page_index = 0;
self.pin_active_target(); self.pin_active_target();
self.enable_domains(&attach_result.session_id).await?; self.enable_domains(&attach_result.session_id).await?;
} else if self.agent_group().is_some() {
// STRICT MULTI-AGENT ISOLATION (relay / the user's real Chrome).
// `page_targets` here are the USER's and OTHER agents' tabs. A tab
// group belongs to exactly ONE agent, so this session must NOT adopt
// any of them — it tracks ONLY tabs it creates (its own colored group)
// plus popups it opens. Adopting foreign tabs is precisely what let
// another concurrent agent's tab churn drop the tab we were driving and
// drift eval/click onto the wrong page (multi-agent failure). Open our
// own dedicated background tab in the session's group and pin it; the
// user's / other agents' tabs stay invisible to us.
self.tab_new(None, None).await?;
} else { } else {
// A browser WE launched: every tab is ours, so adopt them all.
for target in &page_targets { for target in &page_targets {
let attach_result: AttachToTargetResult = self let attach_result: AttachToTargetResult = self
.client .client
@@ -843,24 +908,10 @@ impl BrowserManager {
target_type: target.target_type.clone(), target_type: target.target_type.clone(),
}); });
} }
self.active_page_index = 0;
if self.agent_group().is_some() { self.pin_active_target();
// Relay: the adopted tabs above are the USER's, in their real let session_id = self.pages[0].session_id.clone();
// Chrome. NEVER make one of them the agent's working tab — that is self.enable_domains(&session_id).await?;
// how commands drifted onto whatever page the user was viewing
// between steps (eval/click/get landed on the user's foreground
// tab; #35). Open our own dedicated background tab in the session's
// group and pin THAT as active. The user's tabs stay adopted (so
// `tab list` / explicit `tab switch` can reach them) but are never
// auto-selected — the agent only ever drives a tab it owns.
self.tab_new(None, None).await?;
} else {
// A browser we launched: every tab is ours, so the first is fine.
self.active_page_index = 0;
self.pin_active_target();
let session_id = self.pages[0].session_id.clone();
self.enable_domains(&session_id).await?;
}
} }
Ok(()) Ok(())
@@ -1065,6 +1116,17 @@ impl BrowserManager {
}; };
if let Some(ref error_text) = nav_result.error_text { if let Some(ref error_text) = nav_result.error_text {
// `data:` URLs abort over the extension relay: chrome.debugger /
// chrome.tabs can't drive a top-frame data: navigation, so it comes
// back net::ERR_ABORTED on an about:blank tab. Explain it instead of
// leaking the cryptic code (data: works fine under `--launch`).
if url.starts_with("data:") && error_text.contains("ERR_ABORTED") {
return Err(format!(
"Navigation failed: {error_text}. Chrome blocks top-frame `data:` URLs over \
the extension relay use a real http(s):// or file:// URL, or run with \
`--launch` (where data: URLs work)."
));
}
return Err(format!("Navigation failed: {}", error_text)); return Err(format!("Navigation failed: {}", error_text));
} }
@@ -1202,15 +1264,31 @@ impl BrowserManager {
pub async fn evaluate(&self, script: &str, _args: Option<Value>) -> Result<Value, String> { pub async fn evaluate(&self, script: &str, _args: Option<Value>) -> Result<Value, String> {
let session_id = self.active_session_id()?.to_string(); let session_id = self.active_session_id()?.to_string();
// `replMode: true` lets successive `eval`s re-declare top-level
// `let`/`const` instead of throwing "Identifier 'x' has already been
// declared" (issue #38 — independent `eval` steps in a test suite collided
// in the page's shared lexical scope). BUT replMode and `awaitPromise` are
// mutually exclusive in Chrome: under replMode a returned promise is NOT
// awaited (it serialises to `{}`), which breaks `fetch(...).then(...)` and
// every other async eval. So enable replMode ONLY for synchronous scripts
// that declare a top-level `let`/`const`; promise-returning scripts keep
// `awaitPromise` (no replMode) — exactly the pre-#38 behaviour.
let mentions_async = script.contains("await")
|| script.contains(".then(")
|| script.contains("fetch(")
|| script.contains("Promise");
let declares = script.contains("let ") || script.contains("const ");
let repl_mode = declares && !mentions_async;
let result: EvaluateResult = self let result: EvaluateResult = self
.client .client
.send_command_typed( .send_command_typed(
"Runtime.evaluate", "Runtime.evaluate",
&EvaluateParams { &json!({
expression: script.to_string(), "expression": script,
return_by_value: Some(true), "returnByValue": true,
await_promise: Some(true), "awaitPromise": !repl_mode,
}, "replMode": repl_mode,
}),
Some(&session_id), Some(&session_id),
) )
.await?; .await?;
@@ -1493,6 +1571,18 @@ impl BrowserManager {
/// the active tab, per #7/#8.1); the caller surfaces it so the agent knows a /// the active tab, per #7/#8.1); the caller surfaces it so the agent knows a
/// tab opened instead of seeing the old page (issue #24-A). /// tab opened instead of seeing the old page (issue #24-A).
pub async fn adopt_newly_opened(&mut self, before: &HashSet<String>) -> Option<PageInfo> { pub async fn adopt_newly_opened(&mut self, before: &HashSet<String>) -> Option<PageInfo> {
// STRICT MULTI-AGENT ISOLATION: on the relay this session's `before` set is
// only its OWN tabs, so EVERY foreign tab (the user's, other agents') looks
// "new" relative to it and would be adopted here — exactly the leak where a
// concurrent agent's tabs (github/Lark/iphone-use) showed up in this
// session mid-flow. A tab the agent itself opened (a pop-up) can't be
// distinguished from a foreign tab over the relay (no opener/window/group
// in the synthesized targetInfo), so don't adopt anything: the agent drives
// only tabs it explicitly created, and pop-ups (e.g. an OAuth/login window)
// are the user's. A launched browser (every tab ours) still follows pop-ups.
if self.agent_group().is_some() {
return None;
}
let result: GetTargetsResult = self let result: GetTargetsResult = self
.client .client
.send_command_typed("Target.getTargets", &json!({}), None) .send_command_typed("Target.getTargets", &json!({}), None)
@@ -1535,6 +1625,11 @@ impl BrowserManager {
title: sanitize_title(&target.title), title: sanitize_title(&target.title),
target_type: target.target_type.clone(), target_type: target.target_type.clone(),
}; };
// A tab that appeared right after THIS session's action (a click that
// opened a popup/new tab) is ours — record it as owned so it's tracked,
// protected from churn-pruning, and cleaned up on close, consistent with
// strict multi-agent isolation (we only ever own tabs we created/opened).
self.created_targets.insert(target.target_id.clone());
self.add_background_page(page.clone()); self.add_background_page(page.clone());
let _ = self.enable_domains(&attach.session_id).await; let _ = self.enable_domains(&attach.session_id).await;
if opened.is_none() { if opened.is_none() {
@@ -1562,13 +1657,19 @@ impl BrowserManager {
.filter(should_track_target) .filter(should_track_target)
.collect(); .collect();
let live_ids: HashSet<String> = live.iter().map(|t| t.target_id.clone()).collect(); let live_ids: HashSet<String> = live.iter().map(|t| t.target_id.clone()).collect();
let on_relay = self.agent_group().is_some();
for target in &live { for target in &live {
if self.update_page_target_info(target) { if self.update_page_target_info(target) {
continue; continue;
} }
// A target this session hasn't tracked yet — attach and add it in the // STRICT MULTI-AGENT ISOLATION: on the relay (the user's real Chrome,
// background so it's listable/adoptable without stealing the active tab. // shared with other agents), NEVER adopt a tab this session didn't
// create — it belongs to the user or another agent's group. Only a
// browser we launched (every tab ours) adopts unknown targets.
if on_relay {
continue;
}
let attach_result: AttachToTargetResult = match self let attach_result: AttachToTargetResult = match self
.client .client
.send_command_typed( .send_command_typed(
@@ -1599,12 +1700,26 @@ impl BrowserManager {
let _ = self.enable_domains(&attach_result.session_id).await; let _ = self.enable_domains(&attach_result.session_id).await;
} }
// Drop tabs that no longer exist so `tab list` doesn't show phantom rows — // Prune tabs that are gone. On a LAUNCHED browser a missing target really
// but never prune the explicitly-pinned active target on a transient // is closed, so prune immediately. On the RELAY a single `getTargets`
// getTargets snapshot (issue #31; see `prunable_target_ids`). // snapshot routinely omits live tabs (multi-agent churn, a brief
let gone = prunable_target_ids(&self.pages, &live_ids, self.active_target_id.as_deref()); // cross-process-nav gap) — dropping the tab we're driving on one bad
for tid in gone { // snapshot is the failure we're fixing — so prune only after the tab has
self.remove_page_by_target_id(&tid); // been absent for several CONSECUTIVE snapshots (debounced). The pinned
// active target is protected either way (issue #31).
let gone = if on_relay {
debounced_prune_ids(
&self.pages,
&live_ids,
self.active_target_id.as_deref(),
&mut self.relay_target_misses,
)
} else {
prunable_target_ids(&self.pages, &live_ids, self.active_target_id.as_deref())
};
for tid in &gone {
self.relay_target_misses.remove(tid);
self.remove_page_by_target_id(tid);
} }
// Refresh url/title from each live tab. The relay only stamps target_info // Refresh url/title from each live tab. The relay only stamps target_info
@@ -2514,6 +2629,7 @@ async fn initialize_lightpanda_manager(
visited_origins: HashSet::new(), visited_origins: HashSet::new(),
created_targets: HashSet::new(), created_targets: HashSet::new(),
active_target_id: None, active_target_id: None,
relay_target_misses: HashMap::new(),
next_tab_id: 1, next_tab_id: 1,
capture_console: console_capture_enabled(), capture_console: console_capture_enabled(),
}; };
@@ -2807,7 +2923,9 @@ mod tests {
its tab is gone (closed, navigated across processes, or lost after an extension \ its tab is gone (closed, navigated across processes, or lost after an extension \
restart). Re-attach by re-opening your target URL before retrying." restart). Re-attach by re-opening your target URL before retrying."
)); ));
assert!(is_stale_target_error("unknown sessionId cb-tab-7 for Page.navigate")); assert!(is_stale_target_error(
"unknown sessionId cb-tab-7 for Page.navigate"
));
assert!(is_stale_target_error("no attached tab for Page.navigate")); assert!(is_stale_target_error("no attached tab for Page.navigate"));
} }
@@ -2815,8 +2933,12 @@ mod tests {
fn stale_target_error_ignores_unrelated_failures() { fn stale_target_error_ignores_unrelated_failures() {
// A genuine navigation failure (bad URL, DNS, blocked) must NOT trigger // A genuine navigation failure (bad URL, DNS, blocked) must NOT trigger
// the open-a-fresh-tab recovery — that would mask the real error. // the open-a-fresh-tab recovery — that would mask the real error.
assert!(!is_stale_target_error("Navigation failed: net::ERR_NAME_NOT_RESOLVED")); assert!(!is_stale_target_error(
assert!(!is_stale_target_error("CDP command timed out: Page.navigate")); "Navigation failed: net::ERR_NAME_NOT_RESOLVED"
));
assert!(!is_stale_target_error(
"CDP command timed out: Page.navigate"
));
} }
fn page(target_id: &str) -> PageInfo { fn page(target_id: &str) -> PageInfo {
@@ -2923,7 +3045,10 @@ mod tests {
let dirty = "\u{200d}\u{2061}\u{200d}\u{2063}\u{200b}\u{2062}\u{feff}GitHub"; let dirty = "\u{200d}\u{2061}\u{200d}\u{2063}\u{200b}\u{2062}\u{feff}GitHub";
assert_eq!(sanitize_title(dirty), "GitHub"); assert_eq!(sanitize_title(dirty), "GitHub");
// Clean titles (incl. CJK + normal punctuation) pass through untouched. // Clean titles (incl. CJK + normal punctuation) pass through untouched.
assert_eq!(sanitize_title("購入手続きへ - メルカリ"), "購入手続きへ - メルカリ"); assert_eq!(
sanitize_title("購入手続きへ - メルカリ"),
"購入手続きへ - メルカリ"
);
assert_eq!(sanitize_title(" Hello World "), "Hello World"); assert_eq!(sanitize_title(" Hello World "), "Hello World");
// Emoji and real content survive; only the invisibles are dropped. // Emoji and real content survive; only the invisibles are dropped.
assert_eq!(sanitize_title("✓ Done\u{200b}"), "✓ Done"); assert_eq!(sanitize_title("✓ Done\u{200b}"), "✓ Done");
@@ -2955,7 +3080,47 @@ mod tests {
// A pinned target that IS in the live set is simply not prunable anyway. // A pinned target that IS in the live set is simply not prunable anyway.
let mut live2 = HashSet::new(); let mut live2 = HashSet::new();
live2.insert("A".to_string()); live2.insert("A".to_string());
assert_eq!(prunable_target_ids(&pages, &live2, Some("A")), vec!["B".to_string()]); assert_eq!(
prunable_target_ids(&pages, &live2, Some("A")),
vec!["B".to_string()]
);
}
#[test]
fn debounced_prune_tolerates_transient_churn() {
// Multi-agent churn: a single getTargets snapshot omits our owned tab "B"
// (another agent opened/closed tabs). It must NOT be pruned on one miss.
let pages = vec![page("A"), page("B")];
let mut misses = HashMap::new();
let empty: HashSet<String> = HashSet::new();
// Misses 1 and 2: B absent but under threshold → not pruned.
assert!(debounced_prune_ids(&pages, &empty, Some("A"), &mut misses).is_empty());
assert!(debounced_prune_ids(&pages, &empty, Some("A"), &mut misses).is_empty());
// Miss 3 (== RELAY_PRUNE_MISSES): genuinely gone → pruned.
assert_eq!(
debounced_prune_ids(&pages, &empty, Some("A"), &mut misses),
vec!["B".to_string()]
);
}
#[test]
fn debounced_prune_resets_on_reappearance_and_protects_pin() {
let pages = vec![page("A"), page("B")];
let mut misses = HashMap::new();
let empty: HashSet<String> = HashSet::new();
let mut live_b: HashSet<String> = HashSet::new();
live_b.insert("B".to_string());
// Two misses for B, then it reappears → counter resets, so it survives
// indefinitely under intermittent churn.
debounced_prune_ids(&pages, &empty, Some("A"), &mut misses);
debounced_prune_ids(&pages, &empty, Some("A"), &mut misses);
assert!(debounced_prune_ids(&pages, &live_b, Some("A"), &mut misses).is_empty());
assert!(debounced_prune_ids(&pages, &empty, Some("A"), &mut misses).is_empty()); // back to miss 1
// The pinned active "A" is never pruned no matter how many misses.
for _ in 0..5 {
let gone = debounced_prune_ids(&pages, &empty, Some("A"), &mut misses);
assert!(!gone.contains(&"A".to_string()));
}
} }
#[test] #[test]
+71
View File
@@ -35,6 +35,7 @@ fn native_test_fixture_html(name: &str) -> &'static str {
"html5_drag_probe" => include_str!("test_fixtures/html5_drag_probe.html"), "html5_drag_probe" => include_str!("test_fixtures/html5_drag_probe.html"),
"pointer_capture_probe" => include_str!("test_fixtures/pointer_capture_probe.html"), "pointer_capture_probe" => include_str!("test_fixtures/pointer_capture_probe.html"),
"upload_probe" => include_str!("test_fixtures/upload_probe.html"), "upload_probe" => include_str!("test_fixtures/upload_probe.html"),
"iframe_button_probe" => include_str!("test_fixtures/iframe_button_probe.html"),
_ => panic!("Unknown native test fixture: {}", name), _ => panic!("Unknown native test fixture: {}", name),
} }
} }
@@ -573,6 +574,76 @@ async fn e2e_snapshot_and_click_ref() {
assert_success(&resp); assert_success(&resp);
} }
/// Clicking a button INSIDE an iframe by `@ref` must deliver a TRUSTED activation
/// (`event.isTrusted === true`), not a synthetic DOM `.click()`. Security-sensitive
/// embedded forms (Google Payments' `保存`) reject `isTrusted:false` clicks, so an
/// enabled submit button silently no-op'd (issue #39). The fix routes iframe-ref
/// clicks to a real `Input.dispatchMouseEvent` on the element's own frame session.
/// The fixture's iframe button writes `clicked:<isTrusted>` into its own text on
/// click, which the cross-frame snapshot reads back.
#[tokio::test]
#[ignore]
async fn e2e_iframe_button_click_is_trusted() {
let mut state = DaemonState::new();
let resp = execute_command(
&json!({ "id": "1", "action": "launch", "headless": true }),
&mut state,
)
.await;
assert_success(&resp);
let resp = execute_command(
&json!({ "id": "2", "action": "navigate", "url": native_test_fixture_url("iframe_button_probe") }),
&mut state,
)
.await;
assert_success(&resp);
// Snapshot (interactive) — the button lives in the iframe and must appear with
// a ref; that ref carries the frame_id so the click resolves into the frame.
let resp = execute_command(
&json!({ "id": "3", "action": "snapshot", "interactive": true }),
&mut state,
)
.await;
assert_success(&resp);
let snapshot = get_data(&resp)["snapshot"].as_str().unwrap_or("");
let ref_id = snapshot
.lines()
.find(|l| l.contains("button \"save\""))
.and_then(|l| l.split("ref=").nth(1))
.map(|r| r.trim_end_matches(']').trim())
.unwrap_or_else(|| panic!("iframe button not found in snapshot:\n{snapshot}"));
// Click it by ref.
let resp = execute_command(
&json!({ "id": "4", "action": "click", "selector": ref_id }),
&mut state,
)
.await;
assert_success(&resp);
tokio::time::sleep(tokio::time::Duration::from_millis(300)).await;
// The button rewrote its own text with the click's isTrusted flag; read it
// back across frames.
let resp = execute_command(
&json!({ "id": "5", "action": "snapshot", "interactive": true }),
&mut state,
)
.await;
assert_success(&resp);
let after = get_data(&resp)["snapshot"].as_str().unwrap_or("");
assert!(
after.contains("clicked:true"),
"iframe button click must be trusted (isTrusted:true); snapshot:\n{after}"
);
let resp = execute_command(&json!({ "id": "99", "action": "close" }), &mut state).await;
assert_success(&resp);
}
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
// Screenshot // Screenshot
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
+7 -2
View File
@@ -1028,7 +1028,9 @@ async fn eval_text_in_frame(client: &CdpClient, session_id: &str, frame_id: &str
.await .await
.ok() .ok()
.and_then(|v| v.get("executionContextId").and_then(|c| c.as_i64())); .and_then(|v| v.get("executionContextId").and_then(|c| c.as_i64()));
let Some(ctx_id) = ctx else { return String::new() }; let Some(ctx_id) = ctx else {
return String::new();
};
let res = client let res = client
.send_command( .send_command(
"Runtime.evaluate", "Runtime.evaluate",
@@ -1099,7 +1101,10 @@ pub async fn collect_all_frames_text(
let (kind, text) = if is_top { let (kind, text) = if is_top {
("top", eval_text_default(client, top_session).await) ("top", eval_text_default(client, top_session).await)
} else { } else {
("inline", eval_text_in_frame(client, top_session, &fid).await) (
"inline",
eval_text_in_frame(client, top_session, &fid).await,
)
}; };
out.push(FrameText { out.push(FrameText {
frame_id: fid, frame_id: fid,
+110 -12
View File
@@ -56,16 +56,35 @@ pub async fn click(
.await; .await;
} }
// Over the extension relay we drive the user's real, in-use Chrome, where a // An element INSIDE an iframe needs a TRUSTED activation: a DOM `.click()` is
// coordinate `Input.dispatchMouseEvent` is NOT reliably confined to our target // `isTrusted:false`, which security-sensitive embedded forms reject — Google
// tab — it can be delivered to whatever tab is in the foreground, and an OOPIF // Payments' enabled `保存` button silently no-ops on a synthetic click (issue
// element's box can't be mapped to a top-viewport point at all. This twice // #39). A coordinate `Input.dispatchMouseEvent` can't help either: `getBoxModel`
// opened an unrelated tab on the user's busy Chrome (issues #31/#36). So on the // for a sub-frame node returns frame-local coordinates that don't compose the
// relay, never use coordinates for a normal left click: DOM-dispatch invokes // iframe's offset, so the click lands in the wrong place. The frame-agnostic
// the element's click in its own (frame) session, always hitting the right // trusted path is keyboard activation — focus the element in its own frame, then
// element in the right tab. Double/right clicks still need true pointer // dispatch a real Enter on the page session; Chrome routes the key to the
// semantics, and `coord` mode is an explicit opt-out. // focused element regardless of frame (same as `type --focused`), and Enter on a
if mode != "coord" && button == "left" && click_count == 1 && prefer_dom_dispatch(ref_map, selector_or_ref) { // focused button/link fires a trusted `click`. `coord` mode opts out.
let in_iframe = ref_map.ref_is_in_iframe(selector_or_ref);
if mode != "coord" && button == "left" && click_count == 1 && in_iframe {
return dom_activate(
client,
session_id,
ref_map,
selector_or_ref,
iframe_sessions,
)
.await;
}
// On the relay (the user's real Chrome) a TOP-document coordinate click used to
// drift onto the foreground tab; that root cause is fixed (#5: the agent drives
// its own pinned tab), but DOM-dispatch stays the conservative default here.
if mode != "coord"
&& button == "left"
&& click_count == 1
&& crate::connect::relay_url().is_some()
{
return dom_click( return dom_click(
client, client,
session_id, session_id,
@@ -266,6 +285,71 @@ async fn dom_click(
Ok(()) Ok(())
} }
/// Trusted activation of an element inside an iframe (issue #39). Focuses the
/// element in its own frame session, then dispatches a real Enter/Space on the
/// page session — Chrome routes the key to the focused element across frames, and
/// Enter/Space on a focused button/link/checkbox fires a `click` with
/// `isTrusted: true`, which security-sensitive embedded forms (Google Payments
/// `保存`) require. Non-activatable roles (a `div[onclick]`) can't be keyboard-
/// activated, so they fall back to a DOM `.click()`.
async fn dom_activate(
client: &CdpClient,
session_id: &str,
ref_map: &RefMap,
selector_or_ref: &str,
iframe_sessions: &HashMap<String, String>,
) -> Result<(), String> {
let role = parse_ref(selector_or_ref)
.and_then(|r| ref_map.get(&r).map(|e| e.role.clone()))
.unwrap_or_default();
// Space toggles checkbox-like controls; Enter activates buttons/links/menus.
let key = match role.as_str() {
"checkbox" | "radio" | "switch" | "option" | "menuitemcheckbox" | "menuitemradio" => {
Some("space")
}
"button" | "link" | "menuitem" | "tab" | "treeitem" => Some("enter"),
_ => None,
};
let Some(key) = key else {
// Not keyboard-activatable — best effort via DOM .click() (untrusted).
return dom_click(
client,
session_id,
ref_map,
selector_or_ref,
iframe_sessions,
)
.await;
};
let (object_id, effective_session_id) = resolve_element_object_id(
client,
session_id,
ref_map,
selector_or_ref,
iframe_sessions,
)
.await?;
// Focus the element in its OWN frame session so the keystroke lands on it.
client
.send_command_typed::<_, Value>(
"Runtime.callFunctionOn",
&CallFunctionOnParams {
function_declaration: "function() { this.focus(); }".to_string(),
object_id: Some(object_id),
arguments: None,
return_by_value: Some(true),
await_promise: Some(false),
},
Some(&effective_session_id),
)
.await?;
// Trusted key on the page session — routed to the focused (in-frame) element.
press_key(client, session_id, key).await?;
wait_for_paint_settled(client, &effective_session_id).await;
Ok(())
}
/// DOM-dispatch a double-click on the element in its own session (no coordinates) /// DOM-dispatch a double-click on the element in its own session (no coordinates)
/// — the relay/iframe-safe counterpart to a coordinate dblclick. Fires the full /// — the relay/iframe-safe counterpart to a coordinate dblclick. Fires the full
/// click,click,dblclick sequence so handlers bound to any of them respond. /// click,click,dblclick sequence so handlers bound to any of them respond.
@@ -319,7 +403,14 @@ pub async fn dblclick(
if std::env::var("AGENT_BROWSER_CLICK_MODE").as_deref() != Ok("coord") if std::env::var("AGENT_BROWSER_CLICK_MODE").as_deref() != Ok("coord")
&& prefer_dom_dispatch(ref_map, selector_or_ref) && prefer_dom_dispatch(ref_map, selector_or_ref)
{ {
return dom_dblclick(client, session_id, ref_map, selector_or_ref, iframe_sessions).await; return dom_dblclick(
client,
session_id,
ref_map,
selector_or_ref,
iframe_sessions,
)
.await;
} }
click( click(
client, client,
@@ -387,7 +478,14 @@ pub async fn hover(
// Coordinate `mouseMoved` drifts to the foreground tab over the relay and // Coordinate `mouseMoved` drifts to the foreground tab over the relay and
// can't reach an OOPIF — DOM-dispatch the hover there (issues #31/#36). // can't reach an OOPIF — DOM-dispatch the hover there (issues #31/#36).
if prefer_dom_dispatch(ref_map, selector_or_ref) { if prefer_dom_dispatch(ref_map, selector_or_ref) {
return dom_hover(client, session_id, ref_map, selector_or_ref, iframe_sessions).await; return dom_hover(
client,
session_id,
ref_map,
selector_or_ref,
iframe_sessions,
)
.await;
} }
let (x, y, _w, _h, effective_session_id) = resolve_element_center( let (x, y, _w, _h, effective_session_id) = resolve_element_center(
client, client,
+10 -1
View File
@@ -345,7 +345,16 @@ pub async fn take_snapshot(
frame_id: Option<&str>, frame_id: Option<&str>,
iframe_sessions: &HashMap<String, String>, iframe_sessions: &HashMap<String, String>,
) -> Result<String, String> { ) -> Result<String, String> {
take_snapshot_at_depth(client, session_id, options, ref_map, frame_id, iframe_sessions, 0).await take_snapshot_at_depth(
client,
session_id,
options,
ref_map,
frame_id,
iframe_sessions,
0,
)
.await
} }
#[allow(clippy::too_many_arguments)] #[allow(clippy::too_many_arguments)]
@@ -0,0 +1,28 @@
<!doctype html>
<html>
<head>
<meta charset="utf-8" />
<title>iframe button probe</title>
</head>
<body>
<h1>iframe button probe</h1>
<iframe
id="frame"
width="320"
height="140"
srcdoc="
<!doctype html>
<html>
<body style='margin:24px'>
<button id='b' style='padding:24px;font-size:22px'>save</button>
<script>
document.getElementById('b').addEventListener('click', function (e) {
this.textContent = 'clicked:' + e.isTrusted;
});
</script>
</body>
</html>
"
></iframe>
</body>
</html>
+41 -8
View File
@@ -228,13 +228,28 @@ pub fn print_response_with_opts(resp: &Response, action: Option<&str>, opts: &Ou
// because its response carries `url`/`title`, which later generic // because its response carries `url`/`title`, which later generic
// renderers would otherwise swallow. // renderers would otherwise swallow.
if action == Some("cf_status") { if action == Some("cf_status") {
let challenged = data.get("challenged").and_then(|v| v.as_bool()).unwrap_or(false); let challenged = data
let rec = data.get("recommendation").and_then(|v| v.as_str()).unwrap_or("?"); .get("challenged")
.and_then(|v| v.as_bool())
.unwrap_or(false);
let rec = data
.get("recommendation")
.and_then(|v| v.as_str())
.unwrap_or("?");
let cl = data.get("clearance"); let cl = data.get("clearance");
let present = cl.and_then(|c| c.get("present")).and_then(|v| v.as_bool()).unwrap_or(false); let present = cl
let expired = cl.and_then(|c| c.get("expired")).and_then(|v| v.as_bool()).unwrap_or(false); .and_then(|c| c.get("present"))
.and_then(|v| v.as_bool())
.unwrap_or(false);
let expired = cl
.and_then(|c| c.get("expired"))
.and_then(|v| v.as_bool())
.unwrap_or(false);
let expires_in = cl.and_then(|c| c.get("expiresIn")).and_then(|v| v.as_i64()); let expires_in = cl.and_then(|c| c.get("expiresIn")).and_then(|v| v.as_i64());
let device = data.get("deviceVerified").and_then(|v| v.as_bool()).unwrap_or(false); let device = data
.get("deviceVerified")
.and_then(|v| v.as_bool())
.unwrap_or(false);
let (icon, headline) = match rec { let (icon, headline) = match rec {
"proceed" => (color::success_indicator().to_string(), "cleared — no challenge, proceed"), "proceed" => (color::success_indicator().to_string(), "cleared — no challenge, proceed"),
@@ -243,7 +258,10 @@ pub fn print_response_with_opts(resp: &Response, action: Option<&str>, opts: &Ou
_ => (color::cyan("").to_string(), "unknown"), _ => (color::cyan("").to_string(), "unknown"),
}; };
println!("{} {}", icon, headline); println!("{} {}", icon, headline);
println!(" challenged: {}", if challenged { "yes" } else { "no" }); println!(
" challenged: {}",
if challenged { "yes" } else { "no" }
);
let cl_desc = if !present { let cl_desc = if !present {
"absent".to_string() "absent".to_string()
} else if expired { } else if expired {
@@ -254,7 +272,14 @@ pub fn print_response_with_opts(resp: &Response, action: Option<&str>, opts: &Ou
"present (session)".to_string() "present (session)".to_string()
}; };
println!(" cf_clearance: {}", cl_desc); println!(" cf_clearance: {}", cl_desc);
println!(" device trusted: {}", if device { "yes (CF_VERIFIED_DEVICE)" } else { "no" }); println!(
" device trusted: {}",
if device {
"yes (CF_VERIFIED_DEVICE)"
} else {
"no"
}
);
return; return;
} }
@@ -382,7 +407,11 @@ pub fn print_response_with_opts(resp: &Response, action: Option<&str>, opts: &Ou
let count = list.len(); let count = list.len();
println!( println!(
"{}", "{}",
color::bold(&format!("{} frame{}", count, if count == 1 { "" } else { "s" })) color::bold(&format!(
"{} frame{}",
count,
if count == 1 { "" } else { "s" }
))
); );
for f in list { for f in list {
let idx = f.get("index").and_then(|v| v.as_i64()).unwrap_or(0); let idx = f.get("index").and_then(|v| v.as_i64()).unwrap_or(0);
@@ -3548,6 +3577,9 @@ iOS Simulator (requires Xcode and Appium):
chrome-use -p ios device list # List simulators chrome-use -p ios device list # List simulators
chrome-use -p ios swipe up # Swipe gesture chrome-use -p ios swipe up # Swipe gesture
chrome-use -p ios tap @e1 # Touch element chrome-use -p ios tap @e1 # Touch element
Hit a bug or rough edge? A 30-second issue genuinely sharpens this tool:
https://github.com/leeguooooo/chrome-use/issues
"# "#
); );
} }
@@ -3630,6 +3662,7 @@ fn print_screenshot_diff(data: &serde_json::Map<String, serde_json::Value>) {
pub fn print_version() { pub fn print_version() {
println!("chrome-use {}", env!("CARGO_PKG_VERSION")); println!("chrome-use {}", env!("CARGO_PKG_VERSION"));
println!("report bugs / rough edges: https://github.com/leeguooooo/chrome-use/issues");
} }
#[cfg(test)] #[cfg(test)]
+1 -1
View File
@@ -1,6 +1,6 @@
{ {
"name": "chrome-use", "name": "chrome-use",
"version": "1.5.12", "version": "1.5.18",
"description": "chrome-use — drive your real, logged-in Chrome from any AI agent, stealth by default", "description": "chrome-use — drive your real, logged-in Chrome from any AI agent, stealth by default",
"type": "module", "type": "module",
"packageManager": "pnpm@11.1.3", "packageManager": "pnpm@11.1.3",
+22 -1
View File
@@ -36,6 +36,17 @@ Refs (`@e1`, `@e2`, ...) are assigned fresh on every snapshot. They become
submits, dynamic re-renders, dialog opens. Always re-snapshot before your submits, dynamic re-renders, dialog opens. Always re-snapshot before your
next ref interaction. next ref interaction.
> **Hard rule: snapshot-first, never screenshot-to-locate.** For form fields and
> buttons, ALWAYS `snapshot -i` and act on refs/selectors. Do **not** reach for
> `screenshot` + coordinate clicks to find or hit an element — `snapshot -i` now
> pierces **cross-origin iframes** (embedded Google Payments / Stripe / checkout /
> KYC forms) and lists their elements by `@ref`, including input values. Use
> coordinates only for canvas/WebGL, or when `snapshot` genuinely returns nothing
> for your target. Screenshots are for *visual verification you report*, never the
> agent's own input — and a full-page `screenshot` of a real retina browser is
> often too large for an image reader anyway. (If you ever feel you *need* a
> screenshot to read state or locate something, that's a bug — please file it.)
> **Snapshot-first, always. Never default to `screenshot` + coordinate clicking > **Snapshot-first, always. Never default to `screenshot` + coordinate clicking
> for form fields or buttons.** Run `snapshot -i` and act on `@refs`. Use > for form fields or buttons.** Run `snapshot -i` and act on `@refs`. Use
> coordinates only for canvas/WebGL, or when `snapshot` genuinely returns nothing > coordinates only for canvas/WebGL, or when `snapshot` genuinely returns nothing
@@ -120,7 +131,17 @@ Each `--session` that connects gets its **own colored Chrome tab group** (named
after the session) and drives only its own tabs — multiple agents share the one after the session) and drives only its own tabs — multiple agents share the one
real browser without cross-talk, and the user's own tabs are never grouped. CDP real browser without cross-talk, and the user's own tabs are never grouped. CDP
drives the page without moving the user's mouse/keyboard, so it doesn't fight drives the page without moving the user's mouse/keyboard, so it doesn't fight
them for control. **Anti-detection ranking: this real logged-in Chrome (extension them for control.
**Strict multi-agent isolation.** A session over the relay tracks and drives
**only the tabs it created** (its own group). It does **not** adopt the user's
existing tabs, other agents' tabs, or pop-ups (e.g. an OAuth/login window — that's
the user's), so several agents (and other tools opening tabs) can work in the same
real Chrome concurrently without ever dropping or stealing each other's tabs —
another agent's tab churn can't make your bound tab vanish or drift your commands
onto the wrong page. Consequence: `tab list` shows only *your* session's tabs; to
drive a specific page, navigate to it in your own tab instead of expecting a
pre-existing or popped-up tab to appear in the list. **Anti-detection ranking: this real logged-in Chrome (extension
connect) > a headed launched browser > headless (forbidden).** A genuine human connect) > a headed launched browser > headless (forbidden).** A genuine human
browser has no headless/automation tells at all, so prefer it for anything browser has no headless/automation tells at all, so prefer it for anything
anti-bot-sensitive. anti-bot-sensitive.