Compare commits

...
35 Commits
Author SHA1 Message Date
leeguooooo eb2bc343a0 chore(release): 1.5.17 — complete multi-agent isolation (no foreign tab/pop-up adoption on click)
Release binaries / Build macOS ARM64 (push) Has been cancelled
Release binaries / Build macOS x64 (push) Has been cancelled
Release binaries / Build Linux ARM64 (push) Has been cancelled
Release binaries / Build Linux musl ARM64 (push) Has been cancelled
Release binaries / Build Linux musl x64 (push) Has been cancelled
Release binaries / Build Linux x64 (push) Has been cancelled
Release binaries / Build Windows x64 (push) Has been cancelled
Release binaries / Attach binaries to GitHub Release (push) Has been cancelled
2026-06-17 13:31:50 +09:00
leeguooooo d86c9c4be2 fix(relay): don't adopt foreign tabs/pop-ups on click — completes multi-agent isolation (#last hole)
After strict isolation (v1.5.16) a session's tracked set is only its OWN tabs, so
in `adopt_newly_opened` (run after every click to follow a pop-up) EVERY foreign
tab looks "new" relative to the session's `before` set and got adopted — a
click-heavy flow on a busy shared Chrome pulled other agents'/the user's tabs
(github, Lark, iphone-use) into the session mid-flow.

A pop-up the agent itself opened can't be told apart from a foreign tab over the
relay (the synthesized targetInfo carries no opener/window/group), so on the relay
`adopt_newly_opened` now adopts nothing: the agent drives only tabs it explicitly
created; pop-ups (OAuth/login windows) are the user's. Launched browsers (every
tab ours) still follow pop-ups. Verified live: a fresh click-heavy relay session
stays clean (only its own tabs), and 5 concurrent agents churning tabs show zero
cross-agent drift.
2026-06-17 13:31:49 +09:00
leeguooooo 32c25a6627 chore(release): 1.5.16 — strict multi-agent tab isolation on the relay
Release binaries / Build macOS ARM64 (push) Has been cancelled
Release binaries / Build macOS x64 (push) Has been cancelled
Release binaries / Build Linux ARM64 (push) Has been cancelled
Release binaries / Build Linux musl ARM64 (push) Has been cancelled
Release binaries / Build Linux musl x64 (push) Has been cancelled
Release binaries / Build Linux x64 (push) Has been cancelled
Release binaries / Build Windows x64 (push) Has been cancelled
Release binaries / Attach binaries to GitHub Release (push) Has been cancelled
2026-06-17 11:50:10 +09:00
leeguooooo a8ce3dd3f8 fix(relay): strict multi-agent isolation — a session owns only its own tabs
Several agents (and other tools opening tabs) share one real Chrome via the
relay. Previously every session adopted ALL tabs from Target.getTargets, so
another agent's tab churn polluted the list, dropped the tab being driven, and
drifted commands onto the wrong page (the W-8BEN tax tab vanished mid-flow when a
concurrent iphone-use agent opened tabs).

A tab group belongs to exactly one agent. On the relay a session now tracks and
drives ONLY the tabs it created (its own colored group) plus pop-ups its own
clicks open — it never adopts the user's or other agents' tabs:

- discover_and_attach (relay): create the session's own tab and pin it; do not
  adopt any existing foreign tab.
- resync_targets (relay): never adopt unknown targets; never prune the session's
  tabs on a single getTargets snapshot (multi-agent churn / cross-process-nav
  gaps routinely omit live tabs) — prune only after RELAY_PRUNE_MISSES
  consecutive absent snapshots (debounced), pinned active always protected.
- adopt_newly_opened: a tab that appears right after this session's action is a
  pop-up we opened — record it as owned.

Launched browsers (every tab ours) keep adopting all tabs. Adds debounced_prune_ids
+ unit tests for the churn tolerance.
2026-06-17 11:50:09 +09:00
leeguooooo 997373fd57 chore(release): 1.5.15 — trusted activation for in-iframe buttons (#39)
Release binaries / Build macOS ARM64 (push) Has been cancelled
Release binaries / Build macOS x64 (push) Has been cancelled
Release binaries / Build Linux ARM64 (push) Has been cancelled
Release binaries / Build Linux musl ARM64 (push) Has been cancelled
Release binaries / Build Linux musl x64 (push) Has been cancelled
Release binaries / Build Linux x64 (push) Has been cancelled
Release binaries / Build Windows x64 (push) Has been cancelled
Release binaries / Attach binaries to GitHub Release (push) Has been cancelled
2026-06-17 10:16:17 +09:00
leeguooooo 5a858af93f fix(iframe): trusted activation for in-iframe buttons — keyboard, not synthetic click (#39)
A DOM `.click()` is isTrusted:false, which security-sensitive embedded forms
reject — Google Payments' enabled `保存` button silently no-op'd, so a
cross-origin payment/checkout/KYC form could be read, scrolled, and typed into
but never submitted. A coordinate click can't help either: getBoxModel for a
sub-frame node returns frame-local coords that don't compose the iframe offset,
so it lands wrong (verified — the same-origin probe came back isTrusted:false
via the coordinate fallback).

Fix: click on an in-iframe ref now focuses the element in its own frame session
and dispatches a real Enter (Space for checkbox-like roles) on the page session.
Chrome routes the key to the focused element across frames (same mechanism as
`type --focused`), and Enter/Space on a focused button/link/checkbox fires a
trusted click. Non-activatable roles fall back to DOM .click().

Adds e2e_iframe_button_click_is_trusted (+ fixture): an in-iframe button records
event.isTrusted into its own text; the test asserts the ref-click delivers
isTrusted:true.
2026-06-17 10:16:15 +09:00
leeguooooo 58dc02bfdc chore(release): 1.5.14 — fix eval await regression (replMode) + default scroll; green CI (#36, #38)
Release binaries / Build macOS ARM64 (push) Has been cancelled
Release binaries / Build macOS x64 (push) Has been cancelled
Release binaries / Build Linux ARM64 (push) Has been cancelled
Release binaries / Build Linux musl ARM64 (push) Has been cancelled
Release binaries / Build Linux musl x64 (push) Has been cancelled
Release binaries / Build Linux x64 (push) Has been cancelled
Release binaries / Build Windows x64 (push) Has been cancelled
Release binaries / Attach binaries to GitHub Release (push) Has been cancelled
2026-06-17 02:34:55 +09:00
leeguooooo c47601bd7b fix(eval): replMode only for sync let/const decls, keep awaitPromise for async (#38)
replMode and awaitPromise are mutually exclusive in Chrome — under replMode a
returned promise serialises to {} instead of being awaited, which broke every
fetch/async eval (e2e_domain_filter, e2e_headers, e2e_react_tree all regressed).
Enable replMode only for synchronous scripts that declare a top-level let/const
(the #38 case); promise-returning scripts keep awaitPromise — restoring the
pre-#38 await behaviour while still fixing the let-redeclaration collision.
2026-06-17 02:08:11 +09:00
leeguooooo 0296bc7a88 fix(scroll): keep default scroll on window.scrollBy; wheel only for --at/--frame (#36)
The centered-wheel default no-op'd on some pages (headless e2e_hover_scroll_press
regressed). Restore window.scrollBy for plain page scroll; the coordinate wheel
stays opt-in via --at/--frame for cross-origin iframe content.
2026-06-17 02:01:23 +09:00
leeguooooo 32e203b908 style: cargo fmt (fixes the CI format-check failure) 2026-06-17 01:33:22 +09:00
leeguooooo fc51cd63ba chore(release): 1.5.13 — eval replMode (re-declarable let/const) + snapshot-first skill rule (#37, #38)
Release binaries / Build macOS ARM64 (push) Has been cancelled
Release binaries / Build macOS x64 (push) Has been cancelled
Release binaries / Build Linux ARM64 (push) Has been cancelled
Release binaries / Build Linux musl ARM64 (push) Has been cancelled
Release binaries / Build Linux musl x64 (push) Has been cancelled
Release binaries / Build Linux x64 (push) Has been cancelled
Release binaries / Build Windows x64 (push) Has been cancelled
Release binaries / Attach binaries to GitHub Release (push) Has been cancelled
2026-06-17 01:15:31 +09:00
leeguooooo f714c7920b fix(eval): replMode so successive evals can re-declare let/const; snapshot-first skill rule (#37, #38)
#38: `chrome-use eval` now runs with Runtime.evaluate replMode (like the DevTools
console) — top-level `let`/`const` no longer throw "already been declared" across
successive evals (independent `eval` steps in a `test` suite collided in the
page's shared lexical scope), and top-level await is allowed. Main-world and
completion-value semantics are unchanged.

#37: core skill gains a hard rule — snapshot-first, never screenshot+coordinates
to locate form fields/buttons; `snapshot -i` now pierces cross-origin iframes and
lists their elements by @ref; screenshots are for visual checks only, and a
full-page retina screenshot often exceeds an image reader's limits.
2026-06-17 01:15:31 +09:00
leeguooooo 1ac8ef7732 chore(release): 1.5.12 — relay-safe hover/dblclick/drag, deeper iframe snapshot, key-events typing (#37)
Release binaries / Build macOS ARM64 (push) Has been cancelled
Release binaries / Build macOS x64 (push) Has been cancelled
Release binaries / Build Linux ARM64 (push) Has been cancelled
Release binaries / Build Linux musl ARM64 (push) Has been cancelled
Release binaries / Build Linux musl x64 (push) Has been cancelled
Release binaries / Build Linux x64 (push) Has been cancelled
Release binaries / Build Windows x64 (push) Has been cancelled
Release binaries / Attach binaries to GitHub Release (push) Has been cancelled
2026-06-17 00:58:23 +09:00
leeguooooo 9f24e66033 fix(relay): DOM-dispatch hover/dblclick/drag; deeper iframe snapshot; key-events typing (#37)
Follow-up to #36 — make the whole interaction surface reach cross-origin OOPIFs
and stop coordinate events drifting onto the user's foreground tab over the relay.

- hover/dblclick/drag now DOM-dispatch over the relay or into an iframe (like
  click already did): a coordinate Input event isn't confined to the target tab
  on a busy real Chrome and can't map an OOPIF element's box to a top-viewport
  point. drag does an HTML5 DnD in the element's frame; cross-frame drag errors
  loudly instead of drifting.
- snapshot recurses iframes to MAX_IFRAME_DEPTH (3) instead of one level, so refs
  inside nested payment/checkout widgets get a frame_id and resolve into the
  right frame.
- relay tab adoption merges several Target.getTargets snapshots — a single flaky
  relay snapshot was dropping live tabs (a driven tab vanished after restart).
- `type --key-events` (alias --keys) sends real per-character keyDown/keyUp
  instead of Input.insertText, so autocomplete/combobox widgets that ignore the
  insertText input event fire (Google address postal lookup; commits Angular
  reactive forms so Save enables).
- SKILL: hard "snapshot-first, never default to screenshot+coordinates" rule;
  snapshot -i pierces cross-origin iframes since v1.5.12; cross-origin iframe
  driving guidance (#37).
2026-06-17 00:58:12 +09:00
leeguooooo 70ab38d35f chore(release): 1.5.11 — cross-origin iframe scroll/click + open auto-reattach (#35, #36)
Release binaries / Build macOS ARM64 (push) Has been cancelled
Release binaries / Build macOS x64 (push) Has been cancelled
Release binaries / Build Linux ARM64 (push) Has been cancelled
Release binaries / Build Linux musl ARM64 (push) Has been cancelled
Release binaries / Build Linux musl x64 (push) Has been cancelled
Release binaries / Build Linux x64 (push) Has been cancelled
Release binaries / Build Windows x64 (push) Has been cancelled
Release binaries / Attach binaries to GitHub Release (push) Has been cancelled
2026-06-16 18:05:15 +09:00
leeguooooo 6830df50ea fix(relay): reach cross-origin iframes; auto-reattach open (#35, #36)
#35: `open` auto-reattaches when the bound relay tab is gone — drops the dead
page, opens a fresh tab in the session's group, and navigates it, instead of
only `tab new` recovering.

#36: scroll and click now reach content inside cross-origin OOPIFs:
- scroll dispatches a real wheel at a viewport point (default center, --at x,y,
  or --frame n) so it scrolls the iframe under the pointer, which
  window.scrollBy on the top document silently no-ops on.
- over the extension relay, clicks always use DOM-dispatch instead of
  coordinate Input events — a coordinate event isn't confined to the target tab
  on a busy real Chrome (it drifted onto the foreground tab) and an OOPIF
  element's box can't be mapped to a top-viewport point.
2026-06-16 18:05:06 +09:00
leeguooooo cd47ec43d0 chore(release): 1.5.10 — warn on debug-port launch while relay is up (#32)
Release binaries / Build macOS ARM64 (push) Has been cancelled
Release binaries / Build macOS x64 (push) Has been cancelled
Release binaries / Build Linux ARM64 (push) Has been cancelled
Release binaries / Build Linux musl ARM64 (push) Has been cancelled
Release binaries / Build Linux musl x64 (push) Has been cancelled
Release binaries / Build Linux x64 (push) Has been cancelled
Release binaries / Build Windows x64 (push) Has been cancelled
Release binaries / Attach binaries to GitHub Release (push) Has been cancelled
2026-06-16 14:32:29 +09:00
leeguooooo 2cd361817d fix(launch): warn when launching a debug-port Chrome while the relay is up (#32)
The connect-mode diagnostic (1.5.5) proved the 'Allow remote debugging?' modal
is NOT Chrome 149 UX (my earlier hypothesis) — it's chrome-use launching a fresh
debug-port Chrome on session=default while the ab-connect relay is up (32 logged
CONSENT-MODAL-RISK launches), almost always from a stray --launch/--no-auto-connect.
A launch now warns loudly when the relay is available, naming the modal and how
to avoid it (drop --launch/--new, don't pass --no-auto-connect), so the modal is
self-explained and the offending caller is fixable.
2026-06-16 14:32:28 +09:00
leeguooooo 42f47c49aa chore(release): 1.5.9 — strip zero-width title unicode (#33) + screenshot --clip/element (#34)
Release binaries / Build macOS ARM64 (push) Has been cancelled
Release binaries / Build macOS x64 (push) Has been cancelled
Release binaries / Build Linux ARM64 (push) Has been cancelled
Release binaries / Build Linux musl ARM64 (push) Has been cancelled
Release binaries / Build Linux musl x64 (push) Has been cancelled
Release binaries / Build Linux x64 (push) Has been cancelled
Release binaries / Build Windows x64 (push) Has been cancelled
Release binaries / Attach binaries to GitHub Release (push) Has been cancelled
2026-06-16 14:26:41 +09:00
leeguooooo e29800df72 fix(tab-list): strip zero-width unicode from titles (#33); feat(screenshot): --clip pixel region + documented element capture (#34)
#33: some sites prepend runs of ZWJ/word-joiner/invisible-times/BOM to
document.title (badging/anti-scrape); left in, they polluted 'tab list', broke
text matching, and wrecked column alignment. sanitize_title() now strips
zero-width/bidi-format chars at every title ingestion point + get_title().

#34: 'screenshot <selector>' (element capture) already worked but was
undocumented; added 'screenshot --clip x,y,w,h' for an explicit pixel region
(CDP captureScreenshot clip), documented both in --help. Verified live.
2026-06-16 14:26:40 +09:00
leeguooooo e7e849ea39 chore(release): 1.5.8 — file upload over the extension relay (#13)
Release binaries / Build macOS ARM64 (push) Has been cancelled
Release binaries / Build macOS x64 (push) Has been cancelled
Release binaries / Build Linux ARM64 (push) Has been cancelled
Release binaries / Build Linux musl ARM64 (push) Has been cancelled
Release binaries / Build Linux musl x64 (push) Has been cancelled
Release binaries / Build Linux x64 (push) Has been cancelled
Release binaries / Build Windows x64 (push) Has been cancelled
Release binaries / Attach binaries to GitHub Release (push) Has been cancelled
2026-06-16 14:05:33 +09:00
leeguooooo ebb02c65c8 fix(relay): file upload now works over the extension relay (#13)
chrome.debugger forbids DOM.setFileInputFiles, so 'upload' used to hard-fail on
the relay and push users to a --launch/direct-CDP session. Now it falls back to
reconstructing the File entirely in the page (Playwright/Cypress-style: build a
File from the bytes, assign input.files = dataTransfer.files, fire input/change;
for drop/paste composers like X, dispatch synthetic paste+drop with the
DataTransfer). The bytes are streamed in <1 MiB base64 chunks because the relay
tunnels CDP through native messaging (1 MiB/message cap) — a whole image as one
arg closed the channel. Verified live over the relay: an 809 KB PNG lands intact
on a file input with change firing. No more direct-CDP needed for uploads.
2026-06-16 14:05:31 +09:00
leeguooooo 4317db636f chore(release): 1.5.7 — cf-status Cloudflare clearance preflight
Release binaries / Build macOS ARM64 (push) Has been cancelled
Release binaries / Build macOS x64 (push) Has been cancelled
Release binaries / Build Linux ARM64 (push) Has been cancelled
Release binaries / Build Linux musl ARM64 (push) Has been cancelled
Release binaries / Build Linux musl x64 (push) Has been cancelled
Release binaries / Build Linux x64 (push) Has been cancelled
Release binaries / Build Windows x64 (push) Has been cancelled
Release binaries / Attach binaries to GitHub Release (push) Has been cancelled
2026-06-16 12:19:26 +09:00
leeguooooo c99838a034 feat(cloudflare): cf-status preflight — skip re-solving when cf_clearance is still valid
Passing a Cloudflare challenge mints an HttpOnly cf_clearance cookie bound to
IP+UA. 'chrome-use cf-status' (aliases cf/cloudflare-status/clearance) reports
whether the active page is currently a CF challenge and whether a still-valid
cf_clearance exists (read via CDP — HttpOnly is invisible to document.cookie),
plus CF_VERIFIED_DEVICE trust, and a recommendation: proceed (already cleared,
don't re-solve) / solve (challenge up, no clearance) / reissue (clearance present
but page still blocks → IP/UA drifted). Lets an agent avoid re-solving what it
already cleared — the persistence optimization. Pure helpers unit-tested; live
-verified on a real cf_clearance.
2026-06-16 12:19:24 +09:00
leeguooooo dc2aa4cade chore(release): 1.5.6 — pin adopted tab against transient relay snapshots (#31)
Release binaries / Build macOS ARM64 (push) Has been cancelled
Release binaries / Build macOS x64 (push) Has been cancelled
Release binaries / Build Linux ARM64 (push) Has been cancelled
Release binaries / Build Linux musl ARM64 (push) Has been cancelled
Release binaries / Build Linux musl x64 (push) Has been cancelled
Release binaries / Build Linux x64 (push) Has been cancelled
Release binaries / Build Windows x64 (push) Has been cancelled
Release binaries / Attach binaries to GitHub Release (push) Has been cancelled
2026-06-15 17:00:42 +09:00
leeguooooo 7085f3bf36 fix(relay): don't prune the pinned target on a transient getTargets snapshot (#31)
Driving a busy real Chrome via the relay, a single Target.getTargets call
occasionally returns a different window's tabs ('tab list hops windows'). resync
pruned every tracked page absent from that snapshot — including the agent's
explicitly-adopted (pinned) tab — after which active-target resolution fell back
to active_page_index and eval/click/snapshot drifted onto a foreign tab
(about:blank / chrome-extension:// / the user's page), breaking any 3+ step flow.

prunable_target_ids() now protects the pinned active target from snapshot-based
pruning; a genuine close still arrives as Target.targetDestroyed (event drain) and
removes it properly. Unit-tested.
2026-06-15 17:00:38 +09:00
leeguooooo 29815ff5f3 chore(release): 1.5.5 — connect-mode diagnostic log for the remote-debugging consent modal (#31)
Release binaries / Build macOS ARM64 (push) Has been cancelled
Release binaries / Build macOS x64 (push) Has been cancelled
Release binaries / Build Linux ARM64 (push) Has been cancelled
Release binaries / Build Linux musl ARM64 (push) Has been cancelled
Release binaries / Build Linux musl x64 (push) Has been cancelled
Release binaries / Build Linux x64 (push) Has been cancelled
Release binaries / Build Windows x64 (push) Has been cancelled
Release binaries / Attach binaries to GitHub Release (push) Has been cancelled
2026-06-15 16:47:14 +09:00
leeguooooo 2a338d4c29 diag(connect): log CDP transport mode to detect 'Allow remote debugging?' modal source (#31)
The consent modal only appears on a raw remote-debugging attach or a browser we
launched with a debug port — never on the ab-connect extension relay. Append one
line per connection to ~/.chrome-use/connect-mode.log (relay | raw-port-attach |
launched | remote-ws), flagging 'CONSENT-MODAL-RISK' when a raw-port/launch path
runs while the relay was available. Lets us tell a code regression from Chrome's
own extension-debugger consent UX when the modal reappears. Best-effort, never
fails a connection. Verified: normal 'open' logs mode=relay (consent-free).
2026-06-15 16:47:12 +09:00
leeguooooo 6fcf52db60 chore(release): 1.5.4 — get text --pierce (closed shadow DOM, #30)
Release binaries / Build macOS ARM64 (push) Has been cancelled
Release binaries / Build macOS x64 (push) Has been cancelled
Release binaries / Build Linux ARM64 (push) Has been cancelled
Release binaries / Build Linux musl ARM64 (push) Has been cancelled
Release binaries / Build Linux musl x64 (push) Has been cancelled
Release binaries / Build Linux x64 (push) Has been cancelled
Release binaries / Build Windows x64 (push) Has been cancelled
Release binaries / Attach binaries to GitHub Release (push) Has been cancelled
2026-06-15 15:46:31 +09:00
leeguooooo af8823b27b feat(text): 'get text --pierce' reads through CLOSED shadow DOM (#30)
Some injected UI (browser-extension debug panels, web components) renders into a
CLOSED shadow root that eval/innerText cannot read. --pierce walks the CDP DOM
tree (DOM.getDocument depth:-1 pierce:true), which includes closed shadow roots
and child documents, and collects text nodes (skipping script/style/etc).

Review-safe: rides the per-tab debugger session already attached, no new Chrome
permission and no ab-connect/extension change — so it works in extension-relay
mode without touching the published extension. Verified live: a closed-shadow
panel that main-world eval reports HIDDEN is read in full via --pierce.

First slice of #30 (read extension/injected-panel content). Deeper extension
introspection (background SW / chrome.storage) stays a launch-mode / raw-CDP
concern, deliberately NOT done by expanding ab-connect's debugger powers.
2026-06-15 15:46:23 +09:00
leeguooooo c85e3faa82 chore(release): 1.5.3 — get text defaults to cross-frame; #29 (sessions/did-you-mean/tab liveness); CI changelog fix
Release binaries / Build macOS ARM64 (push) Has been cancelled
Release binaries / Build macOS x64 (push) Has been cancelled
Release binaries / Build Linux ARM64 (push) Has been cancelled
Release binaries / Build Linux musl ARM64 (push) Has been cancelled
Release binaries / Build Linux musl x64 (push) Has been cancelled
Release binaries / Build Linux x64 (push) Has been cancelled
Release binaries / Build Windows x64 (push) Has been cancelled
Release binaries / Attach binaries to GitHub Release (push) Has been cancelled
2026-06-15 13:42:58 +09:00
leeguooooo b25958946c feat(text): 'get text' (no selector) defaults to cross-frame whole-page read
So an agent never silently misses iframed content (listing descriptions etc.)
without having to know the --all-frames flag. Single-frame pages are unchanged
(identical to the old body read); multi-frame pages now include child frames —
a strict superset. Skill + help updated to make the default and 'frames'/--main
discoverable.
2026-06-15 13:42:48 +09:00
leeguooooo d4ff49caa8 ci(release): don't let an empty changelog section abort the release (bash -e)
The changelog step runs under 'bash -e'. section() returned non-zero when a
commit category was empty (grep no-match / empty [ -n ] test), aborting the
script before the closing heredoc delimiter — so any release whose range lacked
a whole category (e.g. 1.5.2: only feat, no fix) failed to attach binaries.
Add '|| true' + 'return 0' so section() always succeeds.
2026-06-15 13:42:48 +09:00
leeguooooo 4e949fffbf chore(release): 1.5.2 — sessions command + did-you-mean + honest tab-switch liveness (#29) 2026-06-15 13:34:26 +09:00
leeguooooo af46490812 feat(cli): sessions command + 'did you mean' suggestions + honest tab-switch liveness (#29)
- chrome-use sessions: top-level alias for the daemon inventory (the skill
  advertises sessions, so it's a natural guess that used to error).
- Unknown commands now suggest the nearest valid one (Levenshtein + prefix
  match), staying silent when nothing is close (e.g. 'clik' -> click,
  'sesions' -> sessions, 'xyzzy' -> no suggestion).
- tab <id>: probe the switched session and show a warning indicator instead of
  a green check when it isn't responding yet, so a switch onto a re-attaching
  (churned-tabId) session no longer reports false success. The #24 targetId
  recovery self-heals within ~6s, hence a warning rather than a hard error.
2026-06-15 13:25:16 +09:00
17 changed files with 2160 additions and 150 deletions
+9 -2
View File
@@ -152,9 +152,16 @@ jobs:
# Group commit subjects by conventional-commit type so the notes are
# scannable ("what's new / what's fixed") instead of a flat dev log.
LOG="$(git log "$RANGE" --no-merges --pretty='%s' | grep -v '^chore(release)' || true)"
# NOTE: the job runs under `bash -e`. grep returning 1 (no match) and
# the `[ -n "$body" ]` test returning 1 (empty section) must NOT abort
# the script — otherwise a release whose commit range lacks a whole
# category (e.g. only `feat`, no `fix`) dies before writing the closing
# heredoc delimiter and the whole release step fails. `|| true` +
# `return 0` keep section() always-succeeding.
section() { # $1=header $2=grep-pattern
local body; body="$(printf '%s\n' "$LOG" | grep -E "$2" | sed 's/^/- /')"
[ -n "$body" ] && { printf '\n### %s\n%s\n' "$1" "$body"; }
local body; body="$(printf '%s\n' "$LOG" | grep -E "$2" | sed 's/^/- /' || true)"
[ -n "$body" ] && printf '\n### %s\n%s\n' "$1" "$body"
return 0
}
{
echo "notes<<__NOTES_EOF__"
+1 -1
View File
@@ -290,7 +290,7 @@ checksum = "613afe47fcd5fac7ccf1db93babcb082c5994d996f20b8b159f2ad1658eb5724"
[[package]]
name = "chrome-use"
version = "1.5.1"
version = "1.5.17"
dependencies = [
"aes",
"aes-gcm",
+1 -1
View File
@@ -1,6 +1,6 @@
[package]
name = "chrome-use"
version = "1.5.1"
version = "1.5.17"
edition = "2021"
description = "Fast browser automation CLI for AI agents"
license = "Apache-2.0"
+329 -32
View File
@@ -30,12 +30,104 @@ pub enum ParseError {
InvalidSessionName { name: String },
}
/// Top-level commands an agent is likely to mistype, used for "did you mean"
/// suggestions on an unknown command (issue #29). Not exhaustive — just the
/// common verbs plus a few known wrong-guesses mapped to the real command.
const KNOWN_COMMANDS: &[&str] = &[
"open",
"navigate",
"click",
"fill",
"type",
"press",
"snapshot",
"screenshot",
"eval",
"get",
"text",
"html",
"frames",
"find",
"wait",
"scroll",
"hover",
"select",
"check",
"uncheck",
"tab",
"tabs",
"close",
"back",
"forward",
"reload",
"sessions",
"status",
"daemon",
"doctor",
"upgrade",
"connect",
"cookies",
"mouse",
"keyboard",
"stream",
"frame",
"profiles",
"title",
"url",
"is",
"drag",
"dialog",
"upload",
];
/// Levenshtein distance, capped — small inputs only (command names).
fn edit_distance(a: &str, b: &str) -> usize {
let a: Vec<char> = a.chars().collect();
let b: Vec<char> = b.chars().collect();
let mut prev: Vec<usize> = (0..=b.len()).collect();
let mut curr = vec![0usize; b.len() + 1];
for (i, &ca) in a.iter().enumerate() {
curr[0] = i + 1;
for (j, &cb) in b.iter().enumerate() {
let cost = if ca == cb { 0 } else { 1 };
curr[j + 1] = (prev[j + 1] + 1).min(curr[j] + 1).min(prev[j] + cost);
}
std::mem::swap(&mut prev, &mut curr);
}
prev[b.len()]
}
/// Closest known command within a small edit distance, or a prefix/substring
/// match — `None` if nothing is close enough to suggest confidently.
fn nearest_command(input: &str) -> Option<String> {
let lower = input.to_lowercase();
// Exact prefix/substring hits first (e.g. "session" -> "sessions").
if let Some(c) = KNOWN_COMMANDS
.iter()
.find(|c| c.starts_with(&lower) || lower.starts_with(**c))
{
return Some(c.to_string());
}
// Tolerance scales with length: short words get distance 1, longer get 2.
let max_dist = if lower.len() <= 4 { 1 } else { 2 };
KNOWN_COMMANDS
.iter()
.map(|c| (*c, edit_distance(&lower, c)))
.filter(|(_, d)| *d <= max_dist)
.min_by_key(|(_, d)| *d)
.map(|(c, _)| c.to_string())
}
impl ParseError {
pub fn format(&self) -> String {
match self {
ParseError::UnknownCommand { command } => {
format!("Unknown command: {}", command)
}
ParseError::UnknownCommand { command } => match nearest_command(command) {
Some(suggestion) => format!(
"Unknown command: {}\nDid you mean: chrome-use {}?",
command, suggestion
),
None => format!("Unknown command: {}", command),
},
ParseError::UnknownSubcommand {
subcommand,
valid_options,
@@ -472,20 +564,31 @@ fn parse_command_inner(args: &[String], flags: &Flags) -> Result<Value, ParseErr
Ok(json!({ "id": id, "action": "fill", "selector": sel, "value": rest[1..].join(" ") }))
}
"type" => {
// `--key-events` (alias `--keys`): send real per-character keystrokes
// instead of Input.insertText, so autocomplete/combobox widgets that
// only react to key events fire (e.g. Google address postal lookup).
let key_events = rest.iter().any(|a| *a == "--key-events" || *a == "--keys");
let rest: Vec<&str> = rest
.iter()
.copied()
.filter(|a| *a != "--key-events" && *a != "--keys")
.collect();
// `type --focused <text>` types into whatever element currently has
// focus (no selector) — for custom widgets that move focus to a hidden
// input after you open them.
if rest.first() == Some(&"--focused") {
return Ok(json!({
"id": id, "action": "type", "focused": true,
"text": rest[1..].join(" "),
"text": rest[1..].join(" "), "keyEvents": key_events,
}));
}
let sel = rest.first().ok_or_else(|| ParseError::MissingArguments {
context: "type".to_string(),
usage: "type <selector> <text> (or: type --focused <text>)",
usage: "type <selector> <text> (or: type --focused <text>) [--key-events]",
})?;
Ok(json!({ "id": id, "action": "type", "selector": sel, "text": rest[1..].join(" ") }))
Ok(
json!({ "id": id, "action": "type", "selector": sel, "text": rest[1..].join(" "), "keyEvents": key_events }),
)
}
"pick" => {
// pick <selector|@ref> --option "<text>" — atomic combobox select:
@@ -676,10 +779,57 @@ fn parse_command_inner(args: &[String], flags: &Flags) -> Result<Value, ParseErr
} else {
return Err(ParseError::MissingArguments {
context: "scroll --selector".to_string(),
usage: "scroll [direction] [amount] [--selector <sel>]",
usage: "scroll [direction] [amount] [--selector <sel>] [--at <x,y>] [--frame <n>]",
});
}
}
"--at" => {
// `--at x,y`: dispatch the wheel at this viewport pixel, so it
// scrolls whatever element/iframe is under the pointer — including
// cross-origin iframes that `window.scrollBy` can't reach (#36).
let val = rest.get(i + 1).ok_or(ParseError::MissingArguments {
context: "scroll --at".to_string(),
usage: "scroll [direction] [amount] --at <x,y>",
})?;
let mut parts = val.split(',');
match (
parts.next().and_then(|s| s.trim().parse::<f64>().ok()),
parts.next().and_then(|s| s.trim().parse::<f64>().ok()),
) {
(Some(x), Some(y)) => {
obj.insert("at".to_string(), json!([x, y]));
}
_ => {
return Err(ParseError::InvalidValue {
message: format!("scroll --at: invalid coordinate `{}`", val),
usage:
"scroll [direction] [amount] --at <x,y> (e.g. --at 640,400)",
})
}
}
i += 1;
}
"--frame" => {
// `--frame n`: scroll the n-th frame from `chrome-use frames` by
// dispatching the wheel at that frame's center — reaches content in
// a cross-origin iframe without needing a selector into it (#36).
let val = rest.get(i + 1).ok_or(ParseError::MissingArguments {
context: "scroll --frame".to_string(),
usage: "scroll [direction] [amount] --frame <n>",
})?;
match val.trim().parse::<usize>() {
Ok(n) => {
obj.insert("frame".to_string(), json!(n));
}
Err(_) => {
return Err(ParseError::InvalidValue {
message: format!("scroll --frame: invalid index `{}`", val),
usage: "scroll [direction] [amount] --frame <n> (index from `chrome-use frames`)",
})
}
}
i += 1;
}
arg if arg.starts_with('-') => {}
_ => {
match positional_index {
@@ -854,17 +1004,41 @@ fn parse_command_inner(args: &[String], flags: &Flags) -> Result<Value, ParseErr
// selector: @ref or CSS selector
// path: file path (contains / or . or ends with known extension)
let mut full_page = false;
let positional: Vec<&str> = rest
.iter()
.filter(|arg| match **arg {
"--full" | "-f" => {
full_page = true;
false
let mut clip: Option<Value> = None;
let mut positional: Vec<&str> = Vec::new();
let mut i = 0;
while i < rest.len() {
match rest[i] {
"--full" | "-f" => full_page = true,
// `--clip x,y,w,h` captures a pixel region (issue #34).
"--clip" => {
let raw = rest
.get(i + 1)
.ok_or_else(|| ParseError::MissingArguments {
context: "screenshot --clip".to_string(),
usage: "screenshot --clip <x,y,w,h> [path]",
})?;
let nums: Vec<f64> = raw
.split(',')
.filter_map(|n| n.trim().parse::<f64>().ok())
.collect();
if nums.len() != 4 {
return Err(ParseError::InvalidValue {
message: format!(
"--clip expects 'x,y,w,h' (4 numbers), got '{raw}'"
),
usage: "screenshot --clip <x,y,w,h> [path]",
});
}
clip = Some(json!({
"x": nums[0], "y": nums[1], "width": nums[2], "height": nums[3]
}));
i += 1;
}
_ => true,
})
.copied()
.collect();
other => positional.push(other),
}
i += 1;
}
let (selector, path) = match (positional.first(), positional.get(1)) {
(Some(first), Some(second)) => {
// Two args: first is selector, second is path
@@ -895,6 +1069,9 @@ fn parse_command_inner(args: &[String], flags: &Flags) -> Result<Value, ParseErr
"path": path, "selector": selector,
"fullPage": full_page, "annotate": flags.annotate
});
if let Some(c) = clip {
cmd["clip"] = c;
}
if let Some(ref fmt) = flags.screenshot_format {
cmd["format"] = json!(fmt);
}
@@ -1025,6 +1202,14 @@ fn parse_command_inner(args: &[String], flags: &Flags) -> Result<Value, ParseErr
Ok(json!({ "id": id, "action": "stealth_status" }))
}
// `cf-status` — Cloudflare challenge/clearance preflight: is the page
// currently a CF challenge, and is there a still-valid cf_clearance (the
// HttpOnly persistence cookie)? Lets an agent SKIP re-solving when already
// cleared, and know when it must solve. Persistence optimization.
"cf-status" | "cf" | "cloudflare-status" | "clearance" => {
Ok(json!({ "id": id, "action": "cf_status" }))
}
// === Close ===
"close" | "quit" | "exit" => {
// `close <tab>` closes only that tab (and the output says "Tab
@@ -2402,6 +2587,15 @@ fn parse_get(rest: &[&str], id: &str) -> Result<Value, ParseError> {
if all_frames {
return Ok(json!({ "id": id, "action": "gettext", "allFrames": true }));
}
// `get text --pierce` reads through CLOSED shadow DOM / child docs
// via the CDP DOM tree — content eval/innerText can't reach, e.g. an
// extension's injected panel in a closed shadow root (issue #30).
let pierce = rest[1..]
.iter()
.any(|a| matches!(*a, "--pierce" | "--shadow" | "--deep"));
if pierce {
return Ok(json!({ "id": id, "action": "gettext", "pierce": true }));
}
// `get text --main` returns the main-content region (readability),
// skipping header/nav/footer/sidebar boilerplate (issue #27).
let main = rest[1..]
@@ -2410,16 +2604,16 @@ fn parse_get(rest: &[&str], id: &str) -> Result<Value, ParseError> {
if main {
return Ok(json!({ "id": id, "action": "gettext", "main": true }));
}
// `get text` with no selector returns the whole page's text (body) —
// a common convenience; previously it errored without a selector
// (issue #24-D).
let sel = rest
.iter()
.skip(1)
.find(|a| !a.starts_with("--"))
.copied()
.unwrap_or("body");
Ok(json!({ "id": id, "action": "gettext", "selector": sel }))
// `get text` with no selector reads the WHOLE PAGE and now defaults
// to cross-frame aggregation, so an agent gets a page's iframed
// content (listing descriptions etc.) without having to know about
// `--all-frames` (#27). On a single-frame page this is identical to
// the old body read; multi-frame pages get the child frames too —
// a strict superset. An explicit selector stays element-scoped.
match rest.iter().skip(1).find(|a| !a.starts_with("--")).copied() {
Some(sel) => Ok(json!({ "id": id, "action": "gettext", "selector": sel })),
None => Ok(json!({ "id": id, "action": "gettext", "allFrames": true })),
}
}
Some("html") => {
let sel = rest.get(1).ok_or_else(|| ParseError::MissingArguments {
@@ -3973,6 +4167,28 @@ mod tests {
assert_eq!(cmd["action"], "type");
assert_eq!(cmd["selector"], "#input");
assert_eq!(cmd["text"], "some text");
assert_eq!(cmd["keyEvents"], false);
}
#[test]
fn test_type_key_events() {
// --key-events sends real keystrokes (for autocomplete/combobox) and must
// not be swallowed into the typed text.
let cmd = parse_command(
&args("type #postal 201-0001 --key-events"),
&default_flags(),
)
.unwrap();
assert_eq!(cmd["action"], "type");
assert_eq!(cmd["selector"], "#postal");
assert_eq!(cmd["text"], "201-0001");
assert_eq!(cmd["keyEvents"], true);
let focused =
parse_command(&args("type --focused 201-0001 --keys"), &default_flags()).unwrap();
assert_eq!(focused["focused"], true);
assert_eq!(focused["text"], "201-0001");
assert_eq!(focused["keyEvents"], true);
}
#[test]
@@ -4257,6 +4473,24 @@ mod tests {
assert_eq!(cmd["fullPage"], true);
}
#[test]
fn test_screenshot_clip() {
// `--clip x,y,w,h` captures a pixel region (issue #34); the path still parses.
let cmd = parse_command(
&args("screenshot --clip 10,20,200,40 out.png"),
&default_flags(),
)
.unwrap();
assert_eq!(cmd["action"], "screenshot");
assert_eq!(cmd["clip"]["x"], 10.0);
assert_eq!(cmd["clip"]["y"], 20.0);
assert_eq!(cmd["clip"]["width"], 200.0);
assert_eq!(cmd["clip"]["height"], 40.0);
assert_eq!(cmd["path"], "out.png");
// Bad clip is a clear error, not silent.
assert!(parse_command(&args("screenshot --clip 1,2,3"), &default_flags()).is_err());
}
#[test]
fn test_screenshot_with_ref() {
let cmd = parse_command(&args("screenshot @e1"), &default_flags()).unwrap();
@@ -4777,15 +5011,21 @@ mod tests {
}
#[test]
fn test_get_text_defaults_to_body() {
// `get text` with no selector now returns the whole page (body) instead
// of erroring (issue #24-D).
fn test_get_text_defaults_to_all_frames() {
// `get text` with no selector now reads the whole page across ALL frames
// by default (#27), so iframed content isn't silently missed. (Was: a
// top-frame `body` read, #24-D.)
let cmd = parse_command(&args("get text"), &default_flags()).unwrap();
assert_eq!(cmd["action"], "gettext");
assert_eq!(cmd["selector"], "body");
// An explicit selector still wins.
assert_eq!(cmd["allFrames"], true);
assert!(cmd.get("selector").is_none());
// An explicit selector still wins and stays element-scoped.
let cmd2 = parse_command(&args("get text h1"), &default_flags()).unwrap();
assert_eq!(cmd2["selector"], "h1");
assert!(cmd2.get("allFrames").is_none());
// `text` top-level shortcut behaves the same.
let cmd3 = parse_command(&args("text"), &default_flags()).unwrap();
assert_eq!(cmd3["allFrames"], true);
}
#[test]
@@ -4813,6 +5053,24 @@ mod tests {
assert_eq!(cmd["action"], "frames");
}
#[test]
fn test_nearest_command_suggestions() {
assert_eq!(nearest_command("sesions").as_deref(), Some("sessions"));
assert_eq!(nearest_command("session").as_deref(), Some("sessions"));
assert_eq!(nearest_command("clik").as_deref(), Some("click"));
assert_eq!(
nearest_command("screenshits").as_deref(),
Some("screenshot")
);
// Nonsense with no close match stays silent.
assert_eq!(nearest_command("xyzzy"), None);
// The unknown-command error embeds the suggestion.
let err = ParseError::UnknownCommand {
command: "sesions".to_string(),
};
assert!(err.format().contains("Did you mean: chrome-use sessions?"));
}
#[test]
fn test_get_text_main() {
for variant in ["get text --main", "get text --readable", "text -m"] {
@@ -4823,6 +5081,16 @@ mod tests {
}
}
#[test]
fn test_get_text_pierce() {
for variant in ["get text --pierce", "get text --shadow", "text --deep"] {
let cmd = parse_command(&args(variant), &default_flags()).unwrap();
assert_eq!(cmd["action"], "gettext", "{variant}");
assert_eq!(cmd["pierce"], true, "{variant}");
assert!(cmd.get("selector").is_none(), "{variant}");
}
}
#[test]
fn test_tab_activate_flag() {
let plain = parse_command(&args("tab t3"), &default_flags()).unwrap();
@@ -5729,6 +5997,35 @@ mod tests {
assert_eq!(cmd["selector"], ".sidebar");
}
#[test]
fn test_scroll_at_coordinate() {
// `--at x,y` carries a [x, y] array for a wheel dispatched at that pixel
// (issue #36: cross-origin iframe scroll).
let cmd = parse_command(&args("scroll down 700 --at 640,400"), &default_flags()).unwrap();
assert_eq!(cmd["action"], "scroll");
assert_eq!(cmd["direction"], "down");
assert_eq!(cmd["amount"], 700);
assert_eq!(cmd["at"], json!([640.0, 400.0]));
}
#[test]
fn test_scroll_at_rejects_garbage() {
assert!(parse_command(&args("scroll --at nope"), &default_flags()).is_err());
assert!(parse_command(&args("scroll --at 1"), &default_flags()).is_err());
}
#[test]
fn test_scroll_frame_index() {
let cmd = parse_command(&args("scroll down 700 --frame 2"), &default_flags()).unwrap();
assert_eq!(cmd["action"], "scroll");
assert_eq!(cmd["frame"], 2);
}
#[test]
fn test_scroll_frame_rejects_non_integer() {
assert!(parse_command(&args("scroll --frame two"), &default_flags()).is_err());
}
#[test]
fn test_scroll_selector_before_positional() {
let cmd =
+41
View File
@@ -449,6 +449,47 @@ pub fn relay_url() -> Option<String> {
}
}
/// Append a one-line record of how a CDP connection was established, to
/// `~/.chrome-use/connect-mode.log`. This is the smoking-gun detector for the
/// "Allow remote debugging?" consent modal: that modal ONLY appears on a raw
/// remote-debugging attach / a browser we launched with a debug port — NEVER on
/// the extension relay. When the modal reappears, this log says which session
/// took which path and when, so we can tell a code regression (`raw-port` /
/// `launched` while the relay was up) from Chrome's own extension-debugger
/// consent UX. Low volume (one line per connection); best-effort, never fails a
/// connection.
pub fn log_connect_mode(ws_url: &str, launched: bool, session: &str) {
let relay = relay_url();
let relay_up = relay.is_some();
let mode = if launched {
"launched(debug-port)"
} else if relay.as_deref() == Some(ws_url) {
"relay"
} else if ws_url.contains("127.0.0.1") || ws_url.contains("localhost") {
"raw-port-attach"
} else {
"remote-ws"
};
// A raw-port attach or a self-launch while the relay was available is the
// exact thing that pops the consent modal — flag it loudly in the line.
let suspect = (mode == "raw-port-attach" || launched) && relay_up;
let line = format!(
"session={session} mode={mode} relay_up={relay_up}{} ws={ws_url}\n",
if suspect { " CONSENT-MODAL-RISK" } else { "" }
);
if let Some(home) = dirs::home_dir() {
let path = home.join(".chrome-use").join("connect-mode.log");
use std::io::Write;
if let Ok(mut f) = std::fs::OpenOptions::new()
.create(true)
.append(true)
.open(&path)
{
let _ = f.write_all(line.as_bytes());
}
}
}
/// Sidecar recording the connected extension's version, written by the host when
/// it receives the extension's `hello` (sibling of `relay-cdp-url`). Lets
/// `doctor` surface which extension build is live without a CDP round-trip.
+22
View File
@@ -893,6 +893,14 @@ fn main() {
return;
}
// `sessions` is a natural top-level guess for "list my sessions" (the skill
// advertises sessions as a feature) — route it to the daemon inventory the
// same way `daemon status` does (issue #29).
if clean.first().map(|s| s.as_str()) == Some("sessions") {
run_daemon(&["sessions".to_string(), "status".to_string()], flags.json);
return;
}
// Handle close --all: close all active sessions
if matches!(
clean.first().map(|s| s.as_str()),
@@ -1361,6 +1369,20 @@ fn main() {
&& flags.provider.is_none()
&& (flags.force_launch || !flags.auto_connect)
{
// Launching a debug-port Chrome pops Chrome's "Allow remote debugging?"
// consent modal (Chrome 136+). When the ab-connect relay is already up,
// this is almost always unintended — the relay drives the user's real
// Chrome with NO modal. Warn so the modal is self-explained and the
// caller (often a stray --launch / --no-auto-connect) is fixable (#32).
if !flags.json && connect::relay_url().is_some() {
eprintln!(
"{} launching a new Chrome with a debug port — this pops Chrome's \
\"Allow remote debugging?\" modal.\n The ab-connect relay is up; \
drop --launch/--new (and don't pass --no-auto-connect) to drive your \
real Chrome with no modal.",
color::warning_indicator()
);
}
let mut launch_cmd = json!({
"id": gen_id(),
"action": "launch",
+381 -12
View File
@@ -1341,6 +1341,7 @@ pub async fn execute_command(cmd: &Value, state: &mut DaemonState) -> Value {
"forward" => handle_forward(state).await,
"reload" => handle_reload(state).await,
"cookies_get" => handle_cookies_get(cmd, state).await,
"cf_status" => handle_cf_status(cmd, state).await,
"cookies_set" => handle_cookies_set(cmd, state).await,
"cookies_clear" => handle_cookies_clear(state).await,
"storage_get" => handle_storage_get(cmd, state).await,
@@ -2999,6 +3000,14 @@ async fn handle_screenshot(cmd: &Value, state: &mut DaemonState) -> Result<Value
.get("screenshotDir")
.and_then(|v| v.as_str())
.map(String::from),
clip: cmd.get("clip").and_then(|c| {
Some((
c.get("x")?.as_f64()?,
c.get("y")?.as_f64()?,
c.get("width")?.as_f64()?,
c.get("height")?.as_f64()?,
))
}),
};
if annotate {
@@ -3216,6 +3225,14 @@ async fn handle_type(cmd: &Value, state: &mut DaemonState) -> Result<Value, Stri
let mgr = state.browser.as_ref().ok_or("Browser not launched")?;
let session_id = mgr.active_session_id()?.to_string();
// `--key-events`: dispatch real per-character keyDown/keyUp instead of
// Input.insertText, so autocomplete/combobox widgets that only react to key
// events fire (e.g. Google's address postal-code lookup) (issue #4/#36).
let key_events = cmd
.get("keyEvents")
.and_then(|v| v.as_bool())
.unwrap_or(false);
// `type --focused <text>`: type into the currently-focused element without a
// selector (custom widgets that move focus to a hidden input on open).
if cmd
@@ -3227,7 +3244,14 @@ async fn handle_type(cmd: &Value, state: &mut DaemonState) -> Result<Value, Stri
.get("text")
.and_then(|v| v.as_str())
.ok_or("Missing 'text' parameter")?;
interaction::type_text_into_active_context(&mgr.client, &session_id, text, None).await?;
interaction::type_text_into_active_context(
&mgr.client,
&session_id,
text,
None,
key_events,
)
.await?;
return Ok(json!({ "typed": text, "focused": true }));
}
@@ -3251,6 +3275,7 @@ async fn handle_type(cmd: &Value, state: &mut DaemonState) -> Result<Value, Stri
clear,
delay,
&state.iframe_sessions,
key_events,
)
.await?;
Ok(json!({ "typed": text }))
@@ -3458,17 +3483,182 @@ async fn handle_scroll(cmd: &Value, state: &mut DaemonState) -> Result<Value, St
}
}
// An explicit `--selector` keeps the precise element-scroll path (scrollBy on
// the resolved node, same-origin only).
if let Some(sel) = selector {
interaction::scroll(
&mgr.client,
&session_id,
&state.ref_map,
Some(sel),
dx,
dy,
&state.iframe_sessions,
)
.await?;
return Ok(json!({ "scrolled": true, "via": "selector" }));
}
// `--at x,y` / `--frame n`: dispatch a real (isTrusted) wheel at a viewport
// coordinate. This hits the compositor and scrolls whatever scroll container
// is under the pointer — including cross-origin iframes that `window.scrollBy`
// on the top document silently no-ops on (issue #36).
if cmd.get("at").is_some() || cmd.get("frame").is_some() {
let (x, y, via) = if let Some(at) = cmd.get("at").and_then(|v| v.as_array()) {
let x = at.first().and_then(|v| v.as_f64()).unwrap_or(0.0);
let y = at.get(1).and_then(|v| v.as_f64()).unwrap_or(0.0);
(x, y, "at")
} else {
let n = cmd.get("frame").and_then(|v| v.as_u64()).unwrap_or(0);
let (x, y) = frame_center(mgr, &session_id, &state.iframe_sessions, n as usize).await?;
(x, y, "frame")
};
dispatch_wheel(&mgr.client, &session_id, x, y, dx, dy).await?;
return Ok(json!({ "scrolled": true, "via": via, "at": [x, y] }));
}
// Default (no selector/at/frame): scroll the page with `window.scrollBy`. This
// is the reliable path for ordinary page scrolling; a coordinate wheel at the
// viewport centre is NOT a dependable substitute (it no-ops on some pages,
// e.g. headless), so the wheel stays opt-in via `--at`/`--frame` for the
// cross-origin-iframe case (issue #36).
interaction::scroll(
&mgr.client,
&session_id,
&state.ref_map,
selector,
None,
dx,
dy,
&state.iframe_sessions,
)
.await?;
Ok(json!({ "scrolled": true }))
Ok(json!({ "scrolled": true, "via": "page" }))
}
/// Viewport center in CSS pixels, used as the default wheel landing point for
/// `scroll` (issue #36). Falls back to a sane 640×400 center if the page can't
/// be evaluated (e.g. a restricted document).
async fn viewport_center(mgr: &BrowserManager, session_id: &str) -> Result<(f64, f64), String> {
let dims = mgr
.client
.send_command_typed::<_, Value>(
"Runtime.evaluate",
&super::cdp::types::EvaluateParams {
expression: "[window.innerWidth, window.innerHeight]".to_string(),
return_by_value: Some(true),
await_promise: Some(false),
},
Some(session_id),
)
.await
.ok();
let arr = dims
.as_ref()
.and_then(|v| v.get("result"))
.and_then(|v| v.get("value"))
.and_then(|v| v.as_array());
let w = arr
.and_then(|a| a.first())
.and_then(|v| v.as_f64())
.filter(|w| *w > 0.0)
.unwrap_or(1280.0);
let h = arr
.and_then(|a| a.get(1))
.and_then(|v| v.as_f64())
.filter(|h| *h > 0.0)
.unwrap_or(800.0);
Ok((w / 2.0, h / 2.0))
}
/// Center of the `n`-th frame (as listed by `chrome-use frames`) in top-viewport
/// CSS pixels, so `scroll --frame n` lands its wheel inside a cross-origin iframe
/// without needing a selector into it (issue #36). Resolves the frame's owning
/// `<iframe>` element box via `DOM.getFrameOwner` + `DOM.getBoxModel` — exact for
/// a frame nested directly under the top document; for a deeper nesting the box is
/// relative to the intermediate frame, so prefer `--at x,y` from a screenshot.
async fn frame_center(
mgr: &BrowserManager,
session_id: &str,
iframe_sessions: &HashMap<String, String>,
n: usize,
) -> Result<(f64, f64), String> {
let frames =
super::element::collect_all_frames_text(&mgr.client, session_id, iframe_sessions).await?;
let frame = frames.get(n).ok_or_else(|| {
format!(
"frame index {} out of range (run `chrome-use frames`: {} frame(s))",
n,
frames.len()
)
})?;
if n == 0 {
// Frame 0 is the top document — there's no owner element; scroll its center.
return viewport_center(mgr, session_id).await;
}
let owner = mgr
.client
.send_command_typed::<_, Value>(
"DOM.getFrameOwner",
&json!({ "frameId": frame.frame_id }),
Some(session_id),
)
.await
.map_err(|e| format!("can't locate frame {}'s owner element: {}", n, e))?;
let backend_node_id = owner
.get("backendNodeId")
.and_then(|v| v.as_i64())
.ok_or_else(|| format!("frame {} has no owner <iframe> element", n))?;
let box_model = mgr
.client
.send_command_typed::<_, Value>(
"DOM.getBoxModel",
&json!({ "backendNodeId": backend_node_id }),
Some(session_id),
)
.await
.map_err(|e| format!("can't measure frame {}'s box: {}", n, e))?;
let content = box_model
.get("model")
.and_then(|m| m.get("content"))
.and_then(|c| c.as_array())
.ok_or_else(|| format!("frame {} box model has no content quad", n))?;
let coord = |i: usize| content.get(i).and_then(|v| v.as_f64()).unwrap_or(0.0);
// content quad is [x1,y1, x2,y2, x3,y3, x4,y4]; opposite corners are 0 and 2.
let cx = (coord(0) + coord(4)) / 2.0;
let cy = (coord(1) + coord(5)) / 2.0;
Ok((cx, cy))
}
/// Dispatch a trusted mouse wheel at `(x, y)`, humanized like `handle_wheel`.
async fn dispatch_wheel(
client: &super::cdp::client::CdpClient,
session_id: &str,
x: f64,
y: f64,
delta_x: f64,
delta_y: f64,
) -> Result<(), String> {
let level = humanize::active_level();
let seed = humanize::next_seed();
for (dx, dy, delay) in humanize::scroll_segments(delta_x, delta_y, level, seed) {
client
.send_command(
"Input.dispatchMouseEvent",
Some(json!({
"type": "mouseWheel",
"x": x,
"y": y,
"deltaX": dx,
"deltaY": dy,
})),
Some(session_id),
)
.await?;
if !delay.is_zero() {
tokio::time::sleep(delay).await;
}
}
Ok(())
}
async fn handle_select(cmd: &Value, state: &mut DaemonState) -> Result<Value, String> {
@@ -3628,6 +3818,15 @@ async fn handle_gettext(cmd: &Value, state: &mut DaemonState) -> Result<Value, S
}));
}
// `get text --pierce` reads text through CLOSED shadow roots and child
// documents via the CDP DOM tree — content `innerText`/`eval` can't see,
// e.g. an extension's injected panel in a closed shadow DOM (#30).
if cmd.get("pierce").and_then(|v| v.as_bool()) == Some(true) {
let text = super::element::get_pierced_text(&mgr.client, &session_id).await?;
let url = mgr.get_url().await.unwrap_or_default();
return Ok(json!({ "text": text, "origin": url, "pierce": true }));
}
// `get text --main` returns the page's main-content region (readability-lite),
// skipping global header/nav/footer/sidebar boilerplate (#27).
if cmd.get("main").and_then(|v| v.as_bool()) == Some(true) {
@@ -3656,12 +3855,9 @@ async fn handle_gettext(cmd: &Value, state: &mut DaemonState) -> Result<Value, S
async fn handle_frames(_cmd: &Value, state: &mut DaemonState) -> Result<Value, String> {
let mgr = state.browser.as_ref().ok_or("Browser not launched")?;
let session_id = mgr.active_session_id()?.to_string();
let frames = super::element::collect_all_frames_text(
&mgr.client,
&session_id,
&state.iframe_sessions,
)
.await?;
let frames =
super::element::collect_all_frames_text(&mgr.client, &session_id, &state.iframe_sessions)
.await?;
let list: Vec<Value> = frames
.iter()
.enumerate()
@@ -4113,6 +4309,111 @@ async fn handle_cookies_clear(state: &DaemonState) -> Result<Value, String> {
Ok(json!({ "cleared": true }))
}
// Detect whether the active page is *currently* a Cloudflare challenge
// (the full-page "Just a moment…" / "正在进行安全验证" interstitial), so an agent
// knows whether it must solve or can proceed. Runs in the top frame main world.
const CF_CHALLENGE_JS: &str = r#"(function(){
var t = document.title || '';
var challenged =
/just a moment|attention required|checking (your|if)|verify you are human|||||/i.test(t) ||
!!document.querySelector('#challenge-form, #challenge-running, #cf-challenge-running, [id^="cf-chl"], script[src*="/cdn-cgi/challenge-platform/"]');
var turnstile = !!document.querySelector('.cf-turnstile, [data-sitekey]');
return JSON.stringify({ title: t, challenged: challenged, turnstile: turnstile, readyState: document.readyState });
})()"#;
/// Recommendation for a Cloudflare-gated page, from the current challenge state
/// and whether a still-valid `cf_clearance` exists. Pure so it's unit-testable.
/// - not challenged → "proceed" (the page is cleared/loaded)
/// - challenged, valid cookie → "reissue" (clearance present but page still
/// blocks → it's stale or the IP/UA no longer matches what it was issued for)
/// - challenged, no cookie → "solve"
fn cf_recommendation(challenged: bool, clearance_valid: bool) -> &'static str {
if !challenged {
"proceed"
} else if clearance_valid {
"reissue"
} else {
"solve"
}
}
/// `cf_clearance` validity for a cookie's expiry (epoch seconds; <=0 = session
/// cookie, treated as non-expiring). Returns (present, expired). Pure.
fn clearance_state(expires: Option<f64>, now: f64) -> (bool, bool) {
match expires {
None => (false, false),
Some(e) if e <= 0.0 => (true, false), // session cookie: no expiry
Some(e) => (true, e < now),
}
}
async fn handle_cf_status(_cmd: &Value, state: &mut DaemonState) -> Result<Value, String> {
let mgr = state.browser.as_ref().ok_or("Browser not launched")?;
let session_id = mgr.active_session_id()?.to_string();
let url = mgr.get_url().await.unwrap_or_default();
// 1. Is the page a Cloudflare challenge right now?
let probe_raw = mgr
.evaluate(CF_CHALLENGE_JS, None)
.await
.unwrap_or(Value::Null);
let probe = parse_json_string(probe_raw, "cf challenge probe").unwrap_or(Value::Null);
let challenged = probe
.get("challenged")
.and_then(|v| v.as_bool())
.unwrap_or(false);
let turnstile = probe
.get("turnstile")
.and_then(|v| v.as_bool())
.unwrap_or(false);
let title = probe
.get("title")
.and_then(|v| v.as_str())
.unwrap_or("")
.to_string();
// 2. Persistence artifacts: cf_clearance (HttpOnly → must read via CDP, not
// document.cookie) + CF_VERIFIED_DEVICE. Scope to the current URL.
let urls = if url.is_empty() {
None
} else {
Some(vec![url.clone()])
};
let cookies = super::cookies::get_cookies(&mgr.client, &session_id, urls)
.await
.unwrap_or_default();
let clearance = cookies.iter().find(|c| c.name == "cf_clearance");
let now = std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.map(|d| d.as_secs_f64())
.unwrap_or(0.0);
let (present, expired) = clearance_state(clearance.map(|c| c.expires), now);
let expires_in = clearance
.filter(|_| present && !expired)
.map(|c| (c.expires - now).max(0.0) as i64);
let device_verified = cookies
.iter()
.any(|c| c.name.starts_with("CF_VERIFIED_DEVICE"));
let clearance_valid = present && !expired;
let recommendation = cf_recommendation(challenged, clearance_valid);
Ok(json!({
"url": url,
"title": title,
"challenged": challenged,
"turnstile": turnstile,
"clearance": {
"present": present,
"expired": expired,
"expiresIn": expires_in,
"httpOnly": clearance.map(|c| c.http_only).unwrap_or(false),
},
"deviceVerified": device_verified,
"recommendation": recommendation,
}))
}
async fn handle_storage_get(cmd: &Value, state: &DaemonState) -> Result<Value, String> {
let mgr = state.browser.as_ref().ok_or("Browser not launched")?;
let session_id = mgr.active_session_id()?.to_string();
@@ -4450,8 +4751,18 @@ async fn handle_keyboard(cmd: &Value, state: &DaemonState) -> Result<Value, Stri
.get("text")
.and_then(|v| v.as_str())
.ok_or("Missing 'text' parameter")?;
interaction::type_text_into_active_context(&mgr.client, &session_id, text, None)
.await?;
let key_events = cmd
.get("keyEvents")
.and_then(|v| v.as_bool())
.unwrap_or(false);
interaction::type_text_into_active_context(
&mgr.client,
&session_id,
text,
None,
key_events,
)
.await?;
return Ok(json!({ "typed": text }));
}
Some("insertText") => {
@@ -4561,7 +4872,22 @@ async fn handle_tab_switch(cmd: &Value, state: &mut DaemonState) -> Result<Value
state.ref_map.clear();
state.iframe_sessions.clear();
state.active_frame_id = None;
let result = mgr.tab_switch_by_id(tab_id).await?;
let mut result = mgr.tab_switch_by_id(tab_id).await?;
// Liveness probe: confirm the new session actually answers before we report
// success, so `tab <id>` doesn't print a misleading ✓ for a session that's
// stale and will fail on the very next command (issue #29.3). On the churned
// -tabId case the ext-0.4.9 targetId recovery (#24) self-heals within ~6s, so
// we surface a warning rather than a hard error to avoid a false failure
// during that window.
if mgr.evaluate("1", None).await.is_err() {
if let Some(obj) = result.as_object_mut() {
obj.insert(
"warning".to_string(),
json!("switched tab is not responding yet (session re-attaching); retry the next command"),
);
}
}
// `--activate`: raise this tab to the foreground (the switch made it active;
// bring_to_front acts on the active tab) — for handing a specific tab to the
@@ -6965,6 +7291,28 @@ async fn handle_drag(cmd: &Value, state: &mut DaemonState) -> Result<Value, Stri
.and_then(|v| v.as_str())
.ok_or("Missing 'target' parameter")?;
// Over the relay (or into an iframe) a coordinate drag drifts to the
// foreground tab and can't reach an OOPIF — DOM-dispatch an HTML5 drag in the
// element's own session instead (issues #31/#36). `coord` mode forces the
// coordinate path for pointer-driven drags (canvas/sliders) on a launched
// browser.
if std::env::var("AGENT_BROWSER_CLICK_MODE").as_deref() != Ok("coord")
&& (crate::connect::relay_url().is_some()
|| state.ref_map.ref_is_in_iframe(source)
|| state.ref_map.ref_is_in_iframe(target))
{
super::interaction::dom_drag(
&mgr.client,
&session_id,
&state.ref_map,
source,
target,
&state.iframe_sessions,
)
.await?;
return Ok(json!({ "dragged": { "source": source, "target": target }, "via": "dom" }));
}
let (sx, sy, _, _, source_session_id) = super::element::resolve_element_center(
&mgr.client,
&session_id,
@@ -7319,6 +7667,7 @@ async fn handle_diff_screenshot(cmd: &Value, state: &DaemonState) -> Result<Valu
quality: None,
annotate: false,
output_dir: None,
clip: None,
};
let result = screenshot::take_screenshot(
@@ -9045,6 +9394,26 @@ mod tests {
use crate::test_utils::EnvGuard;
use std::fs;
#[test]
fn test_cf_recommendation() {
assert_eq!(cf_recommendation(false, false), "proceed");
assert_eq!(cf_recommendation(false, true), "proceed");
assert_eq!(cf_recommendation(true, false), "solve");
assert_eq!(cf_recommendation(true, true), "reissue");
}
#[test]
fn test_clearance_state() {
// no cookie
assert_eq!(clearance_state(None, 1000.0), (false, false));
// session cookie (expires <= 0) → present, never expired
assert_eq!(clearance_state(Some(-1.0), 1000.0), (true, false));
// valid: expiry in the future
assert_eq!(clearance_state(Some(2000.0), 1000.0), (true, false));
// expired: expiry in the past
assert_eq!(clearance_state(Some(500.0), 1000.0), (true, true));
}
#[test]
fn test_url_glob_to_regex() {
assert_eq!(url_glob_to_regex("**/dashboard"), "^.*/dashboard$");
+588 -58
View File
@@ -121,7 +121,7 @@ fn normalize_url_for_match(url: &str) -> String {
fn update_page_target_info_in_pages(pages: &mut [PageInfo], target: &TargetInfo) -> bool {
if let Some(page) = pages.iter_mut().find(|p| p.target_id == target.target_id) {
page.url = target.url.clone();
page.title = target.title.clone();
page.title = sanitize_title(&target.title);
page.target_type = target.target_type.clone();
return true;
}
@@ -166,6 +166,112 @@ fn resolve_active_index(
active_page_index
}
/// Strip zero-width / invisible / bidi-format Unicode from a page title before
/// we store it. Some sites prepend runs of ZWJ / word-joiner / invisible-times /
/// BOM to `document.title` (badging, watermarking, anti-scrape); left in, they
/// pollute `tab list`, break text matching, and wreck column alignment (#33).
fn sanitize_title(s: &str) -> String {
s.chars()
.filter(|&c| {
!matches!(c as u32,
0x00AD // soft hyphen
| 0x200B..=0x200F // ZWSP, ZWNJ, ZWJ, LRM, RLM
| 0x2028 | 0x2029 // line / paragraph separators
| 0x202A..=0x202E // bidi embedding/override
| 0x2060..=0x2064 // word joiner, invisible operators
| 0x2066..=0x2069 // bidi isolates
| 0x180E // Mongolian vowel separator
| 0xFEFF // BOM / ZW no-break space
)
})
.collect::<String>()
.trim()
.to_string()
}
/// Best-effort MIME type from a filename extension, for the relay file-upload
/// fallback (the page-constructed `File` needs a sensible `type`). Covers the
/// common upload kinds; anything unknown falls back to a generic binary type.
fn mime_for_path(name: &str) -> &'static str {
let ext = name.rsplit('.').next().unwrap_or("").to_lowercase();
match ext.as_str() {
"png" => "image/png",
"jpg" | "jpeg" => "image/jpeg",
"gif" => "image/gif",
"webp" => "image/webp",
"svg" => "image/svg+xml",
"bmp" => "image/bmp",
"pdf" => "application/pdf",
"txt" => "text/plain",
"csv" => "text/csv",
"json" => "application/json",
"mp4" => "video/mp4",
"webm" => "video/webm",
"mov" => "video/quicktime",
"mp3" => "audio/mpeg",
"zip" => "application/zip",
_ => "application/octet-stream",
}
}
/// Target ids to prune after a `Target.getTargets` resync: tracked pages whose
/// target is no longer in the live set — EXCEPT the explicitly-pinned active
/// target, which is protected. The relay against a busy real Chrome occasionally
/// returns a different window's tabs for a single `getTargets` call ("tab list
/// hops windows", issue #31); pruning on that transient snapshot would drop the
/// agent's adopted tab and drift subsequent eval/click onto a foreign tab. A
/// genuine close still arrives as `Target.targetDestroyed` (handled in the event
/// drain), which removes the pin properly — so protecting it here only guards
/// against flaky snapshots, not real closures.
fn prunable_target_ids(
pages: &[PageInfo],
live_ids: &HashSet<String>,
pinned: Option<&str>,
) -> Vec<String> {
pages
.iter()
.map(|p| p.target_id.clone())
.filter(|tid| !live_ids.contains(tid) && pinned != Some(tid.as_str()))
.collect()
}
/// Consecutive missing `getTargets` snapshots before an owned relay tab is
/// pruned. >1 so a single churning/partial snapshot (other agents opening/closing
/// tabs) or a brief cross-process-nav gap can't drop the tab the agent is driving.
const RELAY_PRUNE_MISSES: u32 = 3;
/// Debounced prune for the relay: target ids to drop, mutating per-target miss
/// counters. A tab in `live_ids` resets to 0; an absent (non-pinned) tab
/// increments and is pruned only at `RELAY_PRUNE_MISSES`. Counters for
/// no-longer-tracked targets are forgotten. Pure, so the multi-agent churn
/// tolerance is unit-testable without a live browser.
fn debounced_prune_ids(
pages: &[PageInfo],
live_ids: &HashSet<String>,
pinned: Option<&str>,
misses: &mut HashMap<String, u32>,
) -> Vec<String> {
let tracked: HashSet<&str> = pages.iter().map(|p| p.target_id.as_str()).collect();
misses.retain(|tid, _| tracked.contains(tid.as_str()));
let mut prune = Vec::new();
for p in pages {
let tid = p.target_id.as_str();
if live_ids.contains(tid) {
misses.remove(tid);
continue;
}
if pinned == Some(tid) {
continue;
}
let c = misses.entry(p.target_id.clone()).or_insert(0);
*c += 1;
if *c >= RELAY_PRUNE_MISSES {
prune.push(p.target_id.clone());
}
}
prune
}
/// Whether the resolved active page is a tab the session created (its target_id
/// is in `created_targets`). Pure core of [`BrowserManager::active_is_session_owned`]
/// so the relay no-hijack rule is unit-testable without a live browser.
@@ -185,6 +291,21 @@ fn active_index_is_owned(
.unwrap_or(false)
}
/// Whether a CDP error means the bound relay target is gone — the tab was
/// closed, navigated across processes (renderer swap), or lost after an
/// extension/service-worker restart, and the relay could not re-attach. The
/// ab-connect relay surfaces these as `stale sessionId … its tab is gone`,
/// `unknown sessionId …`, or `no attached tab …`. `navigate` keys its
/// auto-reattach recovery off this (issue #35) so a dead session rebinds to a
/// fresh tab instead of erroring on every command until the user runs `tab new`.
fn is_stale_target_error(error: &str) -> bool {
let lower = error.to_lowercase();
lower.contains("its tab is gone")
|| lower.contains("stale sessionid")
|| lower.contains("unknown sessionid")
|| lower.contains("no attached tab")
}
/// Converts common error messages into AI-friendly, actionable descriptions.
pub fn to_ai_friendly_error(error: &str) -> String {
let lower = error.to_lowercase();
@@ -380,6 +501,14 @@ pub struct BrowserManager {
/// the session's commands onto the wrong page — the wrong-origin-fetch hazard
/// in the dogfood reports. Falls back to the index if the pinned tab is gone.
active_target_id: Option<String>,
/// Per-target count of CONSECUTIVE `resync_targets` snapshots in which an
/// owned tab was missing from `Target.getTargets`. Over the relay a single
/// snapshot routinely omits live tabs (multi-agent churn, a cross-process nav
/// briefly dropping the target), so we must not prune on one miss — that lost
/// the tab the agent was driving. A tab is removed only after it's been absent
/// for `RELAY_PRUNE_MISSES` consecutive snapshots; any snapshot that includes
/// it resets the counter. Keyed by stable target_id.
relay_target_misses: HashMap<String, u32>,
next_tab_id: u32,
/// Whether to enable the CDP `Runtime` domain (console / error / exception capture).
/// OFF by default for stealth: a live `Runtime.enable` is a detectable CDP signal
@@ -490,6 +619,16 @@ impl BrowserManager {
}
};
// A launched browser carries a debug port → it's the other path that can
// pop Chrome's consent modal; record it for #31 diagnosis.
crate::connect::log_connect_mode(
&ws_url,
true,
DAEMON_SESSION
.get()
.map(String::as_str)
.unwrap_or("default"),
);
let manager = if engine == "lightpanda" {
initialize_lightpanda_manager(ws_url, process).await?
} else {
@@ -506,6 +645,7 @@ impl BrowserManager {
visited_origins: HashSet::new(),
created_targets: HashSet::new(),
active_target_id: None,
relay_target_misses: HashMap::new(),
next_tab_id: 1,
capture_console: console_capture_enabled(),
};
@@ -585,6 +725,16 @@ impl BrowserManager {
headers: Option<Vec<(String, String)>>,
) -> Result<Self, String> {
let ws_url = resolve_cdp_url(url).await?;
// Record the transport so a reappearing "Allow remote debugging?" modal
// can be traced to a raw-port attach vs the consent-free relay (#31).
crate::connect::log_connect_mode(
&ws_url,
false,
DAEMON_SESSION
.get()
.map(String::as_str)
.unwrap_or("default"),
);
let client = Arc::new(CdpClient::connect_with_headers(&ws_url, headers).await?);
let mut manager = Self {
client,
@@ -598,6 +748,7 @@ impl BrowserManager {
visited_origins: HashSet::new(),
created_targets: HashSet::new(),
active_target_id: None,
relay_target_misses: HashMap::new(),
next_tab_id: 1,
capture_console: console_capture_enabled(),
};
@@ -627,6 +778,43 @@ impl BrowserManager {
Self::connect_cdp(&ws_url).await
}
/// Page targets to adopt, merging several `Target.getTargets` snapshots over
/// the extension relay. A single relay snapshot is flaky on a busy real Chrome
/// — it can omit live tabs (a different window's set, or a partial list; issue
/// #31) — so a tab the daemon should adopt would silently vanish (e.g. after a
/// daemon restart the page being driven disappeared from the tab list). Taking
/// the union of a few snapshots makes adoption resilient to a transient miss.
/// Off the relay (a browser we launched) one snapshot is authoritative.
async fn collect_page_targets(&self) -> Result<Vec<TargetInfo>, String> {
let rounds = if crate::connect::relay_url().is_some() {
3
} else {
1
};
let mut by_id: HashMap<String, TargetInfo> = HashMap::new();
let mut any_ok = false;
for i in 0..rounds {
if i > 0 {
tokio::time::sleep(Duration::from_millis(150)).await;
}
match self
.client
.send_command_typed::<_, GetTargetsResult>("Target.getTargets", &json!({}), None)
.await
{
Ok(result) => {
any_ok = true;
for t in result.target_infos.into_iter().filter(should_track_target) {
by_id.entry(t.target_id.clone()).or_insert(t);
}
}
Err(e) if i == rounds - 1 && !any_ok => return Err(e),
Err(_) => {}
}
}
Ok(by_id.into_values().collect())
}
async fn discover_and_attach_targets(&mut self) -> Result<(), String> {
self.client
.send_command_typed::<_, Value>(
@@ -636,16 +824,7 @@ impl BrowserManager {
)
.await?;
let result: GetTargetsResult = self
.client
.send_command_typed("Target.getTargets", &json!({}), None)
.await?;
let page_targets: Vec<TargetInfo> = result
.target_infos
.into_iter()
.filter(should_track_target)
.collect();
let page_targets: Vec<TargetInfo> = self.collect_page_targets().await?;
if page_targets.is_empty() {
// Create a new tab
@@ -691,7 +870,19 @@ impl BrowserManager {
self.active_page_index = 0;
self.pin_active_target();
self.enable_domains(&attach_result.session_id).await?;
} else if self.agent_group().is_some() {
// STRICT MULTI-AGENT ISOLATION (relay / the user's real Chrome).
// `page_targets` here are the USER's and OTHER agents' tabs. A tab
// group belongs to exactly ONE agent, so this session must NOT adopt
// any of them — it tracks ONLY tabs it creates (its own colored group)
// plus popups it opens. Adopting foreign tabs is precisely what let
// another concurrent agent's tab churn drop the tab we were driving and
// drift eval/click onto the wrong page (multi-agent failure). Open our
// own dedicated background tab in the session's group and pin it; the
// user's / other agents' tabs stay invisible to us.
self.tab_new(None, None).await?;
} else {
// A browser WE launched: every tab is ours, so adopt them all.
for target in &page_targets {
let attach_result: AttachToTargetResult = self
.client
@@ -713,11 +904,10 @@ impl BrowserManager {
target_id: target.target_id.clone(),
session_id: attach_result.session_id.clone(),
url: target.url.clone(),
title: target.title.clone(),
title: sanitize_title(&target.title),
target_type: target.target_type.clone(),
});
}
self.active_page_index = 0;
self.pin_active_target();
let session_id = self.pages[0].session_id.clone();
@@ -832,6 +1022,24 @@ impl BrowserManager {
)
}
/// Drop the page bound to `session_id` from the tracked list — used when the
/// relay reports its tab is gone (issue #35) so the stale entry can't keep
/// resolving as active. Forgets ownership, unpins it if it was pinned, and
/// keeps `active_page_index` in range.
fn drop_page_by_session(&mut self, session_id: &str) {
let Some(pos) = self.pages.iter().position(|p| p.session_id == session_id) else {
return;
};
let target_id = self.pages[pos].target_id.clone();
self.pages.remove(pos);
self.created_targets.remove(&target_id);
if self.active_target_id.as_deref() == Some(target_id.as_str()) {
self.active_target_id = None;
}
self.active_page_index =
active_page_index_after_removal(self.active_page_index, pos, self.pages.len());
}
/// Pin the current active page by target_id so later commands stick to it.
/// Call after any explicit open / tab new / tab switch.
fn pin_active_target(&mut self) {
@@ -861,10 +1069,10 @@ impl BrowserManager {
if self.agent_group().is_some() && !self.active_is_session_owned() {
self.tab_new(None, None).await?;
}
let session_id = self.active_session_id()?.to_string();
let mut session_id = self.active_session_id()?.to_string();
let mut lifecycle_rx = self.client.subscribe();
let nav_result: PageNavigateResult = self
let nav_result: PageNavigateResult = match self
.client
.send_command_typed(
"Page.navigate",
@@ -874,7 +1082,38 @@ impl BrowserManager {
},
Some(&session_id),
)
.await?;
.await
{
Ok(r) => r,
// Auto-reattach when the bound tab is gone (issue #35). On the shared
// real browser the human can close/swap the agent's tab, and a
// cross-process nav can destroy the target without a re-attachable
// tabId — both leave the cached `cb-tab-<id>` session stale, so every
// command (including `open`) failed on it and only `tab new`
// recovered. The relay error literally says "re-open your target URL
// to re-attach"; fulfil that here: drop the dead page, open a fresh
// owned tab in this session's group, and navigate THAT. Gated on the
// relay (`agent_group`) and on the explicit navigation intent — read
// commands deliberately still fail loudly rather than silently
// recover onto a blank tab and return wrong data (issue #8.1).
Err(e) if self.agent_group().is_some() && is_stale_target_error(&e) => {
self.drop_page_by_session(&session_id);
self.tab_new(None, None).await?;
session_id = self.active_session_id()?.to_string();
lifecycle_rx = self.client.subscribe();
self.client
.send_command_typed(
"Page.navigate",
&PageNavigateParams {
url: url.to_string(),
referrer: None,
},
Some(&session_id),
)
.await?
}
Err(e) => return Err(e),
};
if let Some(ref error_text) = nav_result.error_text {
return Err(format!("Navigation failed: {}", error_text));
@@ -939,7 +1178,7 @@ impl BrowserManager {
self.active_page_index = self.resolved_active_index();
if let Some(page) = self.pages.get_mut(self.active_page_index) {
page.url = page_url.clone();
page.title = title.clone();
page.title = sanitize_title(&title);
}
self.pin_active_target();
@@ -1001,7 +1240,7 @@ impl BrowserManager {
pub async fn get_title(&self) -> Result<String, String> {
let result = self.evaluate_simple("document.title").await?;
Ok(result.as_str().unwrap_or("").to_string())
Ok(sanitize_title(result.as_str().unwrap_or("")))
}
pub async fn get_content(&self) -> Result<String, String> {
@@ -1014,15 +1253,31 @@ impl BrowserManager {
pub async fn evaluate(&self, script: &str, _args: Option<Value>) -> Result<Value, String> {
let session_id = self.active_session_id()?.to_string();
// `replMode: true` lets successive `eval`s re-declare top-level
// `let`/`const` instead of throwing "Identifier 'x' has already been
// declared" (issue #38 — independent `eval` steps in a test suite collided
// in the page's shared lexical scope). BUT replMode and `awaitPromise` are
// mutually exclusive in Chrome: under replMode a returned promise is NOT
// awaited (it serialises to `{}`), which breaks `fetch(...).then(...)` and
// every other async eval. So enable replMode ONLY for synchronous scripts
// that declare a top-level `let`/`const`; promise-returning scripts keep
// `awaitPromise` (no replMode) — exactly the pre-#38 behaviour.
let mentions_async = script.contains("await")
|| script.contains(".then(")
|| script.contains("fetch(")
|| script.contains("Promise");
let declares = script.contains("let ") || script.contains("const ");
let repl_mode = declares && !mentions_async;
let result: EvaluateResult = self
.client
.send_command_typed(
"Runtime.evaluate",
&EvaluateParams {
expression: script.to_string(),
return_by_value: Some(true),
await_promise: Some(true),
},
&json!({
"expression": script,
"returnByValue": true,
"awaitPromise": !repl_mode,
"replMode": repl_mode,
}),
Some(&session_id),
)
.await?;
@@ -1305,6 +1560,18 @@ impl BrowserManager {
/// the active tab, per #7/#8.1); the caller surfaces it so the agent knows a
/// tab opened instead of seeing the old page (issue #24-A).
pub async fn adopt_newly_opened(&mut self, before: &HashSet<String>) -> Option<PageInfo> {
// STRICT MULTI-AGENT ISOLATION: on the relay this session's `before` set is
// only its OWN tabs, so EVERY foreign tab (the user's, other agents') looks
// "new" relative to it and would be adopted here — exactly the leak where a
// concurrent agent's tabs (github/Lark/iphone-use) showed up in this
// session mid-flow. A tab the agent itself opened (a pop-up) can't be
// distinguished from a foreign tab over the relay (no opener/window/group
// in the synthesized targetInfo), so don't adopt anything: the agent drives
// only tabs it explicitly created, and pop-ups (e.g. an OAuth/login window)
// are the user's. A launched browser (every tab ours) still follows pop-ups.
if self.agent_group().is_some() {
return None;
}
let result: GetTargetsResult = self
.client
.send_command_typed("Target.getTargets", &json!({}), None)
@@ -1344,9 +1611,14 @@ impl BrowserManager {
target_id: target.target_id.clone(),
session_id: attach.session_id.clone(),
url: target.url.clone(),
title: target.title.clone(),
title: sanitize_title(&target.title),
target_type: target.target_type.clone(),
};
// A tab that appeared right after THIS session's action (a click that
// opened a popup/new tab) is ours — record it as owned so it's tracked,
// protected from churn-pruning, and cleaned up on close, consistent with
// strict multi-agent isolation (we only ever own tabs we created/opened).
self.created_targets.insert(target.target_id.clone());
self.add_background_page(page.clone());
let _ = self.enable_domains(&attach.session_id).await;
if opened.is_none() {
@@ -1374,13 +1646,19 @@ impl BrowserManager {
.filter(should_track_target)
.collect();
let live_ids: HashSet<String> = live.iter().map(|t| t.target_id.clone()).collect();
let on_relay = self.agent_group().is_some();
for target in &live {
if self.update_page_target_info(target) {
continue;
}
// A target this session hasn't tracked yet — attach and add it in the
// background so it's listable/adoptable without stealing the active tab.
// STRICT MULTI-AGENT ISOLATION: on the relay (the user's real Chrome,
// shared with other agents), NEVER adopt a tab this session didn't
// create — it belongs to the user or another agent's group. Only a
// browser we launched (every tab ours) adopts unknown targets.
if on_relay {
continue;
}
let attach_result: AttachToTargetResult = match self
.client
.send_command_typed(
@@ -1405,21 +1683,32 @@ impl BrowserManager {
target_id: target.target_id.clone(),
session_id: attach_result.session_id.clone(),
url: target.url.clone(),
title: target.title.clone(),
title: sanitize_title(&target.title),
target_type: target.target_type.clone(),
});
let _ = self.enable_domains(&attach_result.session_id).await;
}
// Drop tabs that no longer exist so `tab list` doesn't show phantom rows.
let gone: Vec<String> = self
.pages
.iter()
.map(|p| p.target_id.clone())
.filter(|tid| !live_ids.contains(tid))
.collect();
for tid in gone {
self.remove_page_by_target_id(&tid);
// Prune tabs that are gone. On a LAUNCHED browser a missing target really
// is closed, so prune immediately. On the RELAY a single `getTargets`
// snapshot routinely omits live tabs (multi-agent churn, a brief
// cross-process-nav gap) — dropping the tab we're driving on one bad
// snapshot is the failure we're fixing — so prune only after the tab has
// been absent for several CONSECUTIVE snapshots (debounced). The pinned
// active target is protected either way (issue #31).
let gone = if on_relay {
debounced_prune_ids(
&self.pages,
&live_ids,
self.active_target_id.as_deref(),
&mut self.relay_target_misses,
)
} else {
prunable_target_ids(&self.pages, &live_ids, self.active_target_id.as_deref())
};
for tid in &gone {
self.relay_target_misses.remove(tid);
self.remove_page_by_target_id(tid);
}
// Refresh url/title from each live tab. The relay only stamps target_info
@@ -1451,7 +1740,7 @@ impl BrowserManager {
}
}
if let Some(t) = ti.get("title").and_then(|v| v.as_str()) {
page.title = t.to_string();
page.title = sanitize_title(t);
}
}
}
@@ -1639,7 +1928,7 @@ impl BrowserManager {
if let Some(page) = self.pages.get_mut(index) {
page.url = url.clone();
page.title = title.clone();
page.title = sanitize_title(&title);
}
let page = &self.pages[index];
@@ -1894,7 +2183,8 @@ impl BrowserManager {
.and_then(|v| v.as_i64())
.ok_or("Could not get backendNodeId for file input")?;
self.client
let set_files = self
.client
.send_command(
"DOM.setFileInputFiles",
Some(json!({
@@ -1903,26 +2193,153 @@ impl BrowserManager {
})),
Some(&effective_session_id),
)
.await
.map_err(|e| {
// Chrome's chrome.debugger API (the extension-relay transport)
// forbids DOM.setFileInputFiles for security, surfacing as an
// opaque `-32000 "Not allowed"`. Translate it into an actionable
// message rather than leaking the raw CDP error (issue #13).
if e.contains("Not allowed") || e.contains("-32000") {
"file upload isn't supported over the extension relay — \
Chrome's chrome.debugger API forbids DOM.setFileInputFiles. \
Use a direct-CDP session instead: \
`chrome-use --session up --launch open <url>` (carry your \
login over with `cookies export` | `cookies set --curl`), \
then run `upload` in that session. \
See https://github.com/leeguooooo/chrome-use/issues/13"
.to_string()
} else {
e
}
})?;
.await;
if let Err(e) = set_files {
// Chrome's chrome.debugger API (the extension-relay transport) forbids
// DOM.setFileInputFiles for security, surfacing as an opaque
// `-32000 "Not allowed"`. Fall back to constructing the File entirely
// IN THE PAGE and assigning it to the input — the standard
// Playwright/Cypress trick, which needs no privileged CDP and so works
// over the relay (issue #13).
if e.contains("Not allowed") || e.contains("-32000") {
return self
.upload_files_via_page(object_id, files, &effective_session_id)
.await;
}
return Err(e);
}
Ok(())
}
/// Relay-safe file upload: read each file locally, hand its bytes to the page
/// as base64, and rebuild a `File` there — then either assign it to a file
/// `<input>` (Chrome allows `input.files = dataTransfer.files`) or, for a
/// dropzone/composer, dispatch synthetic `paste`/`drop` events carrying the
/// `DataTransfer`. No `DOM.setFileInputFiles`, so chrome.debugger permits it.
async fn upload_files_via_page(
&self,
object_id: String,
files: &[String],
session_id: &str,
) -> Result<(), String> {
use base64::Engine;
// The relay tunnels every CDP message through Chrome native messaging,
// which caps a single message at ~1 MiB. A whole image's base64 blows
// past that ("CDP response channel closed"), so we STREAM the bytes into
// a page-side buffer in sub-limit chunks, then assemble the File from it.
const CHUNK: usize = 96 * 1024; // base64 chars per message; safe under 1 MiB
// Reset the staging buffer.
self.client
.send_command(
"Runtime.evaluate",
Some(json!({ "expression": "window.__cuUpload = [];", "returnByValue": true })),
Some(session_id),
)
.await
.map_err(|e| format!("relay upload (reset) failed: {}", e))?;
for path in files {
let bytes = std::fs::read(path).map_err(|e| format!("cannot read {}: {}", path, e))?;
let name = std::path::Path::new(path)
.file_name()
.and_then(|n| n.to_str())
.unwrap_or("upload.bin")
.to_string();
let mime = mime_for_path(&name);
let b64 = base64::engine::general_purpose::STANDARD.encode(&bytes);
// Push the file's metadata with an empty buffer.
let init = format!(
"window.__cuUpload.push({{ name: {}, type: {}, b64: '' }});",
serde_json::to_string(&name).unwrap_or_default(),
serde_json::to_string(mime).unwrap_or_default(),
);
self.client
.send_command(
"Runtime.evaluate",
Some(json!({ "expression": init, "returnByValue": true })),
Some(session_id),
)
.await
.map_err(|e| format!("relay upload (init) failed: {}", e))?;
// Stream the base64 in chunks. base64's alphabet (AZaz09+/=) needs
// no escaping inside a single-quoted JS string, so concatenation is safe.
let idx = "window.__cuUpload[window.__cuUpload.length-1].b64";
let mut start = 0;
while start < b64.len() {
let end = (start + CHUNK).min(b64.len());
let chunk = &b64[start..end];
let expr = format!("{idx} += '{chunk}';");
self.client
.send_command(
"Runtime.evaluate",
Some(json!({ "expression": expr, "returnByValue": true })),
Some(session_id),
)
.await
.map_err(|e| format!("relay upload (chunk) failed: {}", e))?;
start = end;
}
}
// Assemble the Files from the buffer and attach to the element, then clean up.
let func = r#"function() {
const filesData = window.__cuUpload || [];
const dt = new DataTransfer();
for (const f of filesData) {
const bin = atob(f.b64);
const arr = new Uint8Array(bin.length);
for (let i = 0; i < bin.length; i++) arr[i] = bin.charCodeAt(i);
dt.items.add(new File([arr], f.name, { type: f.type }));
}
try { delete window.__cuUpload; } catch (e) { window.__cuUpload = undefined; }
const el = this;
if (el.tagName === 'INPUT' && el.type === 'file') {
el.files = dt.files;
el.dispatchEvent(new Event('input', { bubbles: true }));
el.dispatchEvent(new Event('change', { bubbles: true }));
return 'input:' + dt.files.length;
}
// Dropzone / rich composer: replay paste then drop with the files.
try { el.dispatchEvent(new ClipboardEvent('paste', { bubbles: true, clipboardData: dt })); } catch (e) {}
try {
const ev = new DragEvent('drop', { bubbles: true, cancelable: true });
Object.defineProperty(ev, 'dataTransfer', { value: dt });
el.dispatchEvent(ev);
} catch (e) {}
return 'event:' + dt.files.length;
}"#;
let result: EvaluateResult = self
.client
.send_command_typed(
"Runtime.callFunctionOn",
&CallFunctionOnParams {
function_declaration: func.to_string(),
object_id: Some(object_id),
arguments: None,
return_by_value: Some(true),
await_promise: Some(false),
},
Some(session_id),
)
.await
.map_err(|e| format!("relay file-injection failed: {}", e))?;
if let Some(ref details) = result.exception_details {
return Err(format!(
"relay file-injection threw: {}",
details
.exception
.as_ref()
.and_then(|ex| ex.description.as_deref())
.unwrap_or(&details.text)
));
}
Ok(())
}
@@ -2201,6 +2618,7 @@ async fn initialize_lightpanda_manager(
visited_origins: HashSet::new(),
created_targets: HashSet::new(),
active_target_id: None,
relay_target_misses: HashMap::new(),
next_tab_id: 1,
capture_console: console_capture_enabled(),
};
@@ -2485,6 +2903,33 @@ mod tests {
assert_eq!(active_page_index_after_removal(0, 0, 0), 0);
}
#[test]
fn stale_target_error_matches_relay_signatures() {
// The exact relay error `open` must recover from (issue #35), as wrapped
// by send_command's `CDP error (Page.navigate): …` prefix.
assert!(is_stale_target_error(
"CDP error (Page.navigate): stale sessionId cb-tab-1655244623 for Page.navigate: \
its tab is gone (closed, navigated across processes, or lost after an extension \
restart). Re-attach by re-opening your target URL before retrying."
));
assert!(is_stale_target_error(
"unknown sessionId cb-tab-7 for Page.navigate"
));
assert!(is_stale_target_error("no attached tab for Page.navigate"));
}
#[test]
fn stale_target_error_ignores_unrelated_failures() {
// A genuine navigation failure (bad URL, DNS, blocked) must NOT trigger
// the open-a-fresh-tab recovery — that would mask the real error.
assert!(!is_stale_target_error(
"Navigation failed: net::ERR_NAME_NOT_RESOLVED"
));
assert!(!is_stale_target_error(
"CDP command timed out: Page.navigate"
));
}
fn page(target_id: &str) -> PageInfo {
PageInfo {
tab_id: 1,
@@ -2582,6 +3027,91 @@ mod tests {
assert!(!active_index_is_owned(&[], None, 0, &created));
}
#[test]
fn test_sanitize_title() {
// The exact pollution from #33: ZWJ / word-joiner / invisible-times / BOM
// prepended to "GitHub".
let dirty = "\u{200d}\u{2061}\u{200d}\u{2063}\u{200b}\u{2062}\u{feff}GitHub";
assert_eq!(sanitize_title(dirty), "GitHub");
// Clean titles (incl. CJK + normal punctuation) pass through untouched.
assert_eq!(
sanitize_title("購入手続きへ - メルカリ"),
"購入手続きへ - メルカリ"
);
assert_eq!(sanitize_title(" Hello World "), "Hello World");
// Emoji and real content survive; only the invisibles are dropped.
assert_eq!(sanitize_title("✓ Done\u{200b}"), "✓ Done");
}
#[test]
fn test_mime_for_path() {
assert_eq!(mime_for_path("a.png"), "image/png");
assert_eq!(mime_for_path("PHOTO.JPG"), "image/jpeg");
assert_eq!(mime_for_path("clip.webp"), "image/webp");
assert_eq!(mime_for_path("doc.pdf"), "application/pdf");
assert_eq!(mime_for_path("noext"), "application/octet-stream");
assert_eq!(mime_for_path("weird.xyz"), "application/octet-stream");
}
#[test]
fn prune_protects_pinned_target_on_transient_snapshot() {
// The relay returned a getTargets snapshot missing the pinned tab "A"
// (it hopped to another window). "B" is also absent. Without protection
// both would be pruned and the next command would drift; with the pin
// protected, only the genuinely-unpinned "B" is dropped (issue #31).
let pages = vec![page("A"), page("B")];
let live: HashSet<String> = HashSet::new(); // snapshot returned neither
let gone = prunable_target_ids(&pages, &live, Some("A"));
assert_eq!(gone, vec!["B".to_string()]);
// With no pin, both are prunable (unchanged behavior).
let gone_unpinned = prunable_target_ids(&pages, &live, None);
assert_eq!(gone_unpinned.len(), 2);
// A pinned target that IS in the live set is simply not prunable anyway.
let mut live2 = HashSet::new();
live2.insert("A".to_string());
assert_eq!(
prunable_target_ids(&pages, &live2, Some("A")),
vec!["B".to_string()]
);
}
#[test]
fn debounced_prune_tolerates_transient_churn() {
// Multi-agent churn: a single getTargets snapshot omits our owned tab "B"
// (another agent opened/closed tabs). It must NOT be pruned on one miss.
let pages = vec![page("A"), page("B")];
let mut misses = HashMap::new();
let empty: HashSet<String> = HashSet::new();
// Misses 1 and 2: B absent but under threshold → not pruned.
assert!(debounced_prune_ids(&pages, &empty, Some("A"), &mut misses).is_empty());
assert!(debounced_prune_ids(&pages, &empty, Some("A"), &mut misses).is_empty());
// Miss 3 (== RELAY_PRUNE_MISSES): genuinely gone → pruned.
assert_eq!(
debounced_prune_ids(&pages, &empty, Some("A"), &mut misses),
vec!["B".to_string()]
);
}
#[test]
fn debounced_prune_resets_on_reappearance_and_protects_pin() {
let pages = vec![page("A"), page("B")];
let mut misses = HashMap::new();
let empty: HashSet<String> = HashSet::new();
let mut live_b: HashSet<String> = HashSet::new();
live_b.insert("B".to_string());
// Two misses for B, then it reappears → counter resets, so it survives
// indefinitely under intermittent churn.
debounced_prune_ids(&pages, &empty, Some("A"), &mut misses);
debounced_prune_ids(&pages, &empty, Some("A"), &mut misses);
assert!(debounced_prune_ids(&pages, &live_b, Some("A"), &mut misses).is_empty());
assert!(debounced_prune_ids(&pages, &empty, Some("A"), &mut misses).is_empty()); // back to miss 1
// The pinned active "A" is never pruned no matter how many misses.
for _ in 0..5 {
let gone = debounced_prune_ids(&pages, &empty, Some("A"), &mut misses);
assert!(!gone.contains(&"A".to_string()));
}
}
#[test]
fn resolve_active_index_pin_survives_passive_background_tab() {
// A foreign tab ("Z") gets appended by passive discovery after we pinned
+71
View File
@@ -35,6 +35,7 @@ fn native_test_fixture_html(name: &str) -> &'static str {
"html5_drag_probe" => include_str!("test_fixtures/html5_drag_probe.html"),
"pointer_capture_probe" => include_str!("test_fixtures/pointer_capture_probe.html"),
"upload_probe" => include_str!("test_fixtures/upload_probe.html"),
"iframe_button_probe" => include_str!("test_fixtures/iframe_button_probe.html"),
_ => panic!("Unknown native test fixture: {}", name),
}
}
@@ -573,6 +574,76 @@ async fn e2e_snapshot_and_click_ref() {
assert_success(&resp);
}
/// Clicking a button INSIDE an iframe by `@ref` must deliver a TRUSTED activation
/// (`event.isTrusted === true`), not a synthetic DOM `.click()`. Security-sensitive
/// embedded forms (Google Payments' `保存`) reject `isTrusted:false` clicks, so an
/// enabled submit button silently no-op'd (issue #39). The fix routes iframe-ref
/// clicks to a real `Input.dispatchMouseEvent` on the element's own frame session.
/// The fixture's iframe button writes `clicked:<isTrusted>` into its own text on
/// click, which the cross-frame snapshot reads back.
#[tokio::test]
#[ignore]
async fn e2e_iframe_button_click_is_trusted() {
let mut state = DaemonState::new();
let resp = execute_command(
&json!({ "id": "1", "action": "launch", "headless": true }),
&mut state,
)
.await;
assert_success(&resp);
let resp = execute_command(
&json!({ "id": "2", "action": "navigate", "url": native_test_fixture_url("iframe_button_probe") }),
&mut state,
)
.await;
assert_success(&resp);
// Snapshot (interactive) — the button lives in the iframe and must appear with
// a ref; that ref carries the frame_id so the click resolves into the frame.
let resp = execute_command(
&json!({ "id": "3", "action": "snapshot", "interactive": true }),
&mut state,
)
.await;
assert_success(&resp);
let snapshot = get_data(&resp)["snapshot"].as_str().unwrap_or("");
let ref_id = snapshot
.lines()
.find(|l| l.contains("button \"save\""))
.and_then(|l| l.split("ref=").nth(1))
.map(|r| r.trim_end_matches(']').trim())
.unwrap_or_else(|| panic!("iframe button not found in snapshot:\n{snapshot}"));
// Click it by ref.
let resp = execute_command(
&json!({ "id": "4", "action": "click", "selector": ref_id }),
&mut state,
)
.await;
assert_success(&resp);
tokio::time::sleep(tokio::time::Duration::from_millis(300)).await;
// The button rewrote its own text with the click's isTrusted flag; read it
// back across frames.
let resp = execute_command(
&json!({ "id": "5", "action": "snapshot", "interactive": true }),
&mut state,
)
.await;
assert_success(&resp);
let after = get_data(&resp)["snapshot"].as_str().unwrap_or("");
assert!(
after.contains("clicked:true"),
"iframe button click must be trusted (isTrusted:true); snapshot:\n{after}"
);
let resp = execute_command(&json!({ "id": "99", "action": "close" }), &mut state).await;
assert_success(&resp);
}
// ---------------------------------------------------------------------------
// Screenshot
// ---------------------------------------------------------------------------
+104 -2
View File
@@ -100,6 +100,15 @@ impl RefMap {
self.map.get(ref_id)
}
/// Whether `selector_or_ref` is a `@ref` whose snapshot entry lives inside an
/// iframe (has a `frame_id`). Pointer interactions use this to choose
/// DOM-dispatch over coordinates for OOPIF elements (issue #36).
pub fn ref_is_in_iframe(&self, selector_or_ref: &str) -> bool {
parse_ref(selector_or_ref)
.and_then(|r| self.map.get(&r).map(|e| e.frame_id.is_some()))
.unwrap_or(false)
}
pub fn entries_sorted(&self) -> Vec<(String, RefEntry)> {
let mut entries = self
.map
@@ -1019,7 +1028,9 @@ async fn eval_text_in_frame(client: &CdpClient, session_id: &str, frame_id: &str
.await
.ok()
.and_then(|v| v.get("executionContextId").and_then(|c| c.as_i64()));
let Some(ctx_id) = ctx else { return String::new() };
let Some(ctx_id) = ctx else {
return String::new();
};
let res = client
.send_command(
"Runtime.evaluate",
@@ -1090,7 +1101,10 @@ pub async fn collect_all_frames_text(
let (kind, text) = if is_top {
("top", eval_text_default(client, top_session).await)
} else {
("inline", eval_text_in_frame(client, top_session, &fid).await)
(
"inline",
eval_text_in_frame(client, top_session, &fid).await,
)
};
out.push(FrameText {
frame_id: fid,
@@ -1169,6 +1183,69 @@ pub async fn get_main_content_text(client: &CdpClient, session_id: &str) -> Resu
.to_string())
}
// Text nodes whose parent is one of these carry no visible content.
fn is_noise_tag(name: &str) -> bool {
matches!(name, "SCRIPT" | "STYLE" | "NOSCRIPT" | "TEMPLATE" | "HEAD")
}
// Walk a CDP DOM.Node tree, collecting text-node values. Unlike `innerText`
// (JS, blocked by CLOSED shadow roots), the CDP DOM tree from
// `DOM.getDocument(pierce:true)` includes closed shadow roots and child
// documents — so this reaches text JS can't. `parent_noise` carries whether an
// ancestor was <script>/<style>/etc so their text is skipped.
fn collect_dom_text(node: &Value, parent_noise: bool, out: &mut String) {
let node_type = node.get("nodeType").and_then(|v| v.as_i64()).unwrap_or(0);
let node_name = node.get("nodeName").and_then(|v| v.as_str()).unwrap_or("");
if node_type == 3 {
if !parent_noise {
if let Some(t) = node.get("nodeValue").and_then(|v| v.as_str()) {
let t = t.trim();
if !t.is_empty() {
if !out.is_empty() {
out.push(' ');
}
out.push_str(t);
}
}
}
return;
}
let noise = parent_noise || is_noise_tag(node_name);
if let Some(children) = node.get("children").and_then(|v| v.as_array()) {
for child in children {
collect_dom_text(child, noise, out);
}
}
if let Some(shadow) = node.get("shadowRoots").and_then(|v| v.as_array()) {
for sr in shadow {
collect_dom_text(sr, noise, out);
}
}
if let Some(doc) = node.get("contentDocument") {
collect_dom_text(doc, noise, out);
}
}
/// Extract text from the page via the CDP DOM tree with `pierce:true`, which
/// reaches into CLOSED shadow roots and child documents that `innerText`/`eval`
/// cannot. Lets an agent read content rendered into a closed shadow DOM (e.g. an
/// extension's injected debug panel) without any extra Chrome permission — it
/// rides the per-tab debugger session that's already attached (#30).
pub async fn get_pierced_text(client: &CdpClient, session_id: &str) -> Result<String, String> {
let doc = client
.send_command(
"DOM.getDocument",
Some(serde_json::json!({ "depth": -1, "pierce": true })),
Some(session_id),
)
.await?;
let mut out = String::new();
if let Some(root) = doc.get("root") {
collect_dom_text(root, false, &mut out);
}
Ok(out)
}
pub async fn get_element_attribute(
client: &CdpClient,
session_id: &str,
@@ -1642,6 +1719,31 @@ mod tests {
assert_eq!(parse_ref("@e123"), Some("e123".to_string()));
}
#[test]
fn test_collect_dom_text_pierces_closed_shadow_and_skips_noise() {
// A CDP DOM.Node tree: a host element whose CLOSED shadow root holds the
// text, plus a <script> whose text must be skipped.
let tree = serde_json::json!({
"nodeType": 1, "nodeName": "BODY",
"children": [
{ "nodeType": 1, "nodeName": "SCRIPT",
"children": [ { "nodeType": 3, "nodeName": "#text", "nodeValue": "var secret=1;" } ] },
{ "nodeType": 1, "nodeName": "DIV",
"shadowRoots": [
{ "nodeType": 11, "nodeName": "#document-fragment",
"children": [
{ "nodeType": 1, "nodeName": "SPAN",
"children": [ { "nodeType": 3, "nodeName": "#text", "nodeValue": "DECRYPTED 42" } ] }
] }
] }
]
});
let mut out = String::new();
collect_dom_text(&tree, false, &mut out);
assert_eq!(out, "DECRYPTED 42");
assert!(!out.contains("secret"), "script text must be skipped");
}
#[test]
fn test_parse_ref_equals_prefix() {
assert_eq!(parse_ref("ref=e1"), Some("e1".to_string()));
+326 -1
View File
@@ -7,6 +7,17 @@ use super::cdp::types::*;
use super::element::{parse_ref, resolve_element_center, resolve_element_object_id, RefMap};
use super::humanize;
/// Whether a pointer interaction should be DOM-dispatched (invoke the event on
/// the element in its own session) rather than dispatched at a viewport
/// coordinate via `Input.dispatchMouseEvent`. True when the target is inside an
/// iframe (an OOPIF element's box can't be mapped to a top-viewport point) or we
/// drive over the extension relay (a coordinate Input event isn't confined to the
/// target tab on a busy real Chrome — it drifts onto the foreground tab; issues
/// #31/#36). DOM-dispatch always hits the right element in the right tab.
fn prefer_dom_dispatch(ref_map: &RefMap, selector_or_ref: &str) -> bool {
ref_map.ref_is_in_iframe(selector_or_ref) || crate::connect::relay_url().is_some()
}
pub async fn click(
client: &CdpClient,
session_id: &str,
@@ -45,6 +56,45 @@ pub async fn click(
.await;
}
// An element INSIDE an iframe needs a TRUSTED activation: a DOM `.click()` is
// `isTrusted:false`, which security-sensitive embedded forms reject — Google
// Payments' enabled `保存` button silently no-ops on a synthetic click (issue
// #39). A coordinate `Input.dispatchMouseEvent` can't help either: `getBoxModel`
// for a sub-frame node returns frame-local coordinates that don't compose the
// iframe's offset, so the click lands in the wrong place. The frame-agnostic
// trusted path is keyboard activation — focus the element in its own frame, then
// dispatch a real Enter on the page session; Chrome routes the key to the
// focused element regardless of frame (same as `type --focused`), and Enter on a
// focused button/link fires a trusted `click`. `coord` mode opts out.
let in_iframe = ref_map.ref_is_in_iframe(selector_or_ref);
if mode != "coord" && button == "left" && click_count == 1 && in_iframe {
return dom_activate(
client,
session_id,
ref_map,
selector_or_ref,
iframe_sessions,
)
.await;
}
// On the relay (the user's real Chrome) a TOP-document coordinate click used to
// drift onto the foreground tab; that root cause is fixed (#5: the agent drives
// its own pinned tab), but DOM-dispatch stays the conservative default here.
if mode != "coord"
&& button == "left"
&& click_count == 1
&& crate::connect::relay_url().is_some()
{
return dom_click(
client,
session_id,
ref_map,
selector_or_ref,
iframe_sessions,
)
.await;
}
let resolved = resolve_element_center(
client,
session_id,
@@ -235,6 +285,112 @@ async fn dom_click(
Ok(())
}
/// Trusted activation of an element inside an iframe (issue #39). Focuses the
/// element in its own frame session, then dispatches a real Enter/Space on the
/// page session — Chrome routes the key to the focused element across frames, and
/// Enter/Space on a focused button/link/checkbox fires a `click` with
/// `isTrusted: true`, which security-sensitive embedded forms (Google Payments
/// `保存`) require. Non-activatable roles (a `div[onclick]`) can't be keyboard-
/// activated, so they fall back to a DOM `.click()`.
async fn dom_activate(
client: &CdpClient,
session_id: &str,
ref_map: &RefMap,
selector_or_ref: &str,
iframe_sessions: &HashMap<String, String>,
) -> Result<(), String> {
let role = parse_ref(selector_or_ref)
.and_then(|r| ref_map.get(&r).map(|e| e.role.clone()))
.unwrap_or_default();
// Space toggles checkbox-like controls; Enter activates buttons/links/menus.
let key = match role.as_str() {
"checkbox" | "radio" | "switch" | "option" | "menuitemcheckbox" | "menuitemradio" => {
Some("space")
}
"button" | "link" | "menuitem" | "tab" | "treeitem" => Some("enter"),
_ => None,
};
let Some(key) = key else {
// Not keyboard-activatable — best effort via DOM .click() (untrusted).
return dom_click(
client,
session_id,
ref_map,
selector_or_ref,
iframe_sessions,
)
.await;
};
let (object_id, effective_session_id) = resolve_element_object_id(
client,
session_id,
ref_map,
selector_or_ref,
iframe_sessions,
)
.await?;
// Focus the element in its OWN frame session so the keystroke lands on it.
client
.send_command_typed::<_, Value>(
"Runtime.callFunctionOn",
&CallFunctionOnParams {
function_declaration: "function() { this.focus(); }".to_string(),
object_id: Some(object_id),
arguments: None,
return_by_value: Some(true),
await_promise: Some(false),
},
Some(&effective_session_id),
)
.await?;
// Trusted key on the page session — routed to the focused (in-frame) element.
press_key(client, session_id, key).await?;
wait_for_paint_settled(client, &effective_session_id).await;
Ok(())
}
/// DOM-dispatch a double-click on the element in its own session (no coordinates)
/// — the relay/iframe-safe counterpart to a coordinate dblclick. Fires the full
/// click,click,dblclick sequence so handlers bound to any of them respond.
async fn dom_dblclick(
client: &CdpClient,
session_id: &str,
ref_map: &RefMap,
selector_or_ref: &str,
iframe_sessions: &HashMap<String, String>,
) -> Result<(), String> {
let (object_id, effective_session_id) = resolve_element_object_id(
client,
session_id,
ref_map,
selector_or_ref,
iframe_sessions,
)
.await?;
client
.send_command_typed::<_, Value>(
"Runtime.callFunctionOn",
&CallFunctionOnParams {
function_declaration: r#"function() {
const opts = { bubbles: true, cancelable: true, view: window };
this.dispatchEvent(new MouseEvent('click', opts));
this.dispatchEvent(new MouseEvent('click', { ...opts, detail: 2 }));
this.dispatchEvent(new MouseEvent('dblclick', opts));
}"#
.to_string(),
object_id: Some(object_id),
arguments: None,
return_by_value: Some(true),
await_promise: Some(false),
},
Some(&effective_session_id),
)
.await?;
wait_for_paint_settled(client, &effective_session_id).await;
Ok(())
}
pub async fn dblclick(
client: &CdpClient,
session_id: &str,
@@ -242,6 +398,20 @@ pub async fn dblclick(
selector_or_ref: &str,
iframe_sessions: &HashMap<String, String>,
) -> Result<(), String> {
// Same relay/iframe drift hazard as a single click — DOM-dispatch the
// double-click there instead of a coordinate one (issues #31/#36).
if std::env::var("AGENT_BROWSER_CLICK_MODE").as_deref() != Ok("coord")
&& prefer_dom_dispatch(ref_map, selector_or_ref)
{
return dom_dblclick(
client,
session_id,
ref_map,
selector_or_ref,
iframe_sessions,
)
.await;
}
click(
client,
session_id,
@@ -254,6 +424,50 @@ pub async fn dblclick(
.await
}
/// DOM-dispatch a hover (pointer/mouse enter+move) on the element in its own
/// session — reaches OOPIF elements and never drifts to the foreground tab over
/// the relay, unlike a coordinate `mouseMoved` (issues #31/#36).
async fn dom_hover(
client: &CdpClient,
session_id: &str,
ref_map: &RefMap,
selector_or_ref: &str,
iframe_sessions: &HashMap<String, String>,
) -> Result<(), String> {
let (object_id, effective_session_id) = resolve_element_object_id(
client,
session_id,
ref_map,
selector_or_ref,
iframe_sessions,
)
.await?;
client
.send_command_typed::<_, Value>(
"Runtime.callFunctionOn",
&CallFunctionOnParams {
function_declaration: r#"function() {
const r = this.getBoundingClientRect();
const cx = r.left + r.width / 2, cy = r.top + r.height / 2;
const base = { bubbles: true, cancelable: true, view: window, clientX: cx, clientY: cy };
this.dispatchEvent(new PointerEvent('pointerover', base));
this.dispatchEvent(new PointerEvent('pointerenter', { ...base, bubbles: false }));
this.dispatchEvent(new MouseEvent('mouseover', base));
this.dispatchEvent(new MouseEvent('mouseenter', { ...base, bubbles: false }));
this.dispatchEvent(new MouseEvent('mousemove', base));
}"#
.to_string(),
object_id: Some(object_id),
arguments: None,
return_by_value: Some(true),
await_promise: Some(false),
},
Some(&effective_session_id),
)
.await?;
Ok(())
}
pub async fn hover(
client: &CdpClient,
session_id: &str,
@@ -261,6 +475,18 @@ pub async fn hover(
selector_or_ref: &str,
iframe_sessions: &HashMap<String, String>,
) -> Result<(), String> {
// Coordinate `mouseMoved` drifts to the foreground tab over the relay and
// can't reach an OOPIF — DOM-dispatch the hover there (issues #31/#36).
if prefer_dom_dispatch(ref_map, selector_or_ref) {
return dom_hover(
client,
session_id,
ref_map,
selector_or_ref,
iframe_sessions,
)
.await;
}
let (x, y, _w, _h, effective_session_id) = resolve_element_center(
client,
session_id,
@@ -289,6 +515,63 @@ pub async fn hover(
Ok(())
}
/// DOM-dispatch an HTML5 drag-and-drop from `source` to `target` in their shared
/// session — the relay/iframe-safe counterpart to the coordinate drag, which
/// drifts to the foreground tab over the relay and can't reach an OOPIF (issues
/// #31/#36). Covers HTML5 DnD (sortable lists, file/card boards); pointer-driven
/// drag (canvas, sliders) still needs the coordinate path. Errors if source and
/// target live in different frames — a synthetic cross-frame DnD isn't reliable.
pub async fn dom_drag(
client: &CdpClient,
session_id: &str,
ref_map: &RefMap,
source: &str,
target: &str,
iframe_sessions: &HashMap<String, String>,
) -> Result<(), String> {
let (src_obj, src_session) =
resolve_element_object_id(client, session_id, ref_map, source, iframe_sessions).await?;
let (tgt_obj, tgt_session) =
resolve_element_object_id(client, session_id, ref_map, target, iframe_sessions).await?;
if src_session != tgt_session {
return Err(
"drag source and target are in different frames; cross-frame drag-and-drop over the \
relay isn't supported drag within a single frame, or use a launched browser with \
AGENT_BROWSER_CLICK_MODE=coord"
.to_string(),
);
}
client
.send_command_typed::<_, Value>(
"Runtime.callFunctionOn",
&CallFunctionOnParams {
function_declaration: r#"function(target) {
const dt = new DataTransfer();
const ev = (type, el) => el.dispatchEvent(
new DragEvent(type, { bubbles: true, cancelable: true, dataTransfer: dt }));
ev('dragstart', this);
ev('drag', this);
ev('dragenter', target);
ev('dragover', target);
ev('drop', target);
ev('dragend', this);
}"#
.to_string(),
object_id: Some(src_obj),
arguments: Some(vec![CallArgument {
value: None,
object_id: Some(tgt_obj),
}]),
return_by_value: Some(true),
await_promise: Some(false),
},
Some(&src_session),
)
.await?;
wait_for_paint_settled(client, &src_session).await;
Ok(())
}
pub async fn fill(
client: &CdpClient,
session_id: &str,
@@ -372,6 +655,7 @@ pub async fn type_text(
clear: bool,
delay_ms: Option<u64>,
iframe_sessions: &HashMap<String, String>,
key_events: bool,
) -> Result<(), String> {
let (object_id, effective_session_id) = resolve_element_object_id(
client,
@@ -418,7 +702,7 @@ pub async fn type_text(
.await?;
}
type_text_into_active_context(client, session_id, text, delay_ms).await
type_text_into_active_context(client, session_id, text, delay_ms, key_events).await
}
pub async fn type_text_into_active_context(
@@ -426,6 +710,7 @@ pub async fn type_text_into_active_context(
session_id: &str,
text: &str,
delay_ms: Option<u64>,
key_events: bool,
) -> Result<(), String> {
// Per-character timing: an explicit `delay_ms` wins (caller asked for a
// fixed cadence); otherwise fall back to humanize — variable, human-like
@@ -475,6 +760,46 @@ pub async fn type_text_into_active_context(
Some(session_id),
)
.await?;
} else if key_events {
// Real keystrokes (keyDown+keyUp carrying `text`) for autocomplete /
// combobox widgets that only react to key events and ignore the
// `input` that `Input.insertText` fires — e.g. Google's address
// postal-code → city/prefecture lookup (issue #36 / #4). The keyDown's
// `text` still inserts the character, so the field also fills.
let (key, code, key_code) = char_to_key_info(ch);
let s = ch.to_string();
client
.send_command_typed::<_, Value>(
"Input.dispatchKeyEvent",
&DispatchKeyEventParams {
event_type: "keyDown".to_string(),
key: Some(key.clone()),
code: Some(code.clone()),
text: Some(s.clone()),
unmodified_text: Some(s),
windows_virtual_key_code: Some(key_code),
native_virtual_key_code: Some(key_code),
modifiers: None,
},
Some(session_id),
)
.await?;
client
.send_command_typed::<_, Value>(
"Input.dispatchKeyEvent",
&DispatchKeyEventParams {
event_type: "keyUp".to_string(),
key: Some(key),
code: Some(code),
text: None,
unmodified_text: None,
windows_virtual_key_code: Some(key_code),
native_virtual_key_code: Some(key_code),
modifiers: None,
},
Some(session_id),
)
.await?;
} else {
// VS Code/Electron webviews reject repeated dispatchKeyEvent calls
// carrying printable `text`. Insert printable characters directly
+14 -1
View File
@@ -60,6 +60,9 @@ pub struct ScreenshotOptions {
pub quality: Option<i32>,
pub annotate: bool,
pub output_dir: Option<String>,
/// Explicit pixel region (x, y, width, height) — `--clip` (issue #34). Takes
/// precedence over selector/full_page.
pub clip: Option<(f64, f64, f64, f64)>,
}
impl Default for ScreenshotOptions {
@@ -72,6 +75,7 @@ impl Default for ScreenshotOptions {
quality: None,
annotate: false,
output_dir: None,
clip: None,
}
}
}
@@ -187,7 +191,16 @@ async fn capture_screenshot_base64(
capture_beyond_viewport: if options.full_page { Some(true) } else { None },
};
if options.full_page {
if let Some((x, y, width, height)) = options.clip {
// Explicit pixel region wins over selector/full_page (issue #34).
params.clip = Some(Viewport {
x,
y,
width,
height,
scale: 1.0,
});
} else if options.full_page {
let metrics: Value = client
.send_command_no_params("Page.getLayoutMetrics", Some(session_id))
.await?;
+36 -5
View File
@@ -330,6 +330,13 @@ impl RoleNameTracker {
}
}
/// Max iframe nesting depth `take_snapshot` expands. Embedded payment/checkout
/// widgets nest a few frames deep (e.g. AdSense → payments.google.com → an inner
/// form frame); expanding past the first level is what gives those inner refs a
/// `frame_id` so clicks resolve into the right frame (issue #36). Capped to keep
/// a pathological frame tree from blowing up the snapshot.
const MAX_IFRAME_DEPTH: usize = 3;
pub async fn take_snapshot(
client: &CdpClient,
session_id: &str,
@@ -337,6 +344,28 @@ pub async fn take_snapshot(
ref_map: &mut RefMap,
frame_id: Option<&str>,
iframe_sessions: &HashMap<String, String>,
) -> Result<String, String> {
take_snapshot_at_depth(
client,
session_id,
options,
ref_map,
frame_id,
iframe_sessions,
0,
)
.await
}
#[allow(clippy::too_many_arguments)]
async fn take_snapshot_at_depth(
client: &CdpClient,
session_id: &str,
options: &SnapshotOptions,
ref_map: &mut RefMap,
frame_id: Option<&str>,
iframe_sessions: &HashMap<String, String>,
depth: usize,
) -> Result<String, String> {
client
.send_command_no_params("DOM.enable", Some(session_id))
@@ -606,10 +635,11 @@ pub async fn take_snapshot(
}
// Recurse into child iframes: for each Iframe node with a backend_node_id,
// resolve the child frame ID and take a snapshot of its content.
// We only recurse from the main frame (frame_id == None) to avoid
// unbounded depth; nested iframes within iframes are not expanded.
if frame_id.is_none() {
// resolve the child frame ID and snapshot its content. Recurse to
// MAX_IFRAME_DEPTH (not just the main frame) so refs inside nested
// payment/checkout widgets get a `frame_id` and clicks resolve into the right
// frame (issue #36); the cap bounds a pathological frame tree.
if depth < MAX_IFRAME_DEPTH {
let mut iframe_snapshots: Vec<(String, String)> = Vec::new(); // (ref_id, child_snapshot)
for node in tree_nodes.iter() {
if node.role != "Iframe" || !node.has_ref {
@@ -622,13 +652,14 @@ pub async fn take_snapshot(
if let Ok(child_fid) = resolve_iframe_frame_id(client, session_id, bid).await {
// Snapshot the child frame; errors are silently ignored
// (e.g. cross-origin iframes)
if let Ok(child_text) = Box::pin(take_snapshot(
if let Ok(child_text) = Box::pin(take_snapshot_at_depth(
client,
session_id,
options,
ref_map,
Some(&child_fid),
iframe_sessions,
depth + 1,
))
.await
{
@@ -0,0 +1,28 @@
<!doctype html>
<html>
<head>
<meta charset="utf-8" />
<title>iframe button probe</title>
</head>
<body>
<h1>iframe button probe</h1>
<iframe
id="frame"
width="320"
height="140"
srcdoc="
<!doctype html>
<html>
<body style='margin:24px'>
<button id='b' style='padding:24px;font-size:22px'>save</button>
<script>
document.getElementById('b').addEventListener('click', function (e) {
this.textContent = 'clicked:' + e.isTrusted;
});
</script>
</body>
</html>
"
></iframe>
</body>
</html>
+113 -19
View File
@@ -224,6 +224,65 @@ pub fn print_response_with_opts(resp: &Response, action: Option<&str>, opts: &Ou
return;
}
// Cloudflare challenge/clearance preflight (`cf-status`). Checked early
// because its response carries `url`/`title`, which later generic
// renderers would otherwise swallow.
if action == Some("cf_status") {
let challenged = data
.get("challenged")
.and_then(|v| v.as_bool())
.unwrap_or(false);
let rec = data
.get("recommendation")
.and_then(|v| v.as_str())
.unwrap_or("?");
let cl = data.get("clearance");
let present = cl
.and_then(|c| c.get("present"))
.and_then(|v| v.as_bool())
.unwrap_or(false);
let expired = cl
.and_then(|c| c.get("expired"))
.and_then(|v| v.as_bool())
.unwrap_or(false);
let expires_in = cl.and_then(|c| c.get("expiresIn")).and_then(|v| v.as_i64());
let device = data
.get("deviceVerified")
.and_then(|v| v.as_bool())
.unwrap_or(false);
let (icon, headline) = match rec {
"proceed" => (color::success_indicator().to_string(), "cleared — no challenge, proceed"),
"solve" => (color::warning_indicator().to_string(), "Cloudflare challenge active, no valid clearance — solve it"),
"reissue" => (color::warning_indicator().to_string(), "challenge active but a clearance cookie exists — stale (IP/UA changed?), re-solve"),
_ => (color::cyan("").to_string(), "unknown"),
};
println!("{} {}", icon, headline);
println!(
" challenged: {}",
if challenged { "yes" } else { "no" }
);
let cl_desc = if !present {
"absent".to_string()
} else if expired {
"present but EXPIRED".to_string()
} else if let Some(s) = expires_in {
format!("valid, expires in {}m {}s", s / 60, s % 60)
} else {
"present (session)".to_string()
};
println!(" cf_clearance: {}", cl_desc);
println!(
" device trusted: {}",
if device {
"yes (CF_VERIFIED_DEVICE)"
} else {
"no"
}
);
return;
}
// Dialog status response
if action == Some("dialog") {
if let Some(has_dialog) = data.get("hasDialog").and_then(|v| v.as_bool()) {
@@ -348,7 +407,11 @@ pub fn print_response_with_opts(resp: &Response, action: Option<&str>, opts: &Ou
let count = list.len();
println!(
"{}",
color::bold(&format!("{} frame{}", count, if count == 1 { "" } else { "s" }))
color::bold(&format!(
"{} frame{}",
count,
if count == 1 { "" } else { "s" }
))
);
for f in list {
let idx = f.get("index").and_then(|v| v.as_i64()).unwrap_or(0);
@@ -597,19 +660,21 @@ pub fn print_response_with_opts(resp: &Response, action: Option<&str>, opts: &Ou
// Tab switch
if action == Some("tab_switch") {
if let Some(tab_id) = data.get("tabId").and_then(|v| v.as_str()) {
if let Some(url) = data.get("url").and_then(|v| v.as_str()) {
println!(
"{} Switched to tab [{}] ({})",
color::success_indicator(),
tab_id,
url
);
let warning = data.get("warning").and_then(|v| v.as_str());
// A non-responding session isn't a real success — show a warning
// indicator instead of the green ✓ (issue #29.3).
let indicator = if warning.is_some() {
color::warning_indicator()
} else {
println!(
"{} Switched to tab [{}]",
color::success_indicator(),
tab_id
);
color::success_indicator()
};
if let Some(url) = data.get("url").and_then(|v| v.as_str()) {
println!("{} Switched to tab [{}] ({})", indicator, tab_id, url);
} else {
println!("{} Switched to tab [{}]", indicator, tab_id);
}
if let Some(w) = warning {
eprintln!("{}", color::dim(w));
}
return;
}
@@ -1428,6 +1493,12 @@ Usage: chrome-use type <selector> <text>
Types text into the specified element character by character.
Unlike fill, this does not clear existing content first.
Options:
--key-events Send real per-character keyDown/keyUp instead of
(alias --keys) Input.insertText. Use for autocomplete / combobox fields
that only react to key events e.g. a postal-code box
that auto-fills city/prefecture, or Google Places.
Global Options:
--json Output as JSON
--session <name> Use specific session
@@ -1435,6 +1506,7 @@ Global Options:
Examples:
chrome-use type "#search" "hello"
chrome-use type @e2 "additional text"
chrome-use type @e5 "201-0001" --key-events # trigger the address autocomplete
See Also:
For typing into contenteditable editors (Lexical, ProseMirror, etc.)
@@ -1699,12 +1771,23 @@ Usage: chrome-use scroll [direction] [amount] [options]
Scrolls the page or a specific element in the specified direction.
Without --selector, scroll dispatches a real (isTrusted) mouse wheel at a
viewport coordinate, so it scrolls whatever container is under the pointer
including cross-origin iframes (Google Payments, Stripe, embedded checkout/KYC)
that plain page scroll can't reach.
Arguments:
direction up, down, left, right (default: down)
amount Pixels to scroll (default: 300)
Options:
-s, --selector <sel> CSS selector for a scrollable container
-s, --selector <sel> CSS selector for a scrollable container (same-origin)
--at <x,y> Dispatch the wheel at this viewport pixel (read it from a
screenshot) precise way into a cross-origin iframe
--frame <n> Scroll the n-th frame from `chrome-use frames` (wheel at
that frame's center)
Without --selector/--at/--frame the wheel lands at the viewport center.
Global Options:
--json Output as JSON
@@ -1716,6 +1799,8 @@ Examples:
chrome-use scroll up 200
chrome-use scroll left 100
chrome-use scroll down 500 --selector "div.scroll-container"
chrome-use scroll down 700 --at 640,400 # wheel at a pixel over an iframe
chrome-use scroll down 700 --frame 2 # scroll frame 2 from `frames`
"##
}
"scrollintoview" | "scrollinto" => {
@@ -1796,6 +1881,8 @@ Pass --hide-scrollbars false when launching to keep native scrollbars visible.
Options:
--full, -f Capture full page (not just viewport)
[selector] Capture just an element (CSS or @ref), e.g. `screenshot ".header" h.png`
--clip <x,y,w,h> Capture a pixel region, e.g. `screenshot --clip 0,0,200,40 corner.png`
--annotate Overlay numbered labels on interactive elements.
Each label [N] corresponds to ref @eN from snapshot.
Prints a legend mapping labels to element roles/names.
@@ -1816,6 +1903,8 @@ Examples:
chrome-use screenshot
chrome-use screenshot ./screenshot.png
chrome-use screenshot --full ./full-page.png
chrome-use screenshot ".header .indicator" corner.png # just one element
chrome-use screenshot --clip 1600,0,200,40 corner.png # a pixel region
chrome-use screenshot --annotate # Labeled screenshot + legend
chrome-use screenshot --annotate ./page.png # Save annotated screenshot
chrome-use screenshot --annotate --json # JSON output with annotations
@@ -1957,9 +2046,9 @@ Usage: chrome-use get <subcommand> [args]
Retrieves various types of information from elements or the page.
Subcommands:
text <selector> Get text content of element
text --all-frames Aggregate text across ALL frames (incl. iframes)
text [selector] Element text; no selector = WHOLE PAGE, all frames
text --main Main-content text only (skip nav/header/sidebar)
text --pierce Read through CLOSED shadow DOM (injected panels)
html <selector> Get inner HTML of element
value <selector> Get value of input element
attr <selector> <name> Get attribute value
@@ -1975,8 +2064,8 @@ Global Options:
--session <name> Use specific session
Examples:
chrome-use get text @e1
chrome-use get text --all-frames # read iframed content (listing pages)
chrome-use get text # whole page across ALL frames (default)
chrome-use get text @e1 # one element
chrome-use get text --main # main content, no nav/sidebar boilerplate
chrome-use frames # list frames + where the text lives
chrome-use get html "#content"
@@ -3188,11 +3277,15 @@ Navigation:
Get Info: chrome-use get <what> [selector]
text, html, value, attr <name>, title, url, count, box, styles, cdp-url
text --all-frames (cross-frame), text --main (no boilerplate), frames (list)
text (no selector = whole page, all frames), text --main, frames (list)
Check State: chrome-use is <what> <selector>
visible, enabled, checked
Anti-bot: chrome-use stealth | cf-status
stealth stealth self-check (webdriver/UA/plugins + overrides)
cf-status Cloudflare challenge + cf_clearance preflight (skip re-solving)
Find Elements: chrome-use find <locator> <value> <action> [text]
role, text, label, placeholder, alt, title, testid, first, last, nth
@@ -3287,6 +3380,7 @@ Confirmation:
Sessions:
session Show current session name
session list List active sessions
sessions List running session daemons (alias of daemon status)
daemon status List running session daemons (+ relay state)
daemon restart Kill all session daemons; keeps the extension relay
up. Clears stale/cross-leaked state after an upgrade.
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "chrome-use",
"version": "1.5.1",
"version": "1.5.17",
"description": "chrome-use — drive your real, logged-in Chrome from any AI agent, stealth by default",
"type": "module",
"packageManager": "pnpm@11.1.3",
+95 -15
View File
@@ -36,6 +36,29 @@ Refs (`@e1`, `@e2`, ...) are assigned fresh on every snapshot. They become
submits, dynamic re-renders, dialog opens. Always re-snapshot before your
next ref interaction.
> **Hard rule: snapshot-first, never screenshot-to-locate.** For form fields and
> buttons, ALWAYS `snapshot -i` and act on refs/selectors. Do **not** reach for
> `screenshot` + coordinate clicks to find or hit an element — `snapshot -i` now
> pierces **cross-origin iframes** (embedded Google Payments / Stripe / checkout /
> KYC forms) and lists their elements by `@ref`, including input values. Use
> coordinates only for canvas/WebGL, or when `snapshot` genuinely returns nothing
> for your target. Screenshots are for *visual verification you report*, never the
> agent's own input — and a full-page `screenshot` of a real retina browser is
> often too large for an image reader anyway. (If you ever feel you *need* a
> screenshot to read state or locate something, that's a bug — please file it.)
> **Snapshot-first, always. Never default to `screenshot` + coordinate clicking
> for form fields or buttons.** Run `snapshot -i` and act on `@refs`. Use
> coordinates only for canvas/WebGL, or when `snapshot` genuinely returns nothing
> for your target. This holds **even inside cross-origin embedded iframes**
> since v1.5.12 `snapshot -i` pierces out-of-process iframes (Google Payments,
> Stripe, embedded checkout/KYC) and lists their elements with refs, so
> `click @e` / `type @e` / `fill @e` work directly. A screenshot is for a genuine
> *visual* check you report to the user — not your own input. (Full-page
> screenshots of a real retina Chrome are often too large for the image reader
> anyway.) Driving off pixels on the relay also risks a coordinate event drifting
> onto the user's foreground tab — refs never do. See issue #37.
## Before you automate: pick the cheapest tool
Driving a browser is the heavy option. chrome-use earns its keep when you
@@ -108,7 +131,17 @@ Each `--session` that connects gets its **own colored Chrome tab group** (named
after the session) and drives only its own tabs — multiple agents share the one
real browser without cross-talk, and the user's own tabs are never grouped. CDP
drives the page without moving the user's mouse/keyboard, so it doesn't fight
them for control. **Anti-detection ranking: this real logged-in Chrome (extension
them for control.
**Strict multi-agent isolation.** A session over the relay tracks and drives
**only the tabs it created** (its own group). It does **not** adopt the user's
existing tabs, other agents' tabs, or pop-ups (e.g. an OAuth/login window — that's
the user's), so several agents (and other tools opening tabs) can work in the same
real Chrome concurrently without ever dropping or stealing each other's tabs —
another agent's tab churn can't make your bound tab vanish or drift your commands
onto the wrong page. Consequence: `tab list` shows only *your* session's tabs; to
drive a specific page, navigate to it in your own tab instead of expecting a
pre-existing or popped-up tab to appear in the list. **Anti-detection ranking: this real logged-in Chrome (extension
connect) > a headed launched browser > headless (forbidden).** A genuine human
browser has no headless/automation tells at all, so prefer it for anything
anti-bot-sensitive.
@@ -127,6 +160,19 @@ cadence, and scroll/drag ease. Default `off`; a per-navigation detector
auto-escalates pages guarded by Akamai/PerimeterX/DataDome to `human`. Leave it
on auto; force `human` only when you already know the target scores behaviour.
**Cloudflare clearance — solve once, reuse.** Passing a Cloudflare challenge
mints a `cf_clearance` cookie (HttpOnly — invisible to `eval`/`document.cookie`;
read it via `chrome-use cookies`). It's bound to your **IP + User-Agent**: reuse
the same exit IP and UA and you skip the challenge until it expires. Driving the
user's real Chrome (relay) persists it natively; for isolated sessions,
`--session-name <name>` save/restores it. Before spending effort solving, run
`chrome-use cf-status` (aliases `cf`, `clearance`): it reports whether the page
is *currently* a Cloudflare challenge and whether a still-valid `cf_clearance`
exists, with a recommendation — `proceed` (already cleared, don't re-solve),
`solve` (challenge up, no clearance), or `reissue` (clearance present but page
still blocks → IP/UA drifted, re-solve). Use it as a preflight to avoid
re-solving what you already cleared.
## Two ways to drive a page — and when to drop to `eval`
You have a **real Chrome with the user's DOM**. Two layers, mix them freely:
@@ -207,9 +253,10 @@ assigned fresh on every snapshot.
For unstructured reading (no refs needed):
```bash
chrome-use get text @e1 # visible text of an element
chrome-use get text --all-frames # whole page, aggregated across ALL frames
chrome-use get text # WHOLE PAGE — all frames by default (see below)
chrome-use get text @e1 # visible text of one element (or a CSS selector)
chrome-use get text --main # main content only — skip nav/header/sidebar
chrome-use get text --pierce # read through CLOSED shadow DOM (injected panels)
chrome-use frames # list every frame + where the text lives
chrome-use get html @e1 # innerHTML
chrome-use get attr @e1 href # any attribute
@@ -219,13 +266,26 @@ chrome-use get url # current URL
chrome-use get count ".item" # count matching elements
```
On listing/marketplace pages (Yahoo Auctions, Rakuten, Mercari shops) the seller's
description often lives in a **child frame** or is buried under a "related items"
sidebar, so a plain `get text body` returns only header/nav boilerplate. When the
text you expect is missing: run `chrome-use frames` to see where it is, then
`get text --all-frames` (reads every reachable frame incl. cross-origin iframes)
or `get text --main` (drops the global chrome). If the content is lazy-loaded,
`scroll` it into view first.
**Whole-page text is cross-frame by default.** `chrome-use get text` with no
selector aggregates visible text across **every** frame — top document plus
same-process child frames plus cross-origin iframes — so you never silently miss
content that lives in an iframe (Yahoo Auctions / Rakuten / Mercari shop
descriptions, embedded checkout/spec frames). Each child frame is delimited with
a `----- frame [kind] url -----` marker. You do **not** need to remember a flag —
the default already reads all frames. (`--all-frames` is still accepted as an
explicit alias.)
So: when text looks missing or wrong, you don't have to guess — just
`chrome-use get text` reads everything. To **see** the structure (which frame
holds what), run `chrome-use frames`. To **cut boilerplate** (global nav/header/
footer, "related items" sidebars), use `chrome-use get text --main`. If content
is lazy-loaded, `scroll` it into view first, then read.
**Closed shadow DOM.** Some injected UI (browser-extension debug panels, web
components) renders into a *closed* shadow root that `eval`/`innerText` cannot
read. `chrome-use get text --pierce` reads through closed shadow roots and child
documents via the CDP DOM tree — use it when content is clearly on screen (you
see it in a screenshot) but `get text`/`eval` come back empty.
## Interacting
@@ -237,6 +297,10 @@ chrome-use hover @e1 # hover
chrome-use focus @e1 # focus (useful before keyboard input)
chrome-use fill @e2 "hello" # clear then type
chrome-use type @e2 " world" # type without clearing
chrome-use type @e5 "201-0001" --key-events # real keystrokes (not insertText) —
# use for autocomplete/combobox fields that
# only react to key events (e.g. a postal box
# that auto-fills city/prefecture, Google Places)
chrome-use press Enter # press a key at current focus (down+up)
chrome-use press Control+a # key combination
chrome-use keydown d # HOLD a key down (no auto-release)
@@ -254,16 +318,32 @@ chrome-use pick @e4 --option "Europe" # ANY combobox (react-select / ARIA /
# (no silent no-op). Use this for custom
# dropdowns where `select` returns ✓ but
# changes nothing.
chrome-use upload @e5 file1.pdf # upload file(s) — NOTE: needs a --launch/direct-CDP
# session. Over the extension relay it CANNOT work
# (Chrome's chrome.debugger forbids it); chrome-use
# errors with a hint. Carry your login into a launched
# session via `cookies export` | `cookies set --curl`.
chrome-use upload @e5 file1.pdf # upload file(s) — works over the extension relay too:
# chrome.debugger forbids setFileInputFiles, so the
# file's bytes are streamed into the page and rebuilt as
# a File there (chunked under native-messaging's 1 MiB cap).
# Works on file <input>s and drop/paste composers (e.g. X).
chrome-use scroll down 500 # scroll page (up/down/left/right)
chrome-use scroll down 700 --at 640,400 # wheel at a pixel — scrolls a cross-origin
# iframe (Payments/Stripe/checkout/KYC) that
# plain page scroll can't reach
chrome-use scroll down 700 --frame 2 # scroll frame 2 from `chrome-use frames`
chrome-use scrollintoview @e1 # scroll element into view
chrome-use drag @e1 @e2 # drag and drop
```
**Cross-origin iframes (embedded payment / checkout / KYC widgets — Google
Payments, Stripe, etc.) — drive them by ref, never by screenshot.** `snapshot -i`
pierces these out-of-process iframes and lists their elements by `@ref`
(including input values); `get text --all-frames` reads their text. Then just act
on the refs: `click @e`, `type @e`, `hover @e`, `dblclick @e`, `drag @a @b` all
work into the iframe. Over the extension relay these are dispatched through the
DOM (in the element's own frame), so they hit the right element in the right tab
— a coordinate click/scroll there can drift onto whatever tab is in the
foreground, so prefer refs. For below-the-fold content in such a frame, scroll it
with `scroll down N --at x,y` (a pixel over the frame) or `--frame n`. For a
postal/autocomplete box inside the frame, `type @e "…" --key-events`.
### When refs don't work or you don't want to snapshot
Use semantic locators: