Compare commits

...
25 Commits
Author SHA1 Message Date
leeguooooo e7e849ea39 chore(release): 1.5.8 — file upload over the extension relay (#13)
Release binaries / Build macOS ARM64 (push) Has been cancelled
Release binaries / Build macOS x64 (push) Has been cancelled
Release binaries / Build Linux ARM64 (push) Has been cancelled
Release binaries / Build Linux musl ARM64 (push) Has been cancelled
Release binaries / Build Linux musl x64 (push) Has been cancelled
Release binaries / Build Linux x64 (push) Has been cancelled
Release binaries / Build Windows x64 (push) Has been cancelled
Release binaries / Attach binaries to GitHub Release (push) Has been cancelled
2026-06-16 14:05:33 +09:00
leeguooooo ebb02c65c8 fix(relay): file upload now works over the extension relay (#13)
chrome.debugger forbids DOM.setFileInputFiles, so 'upload' used to hard-fail on
the relay and push users to a --launch/direct-CDP session. Now it falls back to
reconstructing the File entirely in the page (Playwright/Cypress-style: build a
File from the bytes, assign input.files = dataTransfer.files, fire input/change;
for drop/paste composers like X, dispatch synthetic paste+drop with the
DataTransfer). The bytes are streamed in <1 MiB base64 chunks because the relay
tunnels CDP through native messaging (1 MiB/message cap) — a whole image as one
arg closed the channel. Verified live over the relay: an 809 KB PNG lands intact
on a file input with change firing. No more direct-CDP needed for uploads.
2026-06-16 14:05:31 +09:00
leeguooooo 4317db636f chore(release): 1.5.7 — cf-status Cloudflare clearance preflight
Release binaries / Build macOS ARM64 (push) Has been cancelled
Release binaries / Build macOS x64 (push) Has been cancelled
Release binaries / Build Linux ARM64 (push) Has been cancelled
Release binaries / Build Linux musl ARM64 (push) Has been cancelled
Release binaries / Build Linux musl x64 (push) Has been cancelled
Release binaries / Build Linux x64 (push) Has been cancelled
Release binaries / Build Windows x64 (push) Has been cancelled
Release binaries / Attach binaries to GitHub Release (push) Has been cancelled
2026-06-16 12:19:26 +09:00
leeguooooo c99838a034 feat(cloudflare): cf-status preflight — skip re-solving when cf_clearance is still valid
Passing a Cloudflare challenge mints an HttpOnly cf_clearance cookie bound to
IP+UA. 'chrome-use cf-status' (aliases cf/cloudflare-status/clearance) reports
whether the active page is currently a CF challenge and whether a still-valid
cf_clearance exists (read via CDP — HttpOnly is invisible to document.cookie),
plus CF_VERIFIED_DEVICE trust, and a recommendation: proceed (already cleared,
don't re-solve) / solve (challenge up, no clearance) / reissue (clearance present
but page still blocks → IP/UA drifted). Lets an agent avoid re-solving what it
already cleared — the persistence optimization. Pure helpers unit-tested; live
-verified on a real cf_clearance.
2026-06-16 12:19:24 +09:00
leeguooooo dc2aa4cade chore(release): 1.5.6 — pin adopted tab against transient relay snapshots (#31)
Release binaries / Build macOS ARM64 (push) Has been cancelled
Release binaries / Build macOS x64 (push) Has been cancelled
Release binaries / Build Linux ARM64 (push) Has been cancelled
Release binaries / Build Linux musl ARM64 (push) Has been cancelled
Release binaries / Build Linux musl x64 (push) Has been cancelled
Release binaries / Build Linux x64 (push) Has been cancelled
Release binaries / Build Windows x64 (push) Has been cancelled
Release binaries / Attach binaries to GitHub Release (push) Has been cancelled
2026-06-15 17:00:42 +09:00
leeguooooo 7085f3bf36 fix(relay): don't prune the pinned target on a transient getTargets snapshot (#31)
Driving a busy real Chrome via the relay, a single Target.getTargets call
occasionally returns a different window's tabs ('tab list hops windows'). resync
pruned every tracked page absent from that snapshot — including the agent's
explicitly-adopted (pinned) tab — after which active-target resolution fell back
to active_page_index and eval/click/snapshot drifted onto a foreign tab
(about:blank / chrome-extension:// / the user's page), breaking any 3+ step flow.

prunable_target_ids() now protects the pinned active target from snapshot-based
pruning; a genuine close still arrives as Target.targetDestroyed (event drain) and
removes it properly. Unit-tested.
2026-06-15 17:00:38 +09:00
leeguooooo 29815ff5f3 chore(release): 1.5.5 — connect-mode diagnostic log for the remote-debugging consent modal (#31)
Release binaries / Build macOS ARM64 (push) Has been cancelled
Release binaries / Build macOS x64 (push) Has been cancelled
Release binaries / Build Linux ARM64 (push) Has been cancelled
Release binaries / Build Linux musl ARM64 (push) Has been cancelled
Release binaries / Build Linux musl x64 (push) Has been cancelled
Release binaries / Build Linux x64 (push) Has been cancelled
Release binaries / Build Windows x64 (push) Has been cancelled
Release binaries / Attach binaries to GitHub Release (push) Has been cancelled
2026-06-15 16:47:14 +09:00
leeguooooo 2a338d4c29 diag(connect): log CDP transport mode to detect 'Allow remote debugging?' modal source (#31)
The consent modal only appears on a raw remote-debugging attach or a browser we
launched with a debug port — never on the ab-connect extension relay. Append one
line per connection to ~/.chrome-use/connect-mode.log (relay | raw-port-attach |
launched | remote-ws), flagging 'CONSENT-MODAL-RISK' when a raw-port/launch path
runs while the relay was available. Lets us tell a code regression from Chrome's
own extension-debugger consent UX when the modal reappears. Best-effort, never
fails a connection. Verified: normal 'open' logs mode=relay (consent-free).
2026-06-15 16:47:12 +09:00
leeguooooo 6fcf52db60 chore(release): 1.5.4 — get text --pierce (closed shadow DOM, #30)
Release binaries / Build macOS ARM64 (push) Has been cancelled
Release binaries / Build macOS x64 (push) Has been cancelled
Release binaries / Build Linux ARM64 (push) Has been cancelled
Release binaries / Build Linux musl ARM64 (push) Has been cancelled
Release binaries / Build Linux musl x64 (push) Has been cancelled
Release binaries / Build Linux x64 (push) Has been cancelled
Release binaries / Build Windows x64 (push) Has been cancelled
Release binaries / Attach binaries to GitHub Release (push) Has been cancelled
2026-06-15 15:46:31 +09:00
leeguooooo af8823b27b feat(text): 'get text --pierce' reads through CLOSED shadow DOM (#30)
Some injected UI (browser-extension debug panels, web components) renders into a
CLOSED shadow root that eval/innerText cannot read. --pierce walks the CDP DOM
tree (DOM.getDocument depth:-1 pierce:true), which includes closed shadow roots
and child documents, and collects text nodes (skipping script/style/etc).

Review-safe: rides the per-tab debugger session already attached, no new Chrome
permission and no ab-connect/extension change — so it works in extension-relay
mode without touching the published extension. Verified live: a closed-shadow
panel that main-world eval reports HIDDEN is read in full via --pierce.

First slice of #30 (read extension/injected-panel content). Deeper extension
introspection (background SW / chrome.storage) stays a launch-mode / raw-CDP
concern, deliberately NOT done by expanding ab-connect's debugger powers.
2026-06-15 15:46:23 +09:00
leeguooooo c85e3faa82 chore(release): 1.5.3 — get text defaults to cross-frame; #29 (sessions/did-you-mean/tab liveness); CI changelog fix
Release binaries / Build macOS ARM64 (push) Has been cancelled
Release binaries / Build macOS x64 (push) Has been cancelled
Release binaries / Build Linux ARM64 (push) Has been cancelled
Release binaries / Build Linux musl ARM64 (push) Has been cancelled
Release binaries / Build Linux musl x64 (push) Has been cancelled
Release binaries / Build Linux x64 (push) Has been cancelled
Release binaries / Build Windows x64 (push) Has been cancelled
Release binaries / Attach binaries to GitHub Release (push) Has been cancelled
2026-06-15 13:42:58 +09:00
leeguooooo b25958946c feat(text): 'get text' (no selector) defaults to cross-frame whole-page read
So an agent never silently misses iframed content (listing descriptions etc.)
without having to know the --all-frames flag. Single-frame pages are unchanged
(identical to the old body read); multi-frame pages now include child frames —
a strict superset. Skill + help updated to make the default and 'frames'/--main
discoverable.
2026-06-15 13:42:48 +09:00
leeguooooo d4ff49caa8 ci(release): don't let an empty changelog section abort the release (bash -e)
The changelog step runs under 'bash -e'. section() returned non-zero when a
commit category was empty (grep no-match / empty [ -n ] test), aborting the
script before the closing heredoc delimiter — so any release whose range lacked
a whole category (e.g. 1.5.2: only feat, no fix) failed to attach binaries.
Add '|| true' + 'return 0' so section() always succeeds.
2026-06-15 13:42:48 +09:00
leeguooooo 4e949fffbf chore(release): 1.5.2 — sessions command + did-you-mean + honest tab-switch liveness (#29) 2026-06-15 13:34:26 +09:00
leeguooooo af46490812 feat(cli): sessions command + 'did you mean' suggestions + honest tab-switch liveness (#29)
- chrome-use sessions: top-level alias for the daemon inventory (the skill
  advertises sessions, so it's a natural guess that used to error).
- Unknown commands now suggest the nearest valid one (Levenshtein + prefix
  match), staying silent when nothing is close (e.g. 'clik' -> click,
  'sesions' -> sessions, 'xyzzy' -> no suggestion).
- tab <id>: probe the switched session and show a warning indicator instead of
  a green check when it isn't responding yet, so a switch onto a re-attaching
  (churned-tabId) session no longer reports false success. The #24 targetId
  recovery self-heals within ~6s, hence a warning rather than a hard error.
2026-06-15 13:25:16 +09:00
leeguooooo 57c52d6517 chore(release): 1.5.1 — frame-aware text extraction (get text --all-frames/--main, frames; #27) + ab-connect 0.4.9 targetId recovery (#24)
Release binaries / Build macOS ARM64 (push) Has been cancelled
Release binaries / Build macOS x64 (push) Has been cancelled
Release binaries / Build Linux ARM64 (push) Has been cancelled
Release binaries / Build Linux musl ARM64 (push) Has been cancelled
Release binaries / Build Linux musl x64 (push) Has been cancelled
Release binaries / Build Linux x64 (push) Has been cancelled
Release binaries / Build Windows x64 (push) Has been cancelled
Release binaries / Attach binaries to GitHub Release (push) Has been cancelled
2026-06-15 13:15:47 +09:00
leeguooooo 2707ceb1c4 feat(text): frame-aware text extraction — get text --all-frames / --main + frames (#27)
On listing/marketplace pages (Yahoo Auctions, Rakuten, Mercari shops) the
seller's description lives in a child frame or under a related-items sidebar,
so 'get text body' returned only header/nav boilerplate.

- get text --all-frames: aggregate visible text across every reachable frame.
  Same-process child frames are read via Page.createIsolatedWorld; OOPIFs via
  their auto-attached debugger session (iframe_sessions). Each non-top frame is
  labelled with a '----- frame [kind] url -----' separator.
- get text --main: readability-lite — prefer the densest <main>/<article>
  region over the whole body, dropping global header/nav/footer chrome.
- frames: enumerate frames (kind + url + per-frame text length) so an agent can
  see where a page's text actually lives and pick the right read.

Verified live: inline srcdoc frame text aggregated through --all-frames; Yahoo
Auctions <main> (2881 chars) extracted via --main, stripping the Yahoo header.
2026-06-15 12:51:59 +09:00
leeguooooo f7a657ac46 ci(release): group changelog by type ( Features / 🐛 Fixes / 🔧 Other)
Release notes were a flat list of commit subjects — hard to tell at a glance what
was added vs fixed (recurring '看不出改了什么'). Group by conventional-commit type
so every future release auto-shows scannable Features/Fixes sections.
2026-06-15 12:17:38 +09:00
leeguooooo 4e295ce139 fix(ab-connect): recover a churned-tabId session by stable CDP targetId (0.4.9, #24)
Live-reproduced #24 on 0.4.8 driving the Mercari signin token-exchange hop
(login.jp.mercari.com): the cross-process nav gives the tab a NEW Chrome tabId
while the CDP targetId stays the same. So cb-tab-<oldTabId> can't be recovered —
recoverSessionTab parsed the old tabId, chrome.tabs.get(oldTabId) failed (gone),
and it gave up → permanent 'stale sessionId ... its tab is gone' until the page
settled ~6s later and something re-attached. current/tab <targetId>/daemon
restart all failed because the relay still mapped the targetId to the dead
session.

Fix: remember each session's targetId across detach (sessionTargets map). When
recoverSessionTab can't recover by the encoded tabId, fall back to the STABLE
targetId — chrome.debugger.getTargets() to find the tab now hosting that target,
attach it, and ALIAS the dead cb-tab-<oldTabId> session to the live tab so the
daemon's session id keeps resolving. Longer retry window (~6s) since this hop
takes seconds to settle. Builds on 0.4.6/0.4.8 reattach; covers the tabId-churn
case those missed.

Needs dogfood on the real Mercari flow (can't repro the tabId churn synthetically).
2026-06-15 12:08:32 +09:00
leeguooooo 3d82f11ff2 chore(release): 1.5.0 — text-selector click + get text→body + tab --activate + click --follow/openedTab (#24); fill fires input/change/blur (#25); close <tab> wording + chrome-use current (#26)
Release binaries / Build macOS ARM64 (push) Has been cancelled
Release binaries / Build macOS x64 (push) Has been cancelled
Release binaries / Build Linux ARM64 (push) Has been cancelled
Release binaries / Build Linux musl ARM64 (push) Has been cancelled
Release binaries / Build Linux musl x64 (push) Has been cancelled
Release binaries / Build Linux x64 (push) Has been cancelled
Release binaries / Build Windows x64 (push) Has been cancelled
Release binaries / Attach binaries to GitHub Release (push) Has been cancelled
2026-06-15 11:32:30 +09:00
leeguooooo 33269adc1a fix(fill/tabs): dispatch real input/change/blur (#25); close <tab> wording + chrome-use current (#26)
#25 — fill() didn't fire the events framework inputs / site autocomplete need:
it set value directly (bypassing React's value-tracker) and typed via
Input.insertText, so controlled components and input/change/blur listeners (e.g.
Mercari's postal-code → 都道府県 lookup) never ran though the value showed. fill
now emulates a real edit: focus, set through the element's prototype value setter
(React _valueTracker registers), then dispatch input → input → change → blur/
focusout. SELECT and contenteditable handled too. type <sel> <text> remains for
per-keystroke sites. Verified live: an input wired with input/change/blur fired
'IICB' from one fill.

#26 (ergonomics):
- 'close <tab>' now closes just that tab and prints 'Tab [tN] closed'; bare
  'close' still closes the browser. Previously 'close t12' ran a browser close
  and alarmingly printed 'Browser closed'.
- new 'chrome-use current': prints the active tab's stable handle (tabId + CDP
  targetId + url/title), refreshed live — so an agent holds the targetId (which
  survives cross-process nav) instead of re-deriving 'which tab is live' from
  'tabs' every step. The deeper tab-id churn is the #21/#23 stable-targetId story.

Tests cover fill events (live), close tab-vs-browser parse, and current.
2026-06-15 11:26:34 +09:00
leeguooooo 9ab8753b48 feat(click): report (and optionally --follow) a tab opened by a click (#24-A)
A click on a target=_blank link / window.open opened a new tab, but the active
tab stayed put, so the post-click snapshot showed the OLD page — looking exactly
like the click failed. On the relay the new tab is discovered only via getTargets
(the relay doesn't push target events to the daemon), so it went unsurfaced.

handle_click now snapshots tracked targets before the click and, after, runs a
lightweight BrowserManager::adopt_newly_opened (one getTargets, attaches only the
new target — far cheaper than a full resync) to detect a freshly-opened tab. It's
reported as openedTab {tabId,url,title} in the response (and a '→ opened new tab
[tN] <url>' hint in text mode). Default keeps focus on the current tab (so
multi-tab flows aren't hijacked, per #7/#8.1); 'click <sel> --follow' switches to
the new tab. Verified live: clicking a _blank link prints
'→ opened new tab [t13] https://example.org/'.

Completes the #24 friction items (B/C/D shipped in 770708b).
2026-06-15 11:13:34 +09:00
leeguooooo 770708b8e6 fix(cli): text-selector click by visible label + get text→body + tab --activate (#24)
Three CLI gaps surfaced driving a Mercari signup→checkout flow:

- #24-B (correctness): a bare label like 'click 購入手続きへ' was fed straight to
  document.querySelector as CSS and failed as an invalid selector, even though
  snapshot listed the button by that exact name. build_find_element_js now tries
  CSS first, then falls back to matching an interactive element by visible text
  (exact then contains) — nested and non-ASCII labels resolve. 'text=<label>'
  forces the text path. CSS still wins when it matches.
- #24-D: 'get text' with no selector now returns the whole page (body).
- #24-C: 'tab <ref> --activate' (alias --front) switches to the tab AND raises it
  to the foreground — to surface a specific tab for the human.

Tests cover the text fallback / text= / xpath builder, body default, activate
flag. The core stale-sessionId-after-cross-process-nav bug is the #20/#23 class,
already fixed in ext 0.4.8 — needs that extension deployed.
2026-06-15 11:00:49 +09:00
leeguooooo 7c594820da docs(stream): document the bidirectional WS as the real-time driving path
Dogfooding (driving a canvas game) showed the slow, low-fidelity way — one
screenshot + one CLI call per action — when chrome-use already ships the right
tool: the session WebSocket is BIDIRECTIONAL. It streams ~60fps screencast frames
AND accepts input_keyboard/input_mouse/input_touch on the same socket, straight
to CDP Input.dispatch* — verified live over the extension relay (217 frames in
3.4s, ~64fps, and the input drove the game). But the inbound input protocol was
undocumented, so agents default to the CLI-per-action grind.

Document it in --help (stream) and the core skill: the frame + input message
schemas and the 'connect once, read frames, send timed input' loop, with a node
snippet. Reserve screenshots for one-off checks; use the WS for sustained
real-time control.
2026-06-14 00:54:30 +09:00
leeguooooo 81d18bbd2e feat(input): press --hold <ms> for precise timed key-holds + document timed-driving pattern
Dogfooding by driving a canvas game surfaced that per-action shell round-trips
(keydown; sleep; keyup) are the slowest, lowest-fidelity way to drive anything
timed — each is a process spawn + relay round-trip with ~250ms jitter, so a
'0.8s hold' is anything but.

- 'press <key> --hold <ms>': keyDown, wait, keyUp all inside the daemon, so the
  hold duration is precise and it's one round-trip. For games (hold-to-move/
  charge) and any press-and-hold.
- Documented the real driving pattern in the core skill + --help: script a timed
  sequence in ONE round-trip with 'batch "press d --hold 900" "press j" "wait 200"'
  (batch sends each step to the running daemon; --hold/wait block in-daemon), and
  prefer reading engine state via main-world 'eval' over guessing from pixels.

Parser test covers plain/held/missing-duration. Builds on the keydown/keyup full
descriptor fix.
2026-06-14 00:43:52 +09:00
17 changed files with 1627 additions and 141 deletions
+19 -4
View File
@@ -147,16 +147,31 @@ jobs:
git fetch --tags --force --quiet origin 2>/dev/null || true git fetch --tags --force --quiet origin 2>/dev/null || true
TAG="${{ github.event.inputs.tag || github.ref_name }}" TAG="${{ github.event.inputs.tag || github.ref_name }}"
PREV="$(git describe --tags --abbrev=0 "${TAG}^" 2>/dev/null || true)" PREV="$(git describe --tags --abbrev=0 "${TAG}^" 2>/dev/null || true)"
RANGE="${TAG}"
[ -n "$PREV" ] && RANGE="${PREV}..${TAG}"
# Group commit subjects by conventional-commit type so the notes are
# scannable ("what's new / what's fixed") instead of a flat dev log.
LOG="$(git log "$RANGE" --no-merges --pretty='%s' | grep -v '^chore(release)' || true)"
# NOTE: the job runs under `bash -e`. grep returning 1 (no match) and
# the `[ -n "$body" ]` test returning 1 (empty section) must NOT abort
# the script — otherwise a release whose commit range lacks a whole
# category (e.g. only `feat`, no `fix`) dies before writing the closing
# heredoc delimiter and the whole release step fails. `|| true` +
# `return 0` keep section() always-succeeding.
section() { # $1=header $2=grep-pattern
local body; body="$(printf '%s\n' "$LOG" | grep -E "$2" | sed 's/^/- /' || true)"
[ -n "$body" ] && printf '\n### %s\n%s\n' "$1" "$body"
return 0
}
{ {
echo "notes<<__NOTES_EOF__" echo "notes<<__NOTES_EOF__"
echo "## What changed" echo "## What changed"
echo "" section "✨ Features" '^feat'
section "🐛 Fixes" '^fix'
section "🔧 Other" '^(perf|refactor|docs|build|ci|test|style|revert)'
if [ -n "$PREV" ]; then if [ -n "$PREV" ]; then
git log "${PREV}..${TAG}" --no-merges --pretty='- %s' | grep -v '^- chore(release)' || true
echo "" echo ""
echo "**Full changelog**: https://github.com/${{ github.repository }}/compare/${PREV}...${TAG}" echo "**Full changelog**: https://github.com/${{ github.repository }}/compare/${PREV}...${TAG}"
else
git log "${TAG}" --no-merges --pretty='- %s' | grep -v '^- chore(release)' || true
fi fi
echo "__NOTES_EOF__" echo "__NOTES_EOF__"
} >> "$GITHUB_OUTPUT" } >> "$GITHUB_OUTPUT"
+1 -1
View File
@@ -290,7 +290,7 @@ checksum = "613afe47fcd5fac7ccf1db93babcb082c5994d996f20b8b159f2ad1658eb5724"
[[package]] [[package]]
name = "chrome-use" name = "chrome-use"
version = "1.4.1" version = "1.5.8"
dependencies = [ dependencies = [
"aes", "aes",
"aes-gcm", "aes-gcm",
+1 -1
View File
@@ -1,6 +1,6 @@
[package] [package]
name = "chrome-use" name = "chrome-use"
version = "1.4.1" version = "1.5.8"
edition = "2021" edition = "2021"
description = "Fast browser automation CLI for AI agents" description = "Fast browser automation CLI for AI agents"
license = "Apache-2.0" license = "Apache-2.0"
+306 -29
View File
@@ -30,12 +30,65 @@ pub enum ParseError {
InvalidSessionName { name: String }, InvalidSessionName { name: String },
} }
/// Top-level commands an agent is likely to mistype, used for "did you mean"
/// suggestions on an unknown command (issue #29). Not exhaustive — just the
/// common verbs plus a few known wrong-guesses mapped to the real command.
const KNOWN_COMMANDS: &[&str] = &[
"open", "navigate", "click", "fill", "type", "press", "snapshot", "screenshot", "eval", "get",
"text", "html", "frames", "find", "wait", "scroll", "hover", "select", "check", "uncheck",
"tab", "tabs", "close", "back", "forward", "reload", "sessions", "status", "daemon", "doctor",
"upgrade", "connect", "cookies", "mouse", "keyboard", "stream", "frame", "profiles", "title",
"url", "is", "drag", "dialog", "upload",
];
/// Levenshtein distance, capped — small inputs only (command names).
fn edit_distance(a: &str, b: &str) -> usize {
let a: Vec<char> = a.chars().collect();
let b: Vec<char> = b.chars().collect();
let mut prev: Vec<usize> = (0..=b.len()).collect();
let mut curr = vec![0usize; b.len() + 1];
for (i, &ca) in a.iter().enumerate() {
curr[0] = i + 1;
for (j, &cb) in b.iter().enumerate() {
let cost = if ca == cb { 0 } else { 1 };
curr[j + 1] = (prev[j + 1] + 1).min(curr[j] + 1).min(prev[j] + cost);
}
std::mem::swap(&mut prev, &mut curr);
}
prev[b.len()]
}
/// Closest known command within a small edit distance, or a prefix/substring
/// match — `None` if nothing is close enough to suggest confidently.
fn nearest_command(input: &str) -> Option<String> {
let lower = input.to_lowercase();
// Exact prefix/substring hits first (e.g. "session" -> "sessions").
if let Some(c) = KNOWN_COMMANDS
.iter()
.find(|c| c.starts_with(&lower) || lower.starts_with(**c))
{
return Some(c.to_string());
}
// Tolerance scales with length: short words get distance 1, longer get 2.
let max_dist = if lower.len() <= 4 { 1 } else { 2 };
KNOWN_COMMANDS
.iter()
.map(|c| (*c, edit_distance(&lower, c)))
.filter(|(_, d)| *d <= max_dist)
.min_by_key(|(_, d)| *d)
.map(|(c, _)| c.to_string())
}
impl ParseError { impl ParseError {
pub fn format(&self) -> String { pub fn format(&self) -> String {
match self { match self {
ParseError::UnknownCommand { command } => { ParseError::UnknownCommand { command } => match nearest_command(command) {
format!("Unknown command: {}", command) Some(suggestion) => format!(
} "Unknown command: {}\nDid you mean: chrome-use {}?",
command, suggestion
),
None => format!("Unknown command: {}", command),
},
ParseError::UnknownSubcommand { ParseError::UnknownSubcommand {
subcommand, subcommand,
valid_options, valid_options,
@@ -424,6 +477,9 @@ fn parse_command_inner(args: &[String], flags: &Flags) -> Result<Value, ParseErr
// === Core Actions === // === Core Actions ===
"click" => { "click" => {
let new_tab = rest.contains(&"--new-tab"); let new_tab = rest.contains(&"--new-tab");
// `--follow`: if the click opens a new tab, switch the active tab to
// it (default reports the opened tab but stays put) (issue #24-A).
let follow = rest.contains(&"--follow");
// Coordinate click as a first-class form (issue #8.4): when the only // Coordinate click as a first-class form (issue #8.4): when the only
// handle is a pixel position, no element/selector is needed. // handle is a pixel position, no element/selector is needed.
// click <x> <y> e.g. click 449 320 // click <x> <y> e.g. click 449 320
@@ -432,23 +488,27 @@ fn parse_command_inner(args: &[String], flags: &Flags) -> Result<Value, ParseErr
let coord_args: Vec<&str> = rest let coord_args: Vec<&str> = rest
.iter() .iter()
.copied() .copied()
.filter(|a| *a != "--new-tab" && *a != "--coords") .filter(|a| !a.starts_with("--"))
.collect(); .collect();
if let Some((x, y)) = parse_coords(&coord_args) { if let Some((x, y)) = parse_coords(&coord_args) {
return Ok(json!({ "id": id, "action": "click", "x": x, "y": y })); return Ok(json!({ "id": id, "action": "click", "x": x, "y": y }));
} }
let sel = rest let sel = rest
.iter() .iter()
.find(|arg| **arg != "--new-tab") .find(|arg| !arg.starts_with("--"))
.ok_or_else(|| ParseError::MissingArguments { .ok_or_else(|| ParseError::MissingArguments {
context: "click".to_string(), context: "click".to_string(),
usage: "click <selector> | click <x> <y> | click --coords <x>,<y> [--new-tab]", usage:
"click <selector> | click <x> <y> | click --coords <x>,<y> [--new-tab] [--follow]",
})?; })?;
let mut cmd = json!({ "id": id, "action": "click", "selector": sel });
if new_tab { if new_tab {
Ok(json!({ "id": id, "action": "click", "selector": sel, "newTab": true })) cmd["newTab"] = json!(true);
} else {
Ok(json!({ "id": id, "action": "click", "selector": sel }))
} }
if follow {
cmd["follow"] = json!(true);
}
Ok(cmd)
} }
"dblclick" => { "dblclick" => {
let sel = rest.first().ok_or_else(|| ParseError::MissingArguments { let sel = rest.first().ok_or_else(|| ParseError::MissingArguments {
@@ -583,11 +643,27 @@ fn parse_command_inner(args: &[String], flags: &Flags) -> Result<Value, ParseErr
// === Keyboard === // === Keyboard ===
"press" | "key" => { "press" | "key" => {
let key = rest.first().ok_or_else(|| ParseError::MissingArguments { let key = rest.iter().find(|a| !a.starts_with("--")).ok_or_else(|| {
ParseError::MissingArguments {
context: "press".to_string(), context: "press".to_string(),
usage: "press <key>", usage: "press <key> [--hold <ms>]",
}
})?; })?;
Ok(json!({ "id": id, "action": "press", "key": key })) let mut c = json!({ "id": id, "action": "press", "key": key });
// `--hold <ms>`: hold the key down for <ms> then release, timed inside
// the daemon (one round-trip, no shell-sleep jitter) — for games and
// hold-to-charge where keydown+sleep+keyup over 3 round-trips is too
// imprecise.
if let Some(i) = rest.iter().position(|a| *a == "--hold") {
let ms = rest.get(i + 1).and_then(|s| s.parse::<u64>().ok()).ok_or(
ParseError::MissingArguments {
context: "press --hold".to_string(),
usage: "press <key> --hold <ms>",
},
)?;
c["hold"] = json!(ms);
}
Ok(c)
} }
"keydown" => { "keydown" => {
let key = rest.first().ok_or_else(|| ParseError::MissingArguments { let key = rest.first().ok_or_else(|| ParseError::MissingArguments {
@@ -1002,8 +1078,31 @@ fn parse_command_inner(args: &[String], flags: &Flags) -> Result<Value, ParseErr
Ok(json!({ "id": id, "action": "stealth_status" })) Ok(json!({ "id": id, "action": "stealth_status" }))
} }
// `cf-status` — Cloudflare challenge/clearance preflight: is the page
// currently a CF challenge, and is there a still-valid cf_clearance (the
// HttpOnly persistence cookie)? Lets an agent SKIP re-solving when already
// cleared, and know when it must solve. Persistence optimization.
"cf-status" | "cf" | "cloudflare-status" | "clearance" => {
Ok(json!({ "id": id, "action": "cf_status" }))
}
// === Close === // === Close ===
"close" | "quit" | "exit" => Ok(json!({ "id": id, "action": "close" })), "close" | "quit" | "exit" => {
// `close <tab>` closes only that tab (and the output says "Tab
// closed"); bare `close` closes the browser/session. `close --all` is
// intercepted earlier in the dispatcher. Previously `close t12` still
// ran a browser close and alarmingly printed "Browser closed" (#26).
if let Some(tab_ref) = rest.iter().find(|a| !a.starts_with("--")) {
Ok(json!({ "id": id, "action": "tab_close", "tabId": tab_ref }))
} else {
Ok(json!({ "id": id, "action": "close" }))
}
}
// The active tab's stable handle — `targetId` survives cross-process
// navigation and is reusable across sessions, so an agent can hold it
// instead of re-deriving "which tab is live" from `tabs` each step (#26).
"current" => Ok(json!({ "id": id, "action": "current" })),
// === Inspect === // === Inspect ===
"inspect" => Ok(json!({ "id": id, "action": "inspect" })), "inspect" => Ok(json!({ "id": id, "action": "inspect" })),
@@ -1258,6 +1357,11 @@ fn parse_command_inner(args: &[String], flags: &Flags) -> Result<Value, ParseErr
// === Get === // === Get ===
"get" => parse_get(&rest, &id), "get" => parse_get(&rest, &id),
// List every frame the session can reach (top + same-process child
// frames + out-of-process iframes), with a text-length per frame so you
// can see where a listing's description actually lives (issue #27).
"frames" => Ok(json!({ "id": id, "action": "frames" })),
// Top-level shortcuts for `get <x>` status reads — users naturally type // Top-level shortcuts for `get <x>` status reads — users naturally type
// `chrome-use url` / `cdp-url` / `title` without the `get` prefix // `chrome-use url` / `cdp-url` / `title` without the `get` prefix
// (and expect `cdp-url`/`cdp_url` to work interchangeably). // (and expect `cdp-url`/`cdp_url` to work interchangeably).
@@ -1559,11 +1663,16 @@ fn parse_command_inner(args: &[String], flags: &Flags) -> Result<Value, ParseErr
} }
Ok(cmd) Ok(cmd)
} }
Some(tab_ref) => Ok(json!({ Some(tab_ref) => {
"id": id, // `tab <ref> --activate` (alias `--front`) switches to the tab
"action": "tab_switch", // AND raises it to the foreground — for handing a specific tab
"tabId": tab_ref, // to the human (SMS code, captcha) (issue #24-C).
})), let mut cmd = json!({ "id": id, "action": "tab_switch", "tabId": tab_ref });
if rest.iter().any(|a| *a == "--activate" || *a == "--front") {
cmd["activate"] = json!(true);
}
Ok(cmd)
}
None => { None => {
let mut cmd = json!({ "id": id, "action": "tab_list" }); let mut cmd = json!({ "id": id, "action": "tab_list" });
if full { if full {
@@ -2345,11 +2454,42 @@ fn parse_get(rest: &[&str], id: &str) -> Result<Value, ParseError> {
match rest.first().copied() { match rest.first().copied() {
Some("text") => { Some("text") => {
let sel = rest.get(1).ok_or_else(|| ParseError::MissingArguments { // `get text --all-frames` aggregates visible text across every
context: "get text".to_string(), // frame, including out-of-process iframes invisible to the top
usage: "get text <selector>", // document (issue #27). The selector is ignored in this mode.
})?; let all_frames = rest[1..]
Ok(json!({ "id": id, "action": "gettext", "selector": sel })) .iter()
.any(|a| matches!(*a, "--all-frames" | "--frames" | "-a"));
if all_frames {
return Ok(json!({ "id": id, "action": "gettext", "allFrames": true }));
}
// `get text --pierce` reads through CLOSED shadow DOM / child docs
// via the CDP DOM tree — content eval/innerText can't reach, e.g. an
// extension's injected panel in a closed shadow root (issue #30).
let pierce = rest[1..]
.iter()
.any(|a| matches!(*a, "--pierce" | "--shadow" | "--deep"));
if pierce {
return Ok(json!({ "id": id, "action": "gettext", "pierce": true }));
}
// `get text --main` returns the main-content region (readability),
// skipping header/nav/footer/sidebar boilerplate (issue #27).
let main = rest[1..]
.iter()
.any(|a| matches!(*a, "--main" | "--readable" | "-m"));
if main {
return Ok(json!({ "id": id, "action": "gettext", "main": true }));
}
// `get text` with no selector reads the WHOLE PAGE and now defaults
// to cross-frame aggregation, so an agent gets a page's iframed
// content (listing descriptions etc.) without having to know about
// `--all-frames` (#27). On a single-frame page this is identical to
// the old body read; multi-frame pages get the child frames too —
// a strict superset. An explicit selector stays element-scoped.
match rest.iter().skip(1).find(|a| !a.starts_with("--")).copied() {
Some(sel) => Ok(json!({ "id": id, "action": "gettext", "selector": sel })),
None => Ok(json!({ "id": id, "action": "gettext", "allFrames": true })),
}
} }
Some("html") => { Some("html") => {
let sel = rest.get(1).ok_or_else(|| ParseError::MissingArguments { let sel = rest.get(1).ok_or_else(|| ParseError::MissingArguments {
@@ -3581,6 +3721,22 @@ mod tests {
assert_eq!(cmd["url"], "https://example.com"); assert_eq!(cmd["url"], "https://example.com");
} }
#[test]
fn test_press_plain_and_hold() {
let cmd = parse_command(&args("press d"), &default_flags()).unwrap();
assert_eq!(cmd["action"], "press");
assert_eq!(cmd["key"], "d");
assert!(cmd.get("hold").is_none());
let held = parse_command(&args("press d --hold 800"), &default_flags()).unwrap();
assert_eq!(held["key"], "d");
assert_eq!(held["hold"], 800);
// Missing/invalid duration is an error, not a silent no-hold.
assert!(parse_command(&args("press d --hold"), &default_flags()).is_err());
assert!(parse_command(&args("press d --hold abc"), &default_flags()).is_err());
}
#[test] #[test]
fn test_navigate_reuse_tab_flag() { fn test_navigate_reuse_tab_flag() {
let cmd = parse_command( let cmd = parse_command(
@@ -3747,6 +3903,21 @@ mod tests {
assert!(cmd.get("x").is_none()); assert!(cmd.get("x").is_none());
} }
#[test]
fn test_click_follow_flag() {
// `--follow` sets the flag; the selector is still found even with the flag
// before it (issue #24-A).
let cmd = parse_command(&args("click @e5 --follow"), &default_flags()).unwrap();
assert_eq!(cmd["selector"], "@e5");
assert_eq!(cmd["follow"], true);
let cmd2 = parse_command(&args("click --follow @e5"), &default_flags()).unwrap();
assert_eq!(cmd2["selector"], "@e5");
assert_eq!(cmd2["follow"], true);
// Absent by default.
let plain = parse_command(&args("click @e5"), &default_flags()).unwrap();
assert!(plain.get("follow").is_none());
}
#[test] #[test]
fn test_tabs_alias_lists() { fn test_tabs_alias_lists() {
assert_eq!( assert_eq!(
@@ -3955,6 +4126,28 @@ mod tests {
assert_eq!(cmd["tabId"], "docs"); assert_eq!(cmd["tabId"], "docs");
} }
#[test]
fn test_close_tab_vs_browser() {
// `close <tab>` closes that tab (says "Tab closed"); bare `close` closes
// the browser (#26).
let tab = parse_command(&args("close t12"), &default_flags()).unwrap();
assert_eq!(tab["action"], "tab_close");
assert_eq!(tab["tabId"], "t12");
let browser = parse_command(&args("close"), &default_flags()).unwrap();
assert_eq!(browser["action"], "close");
// `quit`/`exit` aliases still browser-close.
assert_eq!(
parse_command(&args("quit"), &default_flags()).unwrap()["action"],
"close"
);
}
#[test]
fn test_current_command() {
let cmd = parse_command(&args("current"), &default_flags()).unwrap();
assert_eq!(cmd["action"], "current");
}
#[test] #[test]
fn test_tab_sends_string_tab_id() { fn test_tab_sends_string_tab_id() {
let cmd = parse_command(&args("tab t2"), &default_flags()).unwrap(); let cmd = parse_command(&args("tab t2"), &default_flags()).unwrap();
@@ -4654,12 +4847,96 @@ mod tests {
} }
#[test] #[test]
fn test_get_text_missing_selector() { fn test_get_text_defaults_to_all_frames() {
let result = parse_command(&args("get text"), &default_flags()); // `get text` with no selector now reads the whole page across ALL frames
assert!(result.is_err()); // by default (#27), so iframed content isn't silently missed. (Was: a
let err = result.unwrap_err(); // top-frame `body` read, #24-D.)
assert!(matches!(err, ParseError::MissingArguments { .. })); let cmd = parse_command(&args("get text"), &default_flags()).unwrap();
assert!(err.format().contains("get text")); assert_eq!(cmd["action"], "gettext");
assert_eq!(cmd["allFrames"], true);
assert!(cmd.get("selector").is_none());
// An explicit selector still wins and stays element-scoped.
let cmd2 = parse_command(&args("get text h1"), &default_flags()).unwrap();
assert_eq!(cmd2["selector"], "h1");
assert!(cmd2.get("allFrames").is_none());
// `text` top-level shortcut behaves the same.
let cmd3 = parse_command(&args("text"), &default_flags()).unwrap();
assert_eq!(cmd3["allFrames"], true);
}
#[test]
fn test_get_text_all_frames() {
// `--all-frames` switches to whole-page, cross-frame aggregation and
// drops the selector (issue #27).
for variant in ["get text --all-frames", "get text --frames", "text -a"] {
let cmd = parse_command(&args(variant), &default_flags()).unwrap();
assert_eq!(cmd["action"], "gettext", "{variant}");
assert_eq!(cmd["allFrames"], true, "{variant}");
assert!(cmd.get("selector").is_none(), "{variant}");
}
// A flag mixed with a selector still triggers all-frames.
let cmd = parse_command(&args("get text body --all-frames"), &default_flags()).unwrap();
assert_eq!(cmd["allFrames"], true);
// Without the flag, a leading flag-like token is skipped for the selector.
let cmd = parse_command(&args("get text main"), &default_flags()).unwrap();
assert_eq!(cmd["selector"], "main");
assert!(cmd.get("allFrames").is_none());
}
#[test]
fn test_frames_command() {
let cmd = parse_command(&args("frames"), &default_flags()).unwrap();
assert_eq!(cmd["action"], "frames");
}
#[test]
fn test_nearest_command_suggestions() {
assert_eq!(nearest_command("sesions").as_deref(), Some("sessions"));
assert_eq!(nearest_command("session").as_deref(), Some("sessions"));
assert_eq!(nearest_command("clik").as_deref(), Some("click"));
assert_eq!(nearest_command("screenshits").as_deref(), Some("screenshot"));
// Nonsense with no close match stays silent.
assert_eq!(nearest_command("xyzzy"), None);
// The unknown-command error embeds the suggestion.
let err = ParseError::UnknownCommand {
command: "sesions".to_string(),
};
assert!(err.format().contains("Did you mean: chrome-use sessions?"));
}
#[test]
fn test_get_text_main() {
for variant in ["get text --main", "get text --readable", "text -m"] {
let cmd = parse_command(&args(variant), &default_flags()).unwrap();
assert_eq!(cmd["action"], "gettext", "{variant}");
assert_eq!(cmd["main"], true, "{variant}");
assert!(cmd.get("selector").is_none(), "{variant}");
}
}
#[test]
fn test_get_text_pierce() {
for variant in ["get text --pierce", "get text --shadow", "text --deep"] {
let cmd = parse_command(&args(variant), &default_flags()).unwrap();
assert_eq!(cmd["action"], "gettext", "{variant}");
assert_eq!(cmd["pierce"], true, "{variant}");
assert!(cmd.get("selector").is_none(), "{variant}");
}
}
#[test]
fn test_tab_activate_flag() {
let plain = parse_command(&args("tab t3"), &default_flags()).unwrap();
assert_eq!(plain["action"], "tab_switch");
assert!(plain.get("activate").is_none());
let act = parse_command(&args("tab t3 --activate"), &default_flags()).unwrap();
assert_eq!(act["action"], "tab_switch");
assert_eq!(act["tabId"], "t3");
assert_eq!(act["activate"], true);
// `--front` alias.
let front = parse_command(&args("tab t3 --front"), &default_flags()).unwrap();
assert_eq!(front["activate"], true);
} }
// === Protocol alignment tests === // === Protocol alignment tests ===
+41
View File
@@ -449,6 +449,47 @@ pub fn relay_url() -> Option<String> {
} }
} }
/// Append a one-line record of how a CDP connection was established, to
/// `~/.chrome-use/connect-mode.log`. This is the smoking-gun detector for the
/// "Allow remote debugging?" consent modal: that modal ONLY appears on a raw
/// remote-debugging attach / a browser we launched with a debug port — NEVER on
/// the extension relay. When the modal reappears, this log says which session
/// took which path and when, so we can tell a code regression (`raw-port` /
/// `launched` while the relay was up) from Chrome's own extension-debugger
/// consent UX. Low volume (one line per connection); best-effort, never fails a
/// connection.
pub fn log_connect_mode(ws_url: &str, launched: bool, session: &str) {
let relay = relay_url();
let relay_up = relay.is_some();
let mode = if launched {
"launched(debug-port)"
} else if relay.as_deref() == Some(ws_url) {
"relay"
} else if ws_url.contains("127.0.0.1") || ws_url.contains("localhost") {
"raw-port-attach"
} else {
"remote-ws"
};
// A raw-port attach or a self-launch while the relay was available is the
// exact thing that pops the consent modal — flag it loudly in the line.
let suspect = (mode == "raw-port-attach" || launched) && relay_up;
let line = format!(
"session={session} mode={mode} relay_up={relay_up}{} ws={ws_url}\n",
if suspect { " CONSENT-MODAL-RISK" } else { "" }
);
if let Some(home) = dirs::home_dir() {
let path = home.join(".chrome-use").join("connect-mode.log");
use std::io::Write;
if let Ok(mut f) = std::fs::OpenOptions::new()
.create(true)
.append(true)
.open(&path)
{
let _ = f.write_all(line.as_bytes());
}
}
}
/// Sidecar recording the connected extension's version, written by the host when /// Sidecar recording the connected extension's version, written by the host when
/// it receives the extension's `hello` (sibling of `relay-cdp-url`). Lets /// it receives the extension's `hello` (sibling of `relay-cdp-url`). Lets
/// `doctor` surface which extension build is live without a CDP round-trip. /// `doctor` surface which extension build is live without a CDP round-trip.
+8
View File
@@ -893,6 +893,14 @@ fn main() {
return; return;
} }
// `sessions` is a natural top-level guess for "list my sessions" (the skill
// advertises sessions as a feature) — route it to the daemon inventory the
// same way `daemon status` does (issue #29).
if clean.first().map(|s| s.as_str()) == Some("sessions") {
run_daemon(&["sessions".to_string(), "status".to_string()], flags.json);
return;
}
// Handle close --all: close all active sessions // Handle close --all: close all active sessions
if matches!( if matches!(
clean.first().map(|s| s.as_str()), clean.first().map(|s| s.as_str()),
+279 -2
View File
@@ -1332,6 +1332,7 @@ pub async fn execute_command(cmd: &Value, state: &mut DaemonState) -> Value {
"uncheck" => handle_uncheck(cmd, state).await, "uncheck" => handle_uncheck(cmd, state).await,
"wait" => handle_wait(cmd, state).await, "wait" => handle_wait(cmd, state).await,
"gettext" => handle_gettext(cmd, state).await, "gettext" => handle_gettext(cmd, state).await,
"frames" => handle_frames(cmd, state).await,
"getattribute" => handle_getattribute(cmd, state).await, "getattribute" => handle_getattribute(cmd, state).await,
"isvisible" => handle_isvisible(cmd, state).await, "isvisible" => handle_isvisible(cmd, state).await,
"isenabled" => handle_isenabled(cmd, state).await, "isenabled" => handle_isenabled(cmd, state).await,
@@ -1340,6 +1341,7 @@ pub async fn execute_command(cmd: &Value, state: &mut DaemonState) -> Value {
"forward" => handle_forward(state).await, "forward" => handle_forward(state).await,
"reload" => handle_reload(state).await, "reload" => handle_reload(state).await,
"cookies_get" => handle_cookies_get(cmd, state).await, "cookies_get" => handle_cookies_get(cmd, state).await,
"cf_status" => handle_cf_status(cmd, state).await,
"cookies_set" => handle_cookies_set(cmd, state).await, "cookies_set" => handle_cookies_set(cmd, state).await,
"cookies_clear" => handle_cookies_clear(state).await, "cookies_clear" => handle_cookies_clear(state).await,
"storage_get" => handle_storage_get(cmd, state).await, "storage_get" => handle_storage_get(cmd, state).await,
@@ -1395,6 +1397,7 @@ pub async fn execute_command(cmd: &Value, state: &mut DaemonState) -> Value {
"count" => handle_count(cmd, state).await, "count" => handle_count(cmd, state).await,
"styles" => handle_styles(cmd, state).await, "styles" => handle_styles(cmd, state).await,
"bringtofront" => handle_bringtofront(state).await, "bringtofront" => handle_bringtofront(state).await,
"current" => handle_current(state).await,
"timezone" => handle_timezone(cmd, state).await, "timezone" => handle_timezone(cmd, state).await,
"locale" => handle_locale(cmd, state).await, "locale" => handle_locale(cmd, state).await,
"geolocation" => handle_geolocation(cmd, state).await, "geolocation" => handle_geolocation(cmd, state).await,
@@ -3116,6 +3119,15 @@ async fn handle_click(cmd: &Value, state: &mut DaemonState) -> Result<Value, Str
let button = cmd.get("button").and_then(|v| v.as_str()).unwrap_or("left"); let button = cmd.get("button").and_then(|v| v.as_str()).unwrap_or("left");
let click_count = cmd.get("clickCount").and_then(|v| v.as_i64()).unwrap_or(1) as i32; let click_count = cmd.get("clickCount").and_then(|v| v.as_i64()).unwrap_or(1) as i32;
let follow = cmd.get("follow").and_then(|v| v.as_bool()).unwrap_or(false);
// Snapshot tracked targets so we can tell if this click opened a NEW tab
// (target=_blank link / window.open). On the relay the new tab is discovered
// passively and doesn't steal focus (#7/#8.1), so without surfacing it the
// post-click snapshot shows the OLD page and looks like the click failed
// (issue #24-A).
let before: std::collections::HashSet<String> =
mgr.pages_list().into_iter().map(|p| p.target_id).collect();
interaction::click( interaction::click(
&mgr.client, &mgr.client,
@@ -3128,7 +3140,26 @@ async fn handle_click(cmd: &Value, state: &mut DaemonState) -> Result<Value, Str
) )
.await?; .await?;
Ok(json!({ "clicked": selector })) // Give a just-opened tab a moment to register, then look for it.
let mgr = state.browser.as_mut().ok_or("Browser not launched")?;
tokio::time::sleep(std::time::Duration::from_millis(150)).await;
let opened = mgr.adopt_newly_opened(&before).await;
let mut out = json!({ "clicked": selector });
if let Some(page) = opened {
let tab_id = super::browser::format_tab_id(page.tab_id);
out["openedTab"] = json!({ "tabId": tab_id, "url": page.url, "title": page.title });
// `--follow`: switch the active tab to the newly-opened one (default is
// to report it but stay put, so multi-tab flows aren't hijacked).
if follow {
state.ref_map.clear();
state.iframe_sessions.clear();
state.active_frame_id = None;
let _ = mgr.tab_switch_by_id(page.tab_id).await;
out["followed"] = json!(true);
}
}
Ok(out)
} }
async fn handle_dblclick(cmd: &Value, state: &mut DaemonState) -> Result<Value, String> { async fn handle_dblclick(cmd: &Value, state: &mut DaemonState) -> Result<Value, String> {
@@ -3334,6 +3365,16 @@ async fn handle_press(cmd: &Value, state: &mut DaemonState) -> Result<Value, Str
// Parse modifier+key chords like "Control+a", "Shift+Enter", "Control+Shift+a" // Parse modifier+key chords like "Control+a", "Shift+Enter", "Control+Shift+a"
let (actual_key, modifiers) = parse_key_chord(key); let (actual_key, modifiers) = parse_key_chord(key);
// `--hold <ms>`: keyDown, wait, keyUp — all inside the daemon so the hold
// duration is precise (no shell-sleep / round-trip jitter). For games
// (hold-to-move/charge) and any press-and-hold interaction.
if let Some(ms) = cmd.get("hold").and_then(|v| v.as_u64()) {
interaction::dispatch_single_key(&mgr.client, &session_id, &actual_key, "keyDown").await?;
tokio::time::sleep(std::time::Duration::from_millis(ms)).await;
interaction::dispatch_single_key(&mgr.client, &session_id, &actual_key, "keyUp").await?;
return Ok(json!({ "pressed": key, "heldMs": ms }));
}
interaction::press_key_with_modifiers(&mgr.client, &session_id, &actual_key, modifiers).await?; interaction::press_key_with_modifiers(&mgr.client, &session_id, &actual_key, modifiers).await?;
Ok(json!({ "pressed": key })) Ok(json!({ "pressed": key }))
} }
@@ -3555,6 +3596,56 @@ async fn handle_wait(cmd: &Value, state: &mut DaemonState) -> Result<Value, Stri
async fn handle_gettext(cmd: &Value, state: &mut DaemonState) -> Result<Value, String> { async fn handle_gettext(cmd: &Value, state: &mut DaemonState) -> Result<Value, String> {
let mgr = state.browser.as_ref().ok_or("Browser not launched")?; let mgr = state.browser.as_ref().ok_or("Browser not launched")?;
let session_id = mgr.active_session_id()?.to_string(); let session_id = mgr.active_session_id()?.to_string();
// `get text --all-frames` aggregates visible text across every frame the
// session can reach — including out-of-process iframes that never show up
// in the top document (#27: Yahoo/Rakuten/Mercari listing descriptions).
if cmd.get("allFrames").and_then(|v| v.as_bool()) == Some(true) {
let frames = super::element::collect_all_frames_text(
&mgr.client,
&session_id,
&state.iframe_sessions,
)
.await?;
let mut combined = String::new();
let mut frame_count = 0usize;
for f in &frames {
let t = f.text.trim();
if t.is_empty() {
continue;
}
frame_count += 1;
if f.kind != "top" {
combined.push_str(&format!("\n\n----- frame [{}] {} -----\n", f.kind, f.url));
}
combined.push_str(t);
}
let url = mgr.get_url().await.unwrap_or_default();
return Ok(json!({
"text": combined,
"origin": url,
"frames": frame_count,
"allFrames": true,
}));
}
// `get text --pierce` reads text through CLOSED shadow roots and child
// documents via the CDP DOM tree — content `innerText`/`eval` can't see,
// e.g. an extension's injected panel in a closed shadow DOM (#30).
if cmd.get("pierce").and_then(|v| v.as_bool()) == Some(true) {
let text = super::element::get_pierced_text(&mgr.client, &session_id).await?;
let url = mgr.get_url().await.unwrap_or_default();
return Ok(json!({ "text": text, "origin": url, "pierce": true }));
}
// `get text --main` returns the page's main-content region (readability-lite),
// skipping global header/nav/footer/sidebar boilerplate (#27).
if cmd.get("main").and_then(|v| v.as_bool()) == Some(true) {
let text = super::element::get_main_content_text(&mgr.client, &session_id).await?;
let url = mgr.get_url().await.unwrap_or_default();
return Ok(json!({ "text": text, "origin": url, "main": true }));
}
let selector = cmd let selector = cmd
.get("selector") .get("selector")
.and_then(|v| v.as_str()) .and_then(|v| v.as_str())
@@ -3572,6 +3663,32 @@ async fn handle_gettext(cmd: &Value, state: &mut DaemonState) -> Result<Value, S
Ok(json!({ "text": text, "origin": url })) Ok(json!({ "text": text, "origin": url }))
} }
async fn handle_frames(_cmd: &Value, state: &mut DaemonState) -> Result<Value, String> {
let mgr = state.browser.as_ref().ok_or("Browser not launched")?;
let session_id = mgr.active_session_id()?.to_string();
let frames = super::element::collect_all_frames_text(
&mgr.client,
&session_id,
&state.iframe_sessions,
)
.await?;
let list: Vec<Value> = frames
.iter()
.enumerate()
.map(|(i, f)| {
json!({
"index": i,
"kind": f.kind,
"url": f.url,
"frameId": f.frame_id,
"textLen": f.text.trim().chars().count(),
})
})
.collect();
let url = mgr.get_url().await.unwrap_or_default();
Ok(json!({ "frames": list, "count": list.len(), "origin": url }))
}
async fn handle_getattribute(cmd: &Value, state: &mut DaemonState) -> Result<Value, String> { async fn handle_getattribute(cmd: &Value, state: &mut DaemonState) -> Result<Value, String> {
let mgr = state.browser.as_ref().ok_or("Browser not launched")?; let mgr = state.browser.as_ref().ok_or("Browser not launched")?;
let session_id = mgr.active_session_id()?.to_string(); let session_id = mgr.active_session_id()?.to_string();
@@ -4006,6 +4123,108 @@ async fn handle_cookies_clear(state: &DaemonState) -> Result<Value, String> {
Ok(json!({ "cleared": true })) Ok(json!({ "cleared": true }))
} }
// Detect whether the active page is *currently* a Cloudflare challenge
// (the full-page "Just a moment…" / "正在进行安全验证" interstitial), so an agent
// knows whether it must solve or can proceed. Runs in the top frame main world.
const CF_CHALLENGE_JS: &str = r#"(function(){
var t = document.title || '';
var challenged =
/just a moment|attention required|checking (your|if)|verify you are human|||||/i.test(t) ||
!!document.querySelector('#challenge-form, #challenge-running, #cf-challenge-running, [id^="cf-chl"], script[src*="/cdn-cgi/challenge-platform/"]');
var turnstile = !!document.querySelector('.cf-turnstile, [data-sitekey]');
return JSON.stringify({ title: t, challenged: challenged, turnstile: turnstile, readyState: document.readyState });
})()"#;
/// Recommendation for a Cloudflare-gated page, from the current challenge state
/// and whether a still-valid `cf_clearance` exists. Pure so it's unit-testable.
/// - not challenged → "proceed" (the page is cleared/loaded)
/// - challenged, valid cookie → "reissue" (clearance present but page still
/// blocks → it's stale or the IP/UA no longer matches what it was issued for)
/// - challenged, no cookie → "solve"
fn cf_recommendation(challenged: bool, clearance_valid: bool) -> &'static str {
if !challenged {
"proceed"
} else if clearance_valid {
"reissue"
} else {
"solve"
}
}
/// `cf_clearance` validity for a cookie's expiry (epoch seconds; <=0 = session
/// cookie, treated as non-expiring). Returns (present, expired). Pure.
fn clearance_state(expires: Option<f64>, now: f64) -> (bool, bool) {
match expires {
None => (false, false),
Some(e) if e <= 0.0 => (true, false), // session cookie: no expiry
Some(e) => (true, e < now),
}
}
async fn handle_cf_status(_cmd: &Value, state: &mut DaemonState) -> Result<Value, String> {
let mgr = state.browser.as_ref().ok_or("Browser not launched")?;
let session_id = mgr.active_session_id()?.to_string();
let url = mgr.get_url().await.unwrap_or_default();
// 1. Is the page a Cloudflare challenge right now?
let probe_raw = mgr.evaluate(CF_CHALLENGE_JS, None).await.unwrap_or(Value::Null);
let probe = parse_json_string(probe_raw, "cf challenge probe").unwrap_or(Value::Null);
let challenged = probe
.get("challenged")
.and_then(|v| v.as_bool())
.unwrap_or(false);
let turnstile = probe
.get("turnstile")
.and_then(|v| v.as_bool())
.unwrap_or(false);
let title = probe
.get("title")
.and_then(|v| v.as_str())
.unwrap_or("")
.to_string();
// 2. Persistence artifacts: cf_clearance (HttpOnly → must read via CDP, not
// document.cookie) + CF_VERIFIED_DEVICE. Scope to the current URL.
let urls = if url.is_empty() {
None
} else {
Some(vec![url.clone()])
};
let cookies = super::cookies::get_cookies(&mgr.client, &session_id, urls)
.await
.unwrap_or_default();
let clearance = cookies.iter().find(|c| c.name == "cf_clearance");
let now = std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.map(|d| d.as_secs_f64())
.unwrap_or(0.0);
let (present, expired) = clearance_state(clearance.map(|c| c.expires), now);
let expires_in = clearance
.filter(|_| present && !expired)
.map(|c| (c.expires - now).max(0.0) as i64);
let device_verified = cookies
.iter()
.any(|c| c.name.starts_with("CF_VERIFIED_DEVICE"));
let clearance_valid = present && !expired;
let recommendation = cf_recommendation(challenged, clearance_valid);
Ok(json!({
"url": url,
"title": title,
"challenged": challenged,
"turnstile": turnstile,
"clearance": {
"present": present,
"expired": expired,
"expiresIn": expires_in,
"httpOnly": clearance.map(|c| c.http_only).unwrap_or(false),
},
"deviceVerified": device_verified,
"recommendation": recommendation,
}))
}
async fn handle_storage_get(cmd: &Value, state: &DaemonState) -> Result<Value, String> { async fn handle_storage_get(cmd: &Value, state: &DaemonState) -> Result<Value, String> {
let mgr = state.browser.as_ref().ok_or("Browser not launched")?; let mgr = state.browser.as_ref().ok_or("Browser not launched")?;
let session_id = mgr.active_session_id()?.to_string(); let session_id = mgr.active_session_id()?.to_string();
@@ -4454,7 +4673,33 @@ async fn handle_tab_switch(cmd: &Value, state: &mut DaemonState) -> Result<Value
state.ref_map.clear(); state.ref_map.clear();
state.iframe_sessions.clear(); state.iframe_sessions.clear();
state.active_frame_id = None; state.active_frame_id = None;
let result = mgr.tab_switch_by_id(tab_id).await?; let mut result = mgr.tab_switch_by_id(tab_id).await?;
// Liveness probe: confirm the new session actually answers before we report
// success, so `tab <id>` doesn't print a misleading ✓ for a session that's
// stale and will fail on the very next command (issue #29.3). On the churned
// -tabId case the ext-0.4.9 targetId recovery (#24) self-heals within ~6s, so
// we surface a warning rather than a hard error to avoid a false failure
// during that window.
if mgr.evaluate("1", None).await.is_err() {
if let Some(obj) = result.as_object_mut() {
obj.insert(
"warning".to_string(),
json!("switched tab is not responding yet (session re-attaching); retry the next command"),
);
}
}
// `--activate`: raise this tab to the foreground (the switch made it active;
// bring_to_front acts on the active tab) — for handing a specific tab to the
// human (issue #24-C). Best-effort; don't fail the switch if it can't.
if cmd
.get("activate")
.and_then(|v| v.as_bool())
.unwrap_or(false)
{
let _ = mgr.bring_to_front().await;
}
if let Some(ref server) = state.stream_server { if let Some(ref server) = state.stream_server {
if let Ok(dims) = mgr if let Ok(dims) = mgr
@@ -5284,6 +5529,18 @@ async fn handle_bringtofront(state: &DaemonState) -> Result<Value, String> {
Ok(json!({ "broughtToFront": true })) Ok(json!({ "broughtToFront": true }))
} }
async fn handle_current(state: &mut DaemonState) -> Result<Value, String> {
let mgr = state.browser.as_mut().ok_or("Browser not launched")?;
// Refresh so `current` reflects the live URL/title even after a cross-process
// nav (the relay's cached target_info can lag) (#26).
mgr.resync_targets().await.ok();
let mut info = mgr.active_page_info().ok_or("No active tab")?;
if let Some(obj) = info.as_object_mut() {
obj.insert("current".to_string(), json!(true));
}
Ok(info)
}
async fn handle_timezone(cmd: &Value, state: &DaemonState) -> Result<Value, String> { async fn handle_timezone(cmd: &Value, state: &DaemonState) -> Result<Value, String> {
let mgr = state.browser.as_ref().ok_or("Browser not launched")?; let mgr = state.browser.as_ref().ok_or("Browser not launched")?;
let timezone = cmd let timezone = cmd
@@ -8915,6 +9172,26 @@ mod tests {
use crate::test_utils::EnvGuard; use crate::test_utils::EnvGuard;
use std::fs; use std::fs;
#[test]
fn test_cf_recommendation() {
assert_eq!(cf_recommendation(false, false), "proceed");
assert_eq!(cf_recommendation(false, true), "proceed");
assert_eq!(cf_recommendation(true, false), "solve");
assert_eq!(cf_recommendation(true, true), "reissue");
}
#[test]
fn test_clearance_state() {
// no cookie
assert_eq!(clearance_state(None, 1000.0), (false, false));
// session cookie (expires <= 0) → present, never expired
assert_eq!(clearance_state(Some(-1.0), 1000.0), (true, false));
// valid: expiry in the future
assert_eq!(clearance_state(Some(2000.0), 1000.0), (true, false));
// expired: expiry in the past
assert_eq!(clearance_state(Some(500.0), 1000.0), (true, true));
}
#[test] #[test]
fn test_url_glob_to_regex() { fn test_url_glob_to_regex() {
assert_eq!(url_glob_to_regex("**/dashboard"), "^.*/dashboard$"); assert_eq!(url_glob_to_regex("**/dashboard"), "^.*/dashboard$");
+318 -27
View File
@@ -166,6 +166,52 @@ fn resolve_active_index(
active_page_index active_page_index
} }
/// Best-effort MIME type from a filename extension, for the relay file-upload
/// fallback (the page-constructed `File` needs a sensible `type`). Covers the
/// common upload kinds; anything unknown falls back to a generic binary type.
fn mime_for_path(name: &str) -> &'static str {
let ext = name.rsplit('.').next().unwrap_or("").to_lowercase();
match ext.as_str() {
"png" => "image/png",
"jpg" | "jpeg" => "image/jpeg",
"gif" => "image/gif",
"webp" => "image/webp",
"svg" => "image/svg+xml",
"bmp" => "image/bmp",
"pdf" => "application/pdf",
"txt" => "text/plain",
"csv" => "text/csv",
"json" => "application/json",
"mp4" => "video/mp4",
"webm" => "video/webm",
"mov" => "video/quicktime",
"mp3" => "audio/mpeg",
"zip" => "application/zip",
_ => "application/octet-stream",
}
}
/// Target ids to prune after a `Target.getTargets` resync: tracked pages whose
/// target is no longer in the live set — EXCEPT the explicitly-pinned active
/// target, which is protected. The relay against a busy real Chrome occasionally
/// returns a different window's tabs for a single `getTargets` call ("tab list
/// hops windows", issue #31); pruning on that transient snapshot would drop the
/// agent's adopted tab and drift subsequent eval/click onto a foreign tab. A
/// genuine close still arrives as `Target.targetDestroyed` (handled in the event
/// drain), which removes the pin properly — so protecting it here only guards
/// against flaky snapshots, not real closures.
fn prunable_target_ids(
pages: &[PageInfo],
live_ids: &HashSet<String>,
pinned: Option<&str>,
) -> Vec<String> {
pages
.iter()
.map(|p| p.target_id.clone())
.filter(|tid| !live_ids.contains(tid) && pinned != Some(tid.as_str()))
.collect()
}
/// Whether the resolved active page is a tab the session created (its target_id /// Whether the resolved active page is a tab the session created (its target_id
/// is in `created_targets`). Pure core of [`BrowserManager::active_is_session_owned`] /// is in `created_targets`). Pure core of [`BrowserManager::active_is_session_owned`]
/// so the relay no-hijack rule is unit-testable without a live browser. /// so the relay no-hijack rule is unit-testable without a live browser.
@@ -490,6 +536,13 @@ impl BrowserManager {
} }
}; };
// A launched browser carries a debug port → it's the other path that can
// pop Chrome's consent modal; record it for #31 diagnosis.
crate::connect::log_connect_mode(
&ws_url,
true,
DAEMON_SESSION.get().map(String::as_str).unwrap_or("default"),
);
let manager = if engine == "lightpanda" { let manager = if engine == "lightpanda" {
initialize_lightpanda_manager(ws_url, process).await? initialize_lightpanda_manager(ws_url, process).await?
} else { } else {
@@ -585,6 +638,13 @@ impl BrowserManager {
headers: Option<Vec<(String, String)>>, headers: Option<Vec<(String, String)>>,
) -> Result<Self, String> { ) -> Result<Self, String> {
let ws_url = resolve_cdp_url(url).await?; let ws_url = resolve_cdp_url(url).await?;
// Record the transport so a reappearing "Allow remote debugging?" modal
// can be traced to a raw-port attach vs the consent-free relay (#31).
crate::connect::log_connect_mode(
&ws_url,
false,
DAEMON_SESSION.get().map(String::as_str).unwrap_or("default"),
);
let client = Arc::new(CdpClient::connect_with_headers(&ws_url, headers).await?); let client = Arc::new(CdpClient::connect_with_headers(&ws_url, headers).await?);
let mut manager = Self { let mut manager = Self {
client, client,
@@ -1263,6 +1323,22 @@ impl BrowserManager {
.collect() .collect()
} }
/// The active tab's stable handle + current location, for `chrome-use
/// current` (#26). `targetId` survives cross-process navigation, so it's the
/// handle an agent should hold across a multi-step flow.
pub fn active_page_info(&self) -> Option<Value> {
let i = self.resolved_active_index();
self.pages.get(i).map(|p| {
json!({
"tabId": format_tab_id(p.tab_id),
"targetId": p.target_id,
"label": p.label,
"url": p.url,
"title": p.title,
})
})
}
/// Stable `tab_id` for a page identified by its CDP `targetId`, if tracked. /// Stable `tab_id` for a page identified by its CDP `targetId`, if tracked.
/// Lets callers adopt a tab by the cross-session-stable target id. /// Lets callers adopt a tab by the cross-session-stable target id.
pub fn tab_id_for_target(&self, target_id: &str) -> Option<u32> { pub fn tab_id_for_target(&self, target_id: &str) -> Option<u32> {
@@ -1279,6 +1355,67 @@ impl BrowserManager {
/// the active tab is preserved, and re-pinned if it was pruned. Powers a live /// the active tab is preserved, and re-pinned if it was pruned. Powers a live
/// `tab list` and adopt-by-targetId so a fresh session can reach a stranded, /// `tab list` and adopt-by-targetId so a fresh session can reach a stranded,
/// still-filled tab without reloading it (issue #21). /// still-filled tab without reloading it (issue #21).
/// Detect targets that appeared since the `before` set (e.g. a click that
/// opened a new tab via a `target=_blank` link or `window.open`), attach +
/// track each in the background, and return the first newly-opened page.
///
/// Lighter than [`resync_targets`] — one `getTargets` and work only on the
/// new targets, no whole-tab url/title refresh — so it's cheap enough to run
/// after every click. The new tab is added in the background (never steals
/// the active tab, per #7/#8.1); the caller surfaces it so the agent knows a
/// tab opened instead of seeing the old page (issue #24-A).
pub async fn adopt_newly_opened(&mut self, before: &HashSet<String>) -> Option<PageInfo> {
let result: GetTargetsResult = self
.client
.send_command_typed("Target.getTargets", &json!({}), None)
.await
.ok()?;
let live: Vec<TargetInfo> = result
.target_infos
.into_iter()
.filter(should_track_target)
.collect();
let mut opened: Option<PageInfo> = None;
for target in &live {
if before.contains(&target.target_id)
|| self.pages.iter().any(|p| p.target_id == target.target_id)
{
continue;
}
let attach: AttachToTargetResult = match self
.client
.send_command_typed(
"Target.attachToTarget",
&AttachToTargetParams {
target_id: target.target_id.clone(),
flatten: true,
},
None,
)
.await
{
Ok(r) => r,
Err(_) => continue,
};
let tab_id = self.assign_tab_id();
let page = PageInfo {
tab_id,
label: None,
target_id: target.target_id.clone(),
session_id: attach.session_id.clone(),
url: target.url.clone(),
title: target.title.clone(),
target_type: target.target_type.clone(),
};
self.add_background_page(page.clone());
let _ = self.enable_domains(&attach.session_id).await;
if opened.is_none() {
opened = Some(page);
}
}
opened
}
pub async fn resync_targets(&mut self) -> Result<(), String> { pub async fn resync_targets(&mut self) -> Result<(), String> {
self.client self.client
.send_command_typed::<_, Value>( .send_command_typed::<_, Value>(
@@ -1334,13 +1471,10 @@ impl BrowserManager {
let _ = self.enable_domains(&attach_result.session_id).await; let _ = self.enable_domains(&attach_result.session_id).await;
} }
// Drop tabs that no longer exist so `tab list` doesn't show phantom rows. // Drop tabs that no longer exist so `tab list` doesn't show phantom rows
let gone: Vec<String> = self // but never prune the explicitly-pinned active target on a transient
.pages // getTargets snapshot (issue #31; see `prunable_target_ids`).
.iter() let gone = prunable_target_ids(&self.pages, &live_ids, self.active_target_id.as_deref());
.map(|p| p.target_id.clone())
.filter(|tid| !live_ids.contains(tid))
.collect();
for tid in gone { for tid in gone {
self.remove_page_by_target_id(&tid); self.remove_page_by_target_id(&tid);
} }
@@ -1817,7 +1951,8 @@ impl BrowserManager {
.and_then(|v| v.as_i64()) .and_then(|v| v.as_i64())
.ok_or("Could not get backendNodeId for file input")?; .ok_or("Could not get backendNodeId for file input")?;
self.client let set_files = self
.client
.send_command( .send_command(
"DOM.setFileInputFiles", "DOM.setFileInputFiles",
Some(json!({ Some(json!({
@@ -1826,26 +1961,153 @@ impl BrowserManager {
})), })),
Some(&effective_session_id), Some(&effective_session_id),
) )
.await .await;
.map_err(|e| {
// Chrome's chrome.debugger API (the extension-relay transport)
// forbids DOM.setFileInputFiles for security, surfacing as an
// opaque `-32000 "Not allowed"`. Translate it into an actionable
// message rather than leaking the raw CDP error (issue #13).
if e.contains("Not allowed") || e.contains("-32000") {
"file upload isn't supported over the extension relay — \
Chrome's chrome.debugger API forbids DOM.setFileInputFiles. \
Use a direct-CDP session instead: \
`chrome-use --session up --launch open <url>` (carry your \
login over with `cookies export` | `cookies set --curl`), \
then run `upload` in that session. \
See https://github.com/leeguooooo/chrome-use/issues/13"
.to_string()
} else {
e
}
})?;
if let Err(e) = set_files {
// Chrome's chrome.debugger API (the extension-relay transport) forbids
// DOM.setFileInputFiles for security, surfacing as an opaque
// `-32000 "Not allowed"`. Fall back to constructing the File entirely
// IN THE PAGE and assigning it to the input — the standard
// Playwright/Cypress trick, which needs no privileged CDP and so works
// over the relay (issue #13).
if e.contains("Not allowed") || e.contains("-32000") {
return self
.upload_files_via_page(object_id, files, &effective_session_id)
.await;
}
return Err(e);
}
Ok(())
}
/// Relay-safe file upload: read each file locally, hand its bytes to the page
/// as base64, and rebuild a `File` there — then either assign it to a file
/// `<input>` (Chrome allows `input.files = dataTransfer.files`) or, for a
/// dropzone/composer, dispatch synthetic `paste`/`drop` events carrying the
/// `DataTransfer`. No `DOM.setFileInputFiles`, so chrome.debugger permits it.
async fn upload_files_via_page(
&self,
object_id: String,
files: &[String],
session_id: &str,
) -> Result<(), String> {
use base64::Engine;
// The relay tunnels every CDP message through Chrome native messaging,
// which caps a single message at ~1 MiB. A whole image's base64 blows
// past that ("CDP response channel closed"), so we STREAM the bytes into
// a page-side buffer in sub-limit chunks, then assemble the File from it.
const CHUNK: usize = 96 * 1024; // base64 chars per message; safe under 1 MiB
// Reset the staging buffer.
self.client
.send_command(
"Runtime.evaluate",
Some(json!({ "expression": "window.__cuUpload = [];", "returnByValue": true })),
Some(session_id),
)
.await
.map_err(|e| format!("relay upload (reset) failed: {}", e))?;
for path in files {
let bytes = std::fs::read(path).map_err(|e| format!("cannot read {}: {}", path, e))?;
let name = std::path::Path::new(path)
.file_name()
.and_then(|n| n.to_str())
.unwrap_or("upload.bin")
.to_string();
let mime = mime_for_path(&name);
let b64 = base64::engine::general_purpose::STANDARD.encode(&bytes);
// Push the file's metadata with an empty buffer.
let init = format!(
"window.__cuUpload.push({{ name: {}, type: {}, b64: '' }});",
serde_json::to_string(&name).unwrap_or_default(),
serde_json::to_string(mime).unwrap_or_default(),
);
self.client
.send_command(
"Runtime.evaluate",
Some(json!({ "expression": init, "returnByValue": true })),
Some(session_id),
)
.await
.map_err(|e| format!("relay upload (init) failed: {}", e))?;
// Stream the base64 in chunks. base64's alphabet (AZaz09+/=) needs
// no escaping inside a single-quoted JS string, so concatenation is safe.
let idx = "window.__cuUpload[window.__cuUpload.length-1].b64";
let mut start = 0;
while start < b64.len() {
let end = (start + CHUNK).min(b64.len());
let chunk = &b64[start..end];
let expr = format!("{idx} += '{chunk}';");
self.client
.send_command(
"Runtime.evaluate",
Some(json!({ "expression": expr, "returnByValue": true })),
Some(session_id),
)
.await
.map_err(|e| format!("relay upload (chunk) failed: {}", e))?;
start = end;
}
}
// Assemble the Files from the buffer and attach to the element, then clean up.
let func = r#"function() {
const filesData = window.__cuUpload || [];
const dt = new DataTransfer();
for (const f of filesData) {
const bin = atob(f.b64);
const arr = new Uint8Array(bin.length);
for (let i = 0; i < bin.length; i++) arr[i] = bin.charCodeAt(i);
dt.items.add(new File([arr], f.name, { type: f.type }));
}
try { delete window.__cuUpload; } catch (e) { window.__cuUpload = undefined; }
const el = this;
if (el.tagName === 'INPUT' && el.type === 'file') {
el.files = dt.files;
el.dispatchEvent(new Event('input', { bubbles: true }));
el.dispatchEvent(new Event('change', { bubbles: true }));
return 'input:' + dt.files.length;
}
// Dropzone / rich composer: replay paste then drop with the files.
try { el.dispatchEvent(new ClipboardEvent('paste', { bubbles: true, clipboardData: dt })); } catch (e) {}
try {
const ev = new DragEvent('drop', { bubbles: true, cancelable: true });
Object.defineProperty(ev, 'dataTransfer', { value: dt });
el.dispatchEvent(ev);
} catch (e) {}
return 'event:' + dt.files.length;
}"#;
let result: EvaluateResult = self
.client
.send_command_typed(
"Runtime.callFunctionOn",
&CallFunctionOnParams {
function_declaration: func.to_string(),
object_id: Some(object_id),
arguments: None,
return_by_value: Some(true),
await_promise: Some(false),
},
Some(session_id),
)
.await
.map_err(|e| format!("relay file-injection failed: {}", e))?;
if let Some(ref details) = result.exception_details {
return Err(format!(
"relay file-injection threw: {}",
details
.exception
.as_ref()
.and_then(|ex| ex.description.as_deref())
.unwrap_or(&details.text)
));
}
Ok(()) Ok(())
} }
@@ -2505,6 +2767,35 @@ mod tests {
assert!(!active_index_is_owned(&[], None, 0, &created)); assert!(!active_index_is_owned(&[], None, 0, &created));
} }
#[test]
fn test_mime_for_path() {
assert_eq!(mime_for_path("a.png"), "image/png");
assert_eq!(mime_for_path("PHOTO.JPG"), "image/jpeg");
assert_eq!(mime_for_path("clip.webp"), "image/webp");
assert_eq!(mime_for_path("doc.pdf"), "application/pdf");
assert_eq!(mime_for_path("noext"), "application/octet-stream");
assert_eq!(mime_for_path("weird.xyz"), "application/octet-stream");
}
#[test]
fn prune_protects_pinned_target_on_transient_snapshot() {
// The relay returned a getTargets snapshot missing the pinned tab "A"
// (it hopped to another window). "B" is also absent. Without protection
// both would be pruned and the next command would drift; with the pin
// protected, only the genuinely-unpinned "B" is dropped (issue #31).
let pages = vec![page("A"), page("B")];
let live: HashSet<String> = HashSet::new(); // snapshot returned neither
let gone = prunable_target_ids(&pages, &live, Some("A"));
assert_eq!(gone, vec!["B".to_string()]);
// With no pin, both are prunable (unchanged behavior).
let gone_unpinned = prunable_target_ids(&pages, &live, None);
assert_eq!(gone_unpinned.len(), 2);
// A pinned target that IS in the live set is simply not prunable anyway.
let mut live2 = HashSet::new();
live2.insert("A".to_string());
assert_eq!(prunable_target_ids(&pages, &live2, Some("A")), vec!["B".to_string()]);
}
#[test] #[test]
fn resolve_active_index_pin_survives_passive_background_tab() { fn resolve_active_index_pin_survives_passive_background_tab() {
// A foreign tab ("Z") gets appended by passive discovery after we pinned // A foreign tab ("Z") gets appended by passive discovery after we pinned
+337 -8
View File
@@ -796,16 +796,43 @@ pub(super) fn extract_ax_string(value: &Option<AXValue>) -> String {
/// Build a JS expression that finds a DOM element by CSS selector or XPath. /// Build a JS expression that finds a DOM element by CSS selector or XPath.
fn build_find_element_js(selector: &str) -> String { fn build_find_element_js(selector: &str) -> String {
if let Some(xpath) = selector.strip_prefix("xpath=") { if let Some(xpath) = selector.strip_prefix("xpath=") {
format!( return format!(
"document.evaluate({}, document, null, XPathResult.FIRST_ORDERED_NODE_TYPE, null).singleNodeValue", "document.evaluate({}, document, null, XPathResult.FIRST_ORDERED_NODE_TYPE, null).singleNodeValue",
serde_json::to_string(xpath).unwrap_or_default() serde_json::to_string(xpath).unwrap_or_default()
) );
} else {
format!(
"document.querySelector({})",
serde_json::to_string(selector).unwrap_or_default()
)
} }
// Bare string (or explicit `text=`): try CSS first, then fall back to
// matching an interactive element by its VISIBLE TEXT. snapshot exposes
// buttons/links by their name, so `click "購入手続きへ"` should resolve by
// that label — previously it was fed straight to `querySelector` as CSS and
// failed as an invalid selector even though the button was right there
// (issue #24-B). CSS still wins when it matches, so existing selectors are
// unaffected; nested/non-ASCII labels now resolve too.
let text_only = selector.strip_prefix("text=");
let force_text = text_only.is_some();
let sel_json = serde_json::to_string(selector).unwrap_or_default();
let want_json = serde_json::to_string(text_only.unwrap_or(selector)).unwrap_or_default();
format!(
r#"(() => {{
const sel = {sel};
const css = {force_text} ? null : (() => {{ try {{ return document.querySelector(sel); }} catch (_e) {{ return null; }} }})();
if (css) return css;
const norm = s => (s == null ? '' : String(s)).replace(/\s+/g, ' ').trim();
const w = norm({want}); if (!w) return null;
const wl = w.toLowerCase();
const interactive = Array.from(document.querySelectorAll(
'button,a,[role=button],[role=link],[role=menuitem],[role=tab],[role=option],input[type=submit],input[type=button],input[type=reset],summary,label,[onclick]'));
const textOf = e => norm(e.innerText || e.textContent) || norm(e.value) ||
norm(e.getAttribute && e.getAttribute('aria-label')) || norm(e.getAttribute && e.getAttribute('title'));
let hit = interactive.find(e => textOf(e) === w) || interactive.find(e => textOf(e).toLowerCase().includes(wl));
if (hit) return hit;
const leaves = Array.from(document.querySelectorAll('*')).filter(e => !e.children.length);
return leaves.find(e => norm(e.textContent) === w) || leaves.find(e => norm(e.textContent).toLowerCase().includes(wl)) || null;
}})()"#,
sel = sel_json,
want = want_json,
force_text = force_text
)
} }
/// Build a JS expression that counts matching DOM elements by CSS selector or XPath. /// Build a JS expression that counts matching DOM elements by CSS selector or XPath.
@@ -948,6 +975,263 @@ pub async fn get_element_text(
.unwrap_or_default()) .unwrap_or_default())
} }
/// Text content collected from a single frame of the page.
#[derive(Debug, Clone)]
pub struct FrameText {
pub frame_id: String,
pub url: String,
/// "top" | "inline" (same-process child frame) | "oopif" (out-of-process).
pub kind: &'static str,
pub text: String,
}
// The expression we run in every frame to read its visible text. innerText
// honors CSS visibility (skips display:none), textContent is the fallback.
const FRAME_INNERTEXT_JS: &str = "(function(){try{var b=document.body||document.documentElement;return b?(b.innerText||b.textContent||''):'';}catch(e){return '';}})()";
async fn eval_text_default(client: &CdpClient, session_id: &str) -> String {
let res = client
.send_command(
"Runtime.evaluate",
Some(serde_json::json!({
"expression": FRAME_INNERTEXT_JS,
"returnByValue": true,
})),
Some(session_id),
)
.await;
res.ok()
.and_then(|v| v.get("result").and_then(|r| r.get("value")).cloned())
.and_then(|v| v.as_str().map(|s| s.to_string()))
.unwrap_or_default()
}
// Same-process child frames share the top renderer but live in their own
// execution context. Page.createIsolatedWorld hands us a context id bound to
// that frame so Runtime.evaluate reads the child document, not the parent.
async fn eval_text_in_frame(client: &CdpClient, session_id: &str, frame_id: &str) -> String {
let ctx = client
.send_command(
"Page.createIsolatedWorld",
Some(serde_json::json!({ "frameId": frame_id, "worldName": "chrome_use_text" })),
Some(session_id),
)
.await
.ok()
.and_then(|v| v.get("executionContextId").and_then(|c| c.as_i64()));
let Some(ctx_id) = ctx else { return String::new() };
let res = client
.send_command(
"Runtime.evaluate",
Some(serde_json::json!({
"expression": FRAME_INNERTEXT_JS,
"returnByValue": true,
"contextId": ctx_id,
})),
Some(session_id),
)
.await;
res.ok()
.and_then(|v| v.get("result").and_then(|r| r.get("value")).cloned())
.and_then(|v| v.as_str().map(|s| s.to_string()))
.unwrap_or_default()
}
fn flatten_frame_tree(node: &Value, is_top: bool, out: &mut Vec<(String, String, bool)>) {
if let Some(frame) = node.get("frame") {
if let Some(id) = frame.get("id").and_then(|v| v.as_str()) {
let url = frame
.get("url")
.and_then(|v| v.as_str())
.unwrap_or("")
.to_string();
out.push((id.to_string(), url, is_top));
}
}
if let Some(children) = node.get("childFrames").and_then(|v| v.as_array()) {
for child in children {
flatten_frame_tree(child, false, out);
}
}
}
/// Collect visible text from every frame reachable in the active session,
/// including out-of-process iframes (which never appear in the top frame's
/// `Page.getFrameTree` and so are invisible to `document.body.innerText`).
///
/// Same-process child frames are read through `Page.createIsolatedWorld`;
/// OOPIFs are read through their own auto-attached debugger session
/// (`iframe_sessions`, keyed by frameId == targetId). This is the engine
/// behind `get text --all-frames` and `chrome-use frames` — the fix for
/// listing/marketplace pages whose description lives in a child frame (#27).
pub async fn collect_all_frames_text(
client: &CdpClient,
top_session: &str,
iframe_sessions: &HashMap<String, String>,
) -> Result<Vec<FrameText>, String> {
let mut out: Vec<FrameText> = Vec::new();
let mut seen: std::collections::HashSet<String> = std::collections::HashSet::new();
// 1. Top session: the top frame plus its same-process descendants. OOPIF
// frames that happen to surface here are skipped — they're read via
// their dedicated session in step 2 (cross-process isolated worlds fail).
let tree = client
.send_command_no_params("Page.getFrameTree", Some(top_session))
.await?;
let mut frames: Vec<(String, String, bool)> = Vec::new();
flatten_frame_tree(&tree["frameTree"], true, &mut frames);
for (fid, url, is_top) in frames {
if iframe_sessions.contains_key(&fid) {
continue;
}
if !seen.insert(fid.clone()) {
continue;
}
let (kind, text) = if is_top {
("top", eval_text_default(client, top_session).await)
} else {
("inline", eval_text_in_frame(client, top_session, &fid).await)
};
out.push(FrameText {
frame_id: fid,
url,
kind,
text,
});
}
// 2. Each out-of-process iframe, read through its own session.
for (fid, sid) in iframe_sessions {
if !seen.insert(fid.clone()) {
continue;
}
let url = client
.send_command_no_params("Page.getFrameTree", Some(sid))
.await
.ok()
.and_then(|t| {
t.get("frameTree")
.and_then(|ft| ft.get("frame"))
.and_then(|f| f.get("url"))
.and_then(|v| v.as_str())
.map(|s| s.to_string())
})
.unwrap_or_default();
let text = eval_text_default(client, sid).await;
out.push(FrameText {
frame_id: fid.clone(),
url,
kind: "oopif",
text,
});
}
Ok(out)
}
// Readability-lite: prefer the page's semantic main-content region over the
// whole body so global header/nav/footer chrome (and, on many listing pages,
// the "related items" sidebar) doesn't drown out the actual content. Runs on
// the live, rendered tree (innerText needs layout — a detached clone returns
// empty), so we pick the densest <main>/<article> region rather than cloning
// and stripping. Falls back to <body> when no substantial main region exists.
const MAIN_CONTENT_JS: &str = r#"(function(){
function txt(el){try{return (el.innerText||'').trim();}catch(e){return '';}}
var sels=['main','[role=main]','article','#main','#contents','#l-content'];
var best=null,bestLen=0;
for(var i=0;i<sels.length;i++){
var els=document.querySelectorAll(sels[i]);
for(var j=0;j<els.length;j++){var l=txt(els[j]).length;if(l>bestLen){bestLen=l;best=els[j];}}
}
if(best&&bestLen>200)return txt(best);
return txt(document.body);
})()"#;
/// Extract the page's main-content text (readability-lite), preferring a
/// semantic `<main>`/`<article>` region over the full body. Used by
/// `get text --main` to avoid header/nav/sidebar boilerplate (#27).
pub async fn get_main_content_text(client: &CdpClient, session_id: &str) -> Result<String, String> {
let res = client
.send_command(
"Runtime.evaluate",
Some(serde_json::json!({
"expression": MAIN_CONTENT_JS,
"returnByValue": true,
})),
Some(session_id),
)
.await?;
Ok(res
.get("result")
.and_then(|r| r.get("value"))
.and_then(|v| v.as_str())
.unwrap_or_default()
.to_string())
}
// Text nodes whose parent is one of these carry no visible content.
fn is_noise_tag(name: &str) -> bool {
matches!(name, "SCRIPT" | "STYLE" | "NOSCRIPT" | "TEMPLATE" | "HEAD")
}
// Walk a CDP DOM.Node tree, collecting text-node values. Unlike `innerText`
// (JS, blocked by CLOSED shadow roots), the CDP DOM tree from
// `DOM.getDocument(pierce:true)` includes closed shadow roots and child
// documents — so this reaches text JS can't. `parent_noise` carries whether an
// ancestor was <script>/<style>/etc so their text is skipped.
fn collect_dom_text(node: &Value, parent_noise: bool, out: &mut String) {
let node_type = node.get("nodeType").and_then(|v| v.as_i64()).unwrap_or(0);
let node_name = node.get("nodeName").and_then(|v| v.as_str()).unwrap_or("");
if node_type == 3 {
if !parent_noise {
if let Some(t) = node.get("nodeValue").and_then(|v| v.as_str()) {
let t = t.trim();
if !t.is_empty() {
if !out.is_empty() {
out.push(' ');
}
out.push_str(t);
}
}
}
return;
}
let noise = parent_noise || is_noise_tag(node_name);
if let Some(children) = node.get("children").and_then(|v| v.as_array()) {
for child in children {
collect_dom_text(child, noise, out);
}
}
if let Some(shadow) = node.get("shadowRoots").and_then(|v| v.as_array()) {
for sr in shadow {
collect_dom_text(sr, noise, out);
}
}
if let Some(doc) = node.get("contentDocument") {
collect_dom_text(doc, noise, out);
}
}
/// Extract text from the page via the CDP DOM tree with `pierce:true`, which
/// reaches into CLOSED shadow roots and child documents that `innerText`/`eval`
/// cannot. Lets an agent read content rendered into a closed shadow DOM (e.g. an
/// extension's injected debug panel) without any extra Chrome permission — it
/// rides the per-tab debugger session that's already attached (#30).
pub async fn get_pierced_text(client: &CdpClient, session_id: &str) -> Result<String, String> {
let doc = client
.send_command(
"DOM.getDocument",
Some(serde_json::json!({ "depth": -1, "pierce": true })),
Some(session_id),
)
.await?;
let mut out = String::new();
if let Some(root) = doc.get("root") {
collect_dom_text(root, false, &mut out);
}
Ok(out)
}
pub async fn get_element_attribute( pub async fn get_element_attribute(
client: &CdpClient, client: &CdpClient,
session_id: &str, session_id: &str,
@@ -1421,6 +1705,31 @@ mod tests {
assert_eq!(parse_ref("@e123"), Some("e123".to_string())); assert_eq!(parse_ref("@e123"), Some("e123".to_string()));
} }
#[test]
fn test_collect_dom_text_pierces_closed_shadow_and_skips_noise() {
// A CDP DOM.Node tree: a host element whose CLOSED shadow root holds the
// text, plus a <script> whose text must be skipped.
let tree = serde_json::json!({
"nodeType": 1, "nodeName": "BODY",
"children": [
{ "nodeType": 1, "nodeName": "SCRIPT",
"children": [ { "nodeType": 3, "nodeName": "#text", "nodeValue": "var secret=1;" } ] },
{ "nodeType": 1, "nodeName": "DIV",
"shadowRoots": [
{ "nodeType": 11, "nodeName": "#document-fragment",
"children": [
{ "nodeType": 1, "nodeName": "SPAN",
"children": [ { "nodeType": 3, "nodeName": "#text", "nodeValue": "DECRYPTED 42" } ] }
] }
] }
]
});
let mut out = String::new();
collect_dom_text(&tree, false, &mut out);
assert_eq!(out, "DECRYPTED 42");
assert!(!out.contains("secret"), "script text must be skipped");
}
#[test] #[test]
fn test_parse_ref_equals_prefix() { fn test_parse_ref_equals_prefix() {
assert_eq!(parse_ref("ref=e1"), Some("e1".to_string())); assert_eq!(parse_ref("ref=e1"), Some("e1".to_string()));
@@ -1452,10 +1761,30 @@ mod tests {
#[test] #[test]
fn test_build_selector_js_css() { fn test_build_selector_js_css() {
let js = build_selector_js("#submit-btn"); let js = build_selector_js("#submit-btn");
assert!(js.contains("document.querySelector(\"#submit-btn\")")); // CSS is now tried via a `sel` variable, with a visible-text fallback
// appended (issue #24-B). It must still use querySelector (not xpath).
assert!(js.contains("const sel = \"#submit-btn\""));
assert!(js.contains("document.querySelector(sel)"));
assert!(!js.contains("document.evaluate")); assert!(!js.contains("document.evaluate"));
} }
#[test]
fn test_build_find_element_js_text_fallback() {
// A bare label gets a text-matching fallback so `click "購入手続きへ"`
// resolves by visible text, not just CSS (issue #24-B).
let js = build_find_element_js("購入手続きへ");
assert!(js.contains("購入手続きへ"));
assert!(js.contains("interactive")); // the text-match branch
assert!(js.contains("textOf"));
// `text=` forces the text path (skips CSS).
let forced = build_find_element_js("text=Buy now");
assert!(forced.contains("true ? null")); // force_text => css skipped
// xpath is unchanged.
let xp = build_find_element_js("xpath=//button");
assert!(xp.contains("document.evaluate"));
assert!(!xp.contains("interactive"));
}
#[test] #[test]
fn test_build_selector_js_xpath() { fn test_build_selector_js_xpath() {
let js = build_selector_js("xpath=//button[@id='ok']"); let js = build_selector_js("xpath=//button[@id='ok']");
+39 -32
View File
@@ -306,32 +306,50 @@ pub async fn fill(
) )
.await?; .await?;
// Focus the element // Emulate a real edit so framework-controlled inputs (React/Vue) and
client // site-side listeners actually see the change (issue #25): the old path set
.send_command_typed::<_, Value>( // `this.value` directly and used Input.insertText, which left React's
"Runtime.callFunctionOn", // internal value-tracker out of sync and never fired change/blur — so
&CallFunctionOnParams { // dependent logic (e.g. Mercari's postal-code → 都道府県 autocomplete) never
function_declaration: "function() { this.focus(); }".to_string(), // ran even though the value was visible. Set the value through the element's
object_id: Some(object_id.clone()), // PROTOTYPE setter (which React's _valueTracker hooks), then dispatch
arguments: None, // input → change → blur/focusout. `type <sel> <text>` remains for sites that
return_by_value: Some(true), // need per-keystroke events.
await_promise: Some(false), let fill_js = format!(
}, r#"function() {{
Some(&effective_session_id), const el = this;
) const v = {val};
.await?; try {{ el.focus(); }} catch (e) {{}}
const tag = el.tagName;
const fire = (type, ctor) => el.dispatchEvent(new (ctor || Event)(type, {{ bubbles: true }}));
if (tag === 'SELECT') {{
el.value = v; fire('input'); fire('change'); return true;
}}
if (el.isContentEditable) {{
el.textContent = v; fire('input', window.InputEvent || Event); fire('change');
try {{ el.blur(); }} catch (e) {{}} fire('focusout'); return true;
}}
const proto = tag === 'TEXTAREA' ? window.HTMLTextAreaElement.prototype
: window.HTMLInputElement.prototype;
const desc = Object.getOwnPropertyDescriptor(proto, 'value');
const set = desc && desc.set ? (x) => desc.set.call(el, x) : (x) => {{ el.value = x; }};
set(''); // reset the framework tracker
fire('input', window.InputEvent || Event);
set(v); // native setter → React/Vue registers
fire('input', window.InputEvent || Event);
fire('change');
try {{ el.blur(); }} catch (e) {{}}
fire('focusout'); // blur-triggered lookups/validation
return true;
}}"#,
val = serde_json::to_string(value).unwrap_or_default()
);
// Select all + delete to clear
client client
.send_command_typed::<_, Value>( .send_command_typed::<_, Value>(
"Runtime.callFunctionOn", "Runtime.callFunctionOn",
&CallFunctionOnParams { &CallFunctionOnParams {
function_declaration: r#"function() { function_declaration: fill_js,
this.select && this.select();
this.value = '';
this.dispatchEvent(new Event('input', { bubbles: true }));
}"#
.to_string(),
object_id: Some(object_id), object_id: Some(object_id),
arguments: None, arguments: None,
return_by_value: Some(true), return_by_value: Some(true),
@@ -341,17 +359,6 @@ pub async fn fill(
) )
.await?; .await?;
// Insert text (keyboard input dispatched at page level, use parent session_id)
client
.send_command_typed::<_, Value>(
"Input.insertText",
&InsertTextParams {
text: value.to_string(),
},
Some(session_id),
)
.await?;
Ok(()) Ok(())
} }
+144 -15
View File
@@ -186,6 +186,78 @@ pub fn print_response_with_opts(resp: &Response, action: Option<&str>, opts: &Ou
} }
if let Some(data) = &resp.data { if let Some(data) = &resp.data {
// A click that opened a new tab: surface it so the agent doesn't read the
// unchanged old page as a failed click (issue #24-A).
if let Some(opened) = data.get("openedTab") {
let tid = opened.get("tabId").and_then(|v| v.as_str()).unwrap_or("?");
let url = opened.get("url").and_then(|v| v.as_str()).unwrap_or("");
let followed = data
.get("followed")
.and_then(|v| v.as_bool())
.unwrap_or(false);
let verb = if followed {
"switched to new tab"
} else {
"opened new tab"
};
eprintln!(
"{} {} [{}] {}",
color::cyan(""),
verb,
tid,
color::dim(url)
);
}
// `current`: the active tab's stable handle (#26).
if data
.get("current")
.and_then(|v| v.as_bool())
.unwrap_or(false)
{
let tid = data.get("tabId").and_then(|v| v.as_str()).unwrap_or("?");
let title = data.get("title").and_then(|v| v.as_str()).unwrap_or("");
let url = data.get("url").and_then(|v| v.as_str()).unwrap_or("");
let target = data.get("targetId").and_then(|v| v.as_str()).unwrap_or("");
println!("{} [{}] {} - {}", color::cyan(""), tid, title, url);
println!(" {}", color::dim(&format!("target: {}", target)));
return;
}
// Cloudflare challenge/clearance preflight (`cf-status`). Checked early
// because its response carries `url`/`title`, which later generic
// renderers would otherwise swallow.
if action == Some("cf_status") {
let challenged = data.get("challenged").and_then(|v| v.as_bool()).unwrap_or(false);
let rec = data.get("recommendation").and_then(|v| v.as_str()).unwrap_or("?");
let cl = data.get("clearance");
let present = cl.and_then(|c| c.get("present")).and_then(|v| v.as_bool()).unwrap_or(false);
let expired = cl.and_then(|c| c.get("expired")).and_then(|v| v.as_bool()).unwrap_or(false);
let expires_in = cl.and_then(|c| c.get("expiresIn")).and_then(|v| v.as_i64());
let device = data.get("deviceVerified").and_then(|v| v.as_bool()).unwrap_or(false);
let (icon, headline) = match rec {
"proceed" => (color::success_indicator().to_string(), "cleared — no challenge, proceed"),
"solve" => (color::warning_indicator().to_string(), "Cloudflare challenge active, no valid clearance — solve it"),
"reissue" => (color::warning_indicator().to_string(), "challenge active but a clearance cookie exists — stale (IP/UA changed?), re-solve"),
_ => (color::cyan("").to_string(), "unknown"),
};
println!("{} {}", icon, headline);
println!(" challenged: {}", if challenged { "yes" } else { "no" });
let cl_desc = if !present {
"absent".to_string()
} else if expired {
"present but EXPIRED".to_string()
} else if let Some(s) = expires_in {
format!("valid, expires in {}m {}s", s / 60, s % 60)
} else {
"present (session)".to_string()
};
println!(" cf_clearance: {}", cl_desc);
println!(" device trusted: {}", if device { "yes (CF_VERIFIED_DEVICE)" } else { "no" });
return;
}
// Dialog status response // Dialog status response
if action == Some("dialog") { if action == Some("dialog") {
if let Some(has_dialog) = data.get("hasDialog").and_then(|v| v.as_bool()) { if let Some(has_dialog) = data.get("hasDialog").and_then(|v| v.as_bool()) {
@@ -304,6 +376,34 @@ pub fn print_response_with_opts(resp: &Response, action: Option<&str>, opts: &Ou
} }
return; return;
} }
// Frame list (`chrome-use frames`)
if action == Some("frames") {
if let Some(list) = data.get("frames").and_then(|v| v.as_array()) {
let count = list.len();
println!(
"{}",
color::bold(&format!("{} frame{}", count, if count == 1 { "" } else { "s" }))
);
for f in list {
let idx = f.get("index").and_then(|v| v.as_i64()).unwrap_or(0);
let kind = f.get("kind").and_then(|v| v.as_str()).unwrap_or("?");
let url = f.get("url").and_then(|v| v.as_str()).unwrap_or("");
let len = f.get("textLen").and_then(|v| v.as_i64()).unwrap_or(0);
println!(
" [{}] {:<6} {} chars {}",
idx,
kind,
len,
color::dim(if url.is_empty() { "(about:blank)" } else { url })
);
}
eprintln!(
"{}",
color::dim("read everything with: chrome-use get text --all-frames")
);
}
return;
}
// Title // Title
if let Some(title) = data.get("title").and_then(|v| v.as_str()) { if let Some(title) = data.get("title").and_then(|v| v.as_str()) {
println!("{}", title); println!("{}", title);
@@ -531,19 +631,21 @@ pub fn print_response_with_opts(resp: &Response, action: Option<&str>, opts: &Ou
// Tab switch // Tab switch
if action == Some("tab_switch") { if action == Some("tab_switch") {
if let Some(tab_id) = data.get("tabId").and_then(|v| v.as_str()) { if let Some(tab_id) = data.get("tabId").and_then(|v| v.as_str()) {
if let Some(url) = data.get("url").and_then(|v| v.as_str()) { let warning = data.get("warning").and_then(|v| v.as_str());
println!( // A non-responding session isn't a real success — show a warning
"{} Switched to tab [{}] ({})", // indicator instead of the green ✓ (issue #29.3).
color::success_indicator(), let indicator = if warning.is_some() {
tab_id, color::warning_indicator()
url
);
} else { } else {
println!( color::success_indicator()
"{} Switched to tab [{}]", };
color::success_indicator(), if let Some(url) = data.get("url").and_then(|v| v.as_str()) {
tab_id println!("{} Switched to tab [{}] ({})", indicator, tab_id, url);
); } else {
println!("{} Switched to tab [{}]", indicator, tab_id);
}
if let Some(w) = warning {
eprintln!("{}", color::dim(w));
} }
return; return;
} }
@@ -1891,7 +1993,9 @@ Usage: chrome-use get <subcommand> [args]
Retrieves various types of information from elements or the page. Retrieves various types of information from elements or the page.
Subcommands: Subcommands:
text <selector> Get text content of element text [selector] Element text; no selector = WHOLE PAGE, all frames
text --main Main-content text only (skip nav/header/sidebar)
text --pierce Read through CLOSED shadow DOM (injected panels)
html <selector> Get inner HTML of element html <selector> Get inner HTML of element
value <selector> Get value of input element value <selector> Get value of input element
attr <selector> <name> Get attribute value attr <selector> <name> Get attribute value
@@ -1907,7 +2011,10 @@ Global Options:
--session <name> Use specific session --session <name> Use specific session
Examples: Examples:
chrome-use get text @e1 chrome-use get text # whole page across ALL frames (default)
chrome-use get text @e1 # one element
chrome-use get text --main # main content, no nav/sidebar boilerplate
chrome-use frames # list frames + where the text lives
chrome-use get html "#content" chrome-use get html "#content"
chrome-use get value "#email-input" chrome-use get value "#email-input"
chrome-use get attr "#link" href chrome-use get attr "#link" href
@@ -2778,6 +2885,20 @@ Notes:
- Streaming is always enabled. Set AGENT_BROWSER_STREAM_PORT to bind to a - Streaming is always enabled. Set AGENT_BROWSER_STREAM_PORT to bind to a
specific port instead of the default OS-assigned port. specific port instead of the default OS-assigned port.
The WS is BIDIRECTIONAL the high-throughput way to drive a live/real-time page
(games, canvas apps) instead of one screenshot + one CLI call per action:
- Server -> client (JSON text frames):
{"type":"frame","data":"<base64 jpeg>"} live screencast (~60fps)
plus status / tabs messages.
- Client -> server (send JSON text):
{"type":"input_keyboard","eventType":"keyDown|keyUp","key":" ","code":"Space",
"windowsVirtualKeyCode":32}
{"type":"input_mouse","eventType":"mousePressed|mouseReleased|mouseMoved",
"x":640,"y":360,"button":"left","clickCount":1}
{"type":"input_touch","eventType":"touchStart|touchEnd","touchPoints":[...]}
Connect once and run a tight local loop: read frames, send timed input no
per-action process spawn, no round-trip. Works over the extension relay too.
Global Options: Global Options:
--json Output as JSON --json Output as JSON
--session <name> Use specific session --session <name> Use specific session
@@ -3070,7 +3191,9 @@ Core Commands:
dblclick <sel> Double-click element dblclick <sel> Double-click element
type <sel> <text> Type into element type <sel> <text> Type into element
fill <sel> <text> Clear and fill fill <sel> <text> Clear and fill
press <key> Press key (Enter, Tab, Control+a) press <key> [--hold <ms>] Press key (Enter, Tab, Control+a). --hold keeps it
down <ms> then releases precise (in-daemon), for
games/charge: `press d --hold 800`
keydown <key> Hold a key down (no auto-release) for games/shortcuts keydown <key> Hold a key down (no auto-release) for games/shortcuts
keyup <key> Release a held key. Pair with keydown to hold-to-move: keyup <key> Release a held key. Pair with keydown to hold-to-move:
`keydown d` `keyup d` `keydown d` `keyup d`
@@ -3101,10 +3224,15 @@ Navigation:
Get Info: chrome-use get <what> [selector] Get Info: chrome-use get <what> [selector]
text, html, value, attr <name>, title, url, count, box, styles, cdp-url text, html, value, attr <name>, title, url, count, box, styles, cdp-url
text (no selector = whole page, all frames), text --main, frames (list)
Check State: chrome-use is <what> <selector> Check State: chrome-use is <what> <selector>
visible, enabled, checked visible, enabled, checked
Anti-bot: chrome-use stealth | cf-status
stealth stealth self-check (webdriver/UA/plugins + overrides)
cf-status Cloudflare challenge + cf_clearance preflight (skip re-solving)
Find Elements: chrome-use find <locator> <value> <action> [text] Find Elements: chrome-use find <locator> <value> <action> [text]
role, text, label, placeholder, alt, title, testid, first, last, nth role, text, label, placeholder, alt, title, testid, first, last, nth
@@ -3199,6 +3327,7 @@ Confirmation:
Sessions: Sessions:
session Show current session name session Show current session name
session list List active sessions session list List active sessions
sessions List running session daemons (alias of daemon status)
daemon status List running session daemons (+ relay state) daemon status List running session daemons (+ relay state)
daemon restart Kill all session daemons; keeps the extension relay daemon restart Kill all session daemons; keeps the extension relay
up. Clears stale/cross-leaked state after an upgrade. up. Clears stale/cross-leaked state after an upgrade.
Binary file not shown.
Binary file not shown.
+46 -3
View File
@@ -30,6 +30,16 @@ const tabs = new Map()
const sessionToTab = new Map() const sessionToTab = new Map()
/** child (OOPIF/worker) sessionId -> tabId */ /** child (OOPIF/worker) sessionId -> tabId */
const childSessionToTab = new Map() const childSessionToTab = new Map()
/** sessionId -> CDP targetId, kept ACROSS detach so a dead `cb-tab-<oldTabId>`
* session can be recovered by its stable targetId when the cross-process nav
* gave the tab a new Chrome tabId (issue #24). Capped to bound memory. */
const sessionTargets = new Map()
function rememberSessionTarget(sessionId, targetId) {
if (!sessionId || !targetId) return
sessionTargets.delete(sessionId)
sessionTargets.set(sessionId, targetId)
if (sessionTargets.size > 256) sessionTargets.delete(sessionTargets.keys().next().value)
}
/** tab-group name -> chrome tabGroups id (best-effort cache) */ /** tab-group name -> chrome tabGroups id (best-effort cache) */
const groupIdByName = new Map() const groupIdByName = new Map()
@@ -172,18 +182,50 @@ function tabIdFromSession(sessionId) {
// (closed / restricted). (issues #20.1, #23) // (closed / restricted). (issues #20.1, #23)
async function recoverSessionTab(sessionId) { async function recoverSessionTab(sessionId) {
const tabId = tabIdFromSession(sessionId) const tabId = tabIdFromSession(sessionId)
if (tabId == null) return null // 1) Fast path: the encoded Chrome tabId still exists — re-attach it (covers
// the common renderer-process swap where the tabId is preserved, #23).
if (tabId != null) {
for (let i = 0; i < 3; i++) { for (let i = 0; i < 3; i++) {
const tab = await chrome.tabs.get(tabId).catch(() => null) const tab = await chrome.tabs.get(tabId).catch(() => null)
if (!eligible(tab)) return null if (!eligible(tab)) break // tabId is gone — fall through to targetId recovery
try { try {
await attachTab(tabId) await attachTab(tabId)
if (tabs.has(tabId)) return tabId if (tabs.has(tabId)) return tabId
} catch { } catch {
// mid-swap: the tab exists but isn't attachable yet — back off and retry. // mid-swap: tab exists but isn't attachable yet — back off and retry.
} }
await new Promise((r) => setTimeout(r, 120 + i * 150)) await new Promise((r) => setTimeout(r, 120 + i * 150))
} }
}
// 2) The Chrome tabId is gone, but the CDP targetId is STABLE across the nav.
// Some cross-process hops (Mercari's signin token exchange) give the tab a
// NEW tabId while keeping the same target, so `cb-tab-<oldTabId>` can't be
// recovered by tabId. Find the tab now hosting our remembered targetId via
// chrome.debugger.getTargets(), attach it, and ALIAS the dead session to it
// so the daemon's session id keeps resolving. Longer window: this hop can
// take several seconds to settle (issue #24).
const targetId = sessionTargets.get(sessionId)
if (targetId) {
for (let i = 0; i < 6; i++) {
const targets = await chrome.debugger.getTargets().catch(() => null)
const t = targets && targets.find((x) => x.id === targetId && x.tabId != null)
if (t && t.tabId != null) {
const tab = await chrome.tabs.get(t.tabId).catch(() => null)
if (eligible(tab)) {
try {
await attachTab(t.tabId)
if (tabs.has(t.tabId)) {
sessionToTab.set(sessionId, t.tabId) // alias dead session -> live tab
return t.tabId
}
} catch {
// not attachable yet — keep waiting for the swap to settle.
}
}
}
await new Promise((r) => setTimeout(r, 300 + i * 300))
}
}
return null return null
} }
@@ -343,6 +385,7 @@ async function attachTab(tabId) {
const entry = { sessionId, targetId } const entry = { sessionId, targetId }
tabs.set(tabId, entry) tabs.set(tabId, entry)
sessionToTab.set(sessionId, tabId) sessionToTab.set(sessionId, tabId)
rememberSessionTarget(sessionId, targetId)
setBadge(tabId, port ? 'on' : 'connecting') setBadge(tabId, port ? 'on' : 'connecting')
postToHost({ postToHost({
method: 'forwardCDPEvent', method: 'forwardCDPEvent',
+1 -1
View File
@@ -1,7 +1,7 @@
{ {
"manifest_version": 3, "manifest_version": 3,
"name": "chrome-use", "name": "chrome-use",
"version": "0.4.8", "version": "0.4.9",
"description": "Let chrome-use drive your logged-in Chrome \u2014 install once, no token, no per-use confirmation.", "description": "Let chrome-use drive your logged-in Chrome \u2014 install once, no token, no per-use confirmation.",
"key": "MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA6vQIyscGIPYPZdSpPwPL0+0gxUROyRgCpmvCSDoc8XUm4qm97VbKnD9Ijc1lV22lNWZtE78gaRjt6BeSfuMgnBymnhLKjN1gU6AI5QUU0mrJyeHdWKvrKQR5FmsM2A7Xr1ykE2SiiS8zNUS3Y/6O5l+Nva7wrVy6E4a2dkBVQkOsu+DV+nEZvhIyuDY5D5SPXqNwUTWTaglwj5mjvHz36xSwCWlPmrtJ+ED0AUyrb2z4GIOmvk4kqtBVrh/UD058klLo4CkYOnIybB5aV6WYuwarfPY4bF/dLggPem+ewLNTUNBuwrxj/A4nUv0LJTuRO8rR7f8WR9qnRCY0Ic5saQIDAQAB", "key": "MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA6vQIyscGIPYPZdSpPwPL0+0gxUROyRgCpmvCSDoc8XUm4qm97VbKnD9Ijc1lV22lNWZtE78gaRjt6BeSfuMgnBymnhLKjN1gU6AI5QUU0mrJyeHdWKvrKQR5FmsM2A7Xr1ykE2SiiS8zNUS3Y/6O5l+Nva7wrVy6E4a2dkBVQkOsu+DV+nEZvhIyuDY5D5SPXqNwUTWTaglwj5mjvHz36xSwCWlPmrtJ+ED0AUyrb2z4GIOmvk4kqtBVrh/UD058klLo4CkYOnIybB5aV6WYuwarfPY4bF/dLggPem+ewLNTUNBuwrxj/A4nUv0LJTuRO8rR7f8WR9qnRCY0Ic5saQIDAQAB",
"icons": { "icons": {
+1 -1
View File
@@ -1,6 +1,6 @@
{ {
"name": "chrome-use", "name": "chrome-use",
"version": "1.4.1", "version": "1.5.8",
"description": "chrome-use — drive your real, logged-in Chrome from any AI agent, stealth by default", "description": "chrome-use — drive your real, logged-in Chrome from any AI agent, stealth by default",
"type": "module", "type": "module",
"packageManager": "pnpm@11.1.3", "packageManager": "pnpm@11.1.3",
+78 -9
View File
@@ -127,6 +127,19 @@ cadence, and scroll/drag ease. Default `off`; a per-navigation detector
auto-escalates pages guarded by Akamai/PerimeterX/DataDome to `human`. Leave it auto-escalates pages guarded by Akamai/PerimeterX/DataDome to `human`. Leave it
on auto; force `human` only when you already know the target scores behaviour. on auto; force `human` only when you already know the target scores behaviour.
**Cloudflare clearance — solve once, reuse.** Passing a Cloudflare challenge
mints a `cf_clearance` cookie (HttpOnly — invisible to `eval`/`document.cookie`;
read it via `chrome-use cookies`). It's bound to your **IP + User-Agent**: reuse
the same exit IP and UA and you skip the challenge until it expires. Driving the
user's real Chrome (relay) persists it natively; for isolated sessions,
`--session-name <name>` save/restores it. Before spending effort solving, run
`chrome-use cf-status` (aliases `cf`, `clearance`): it reports whether the page
is *currently* a Cloudflare challenge and whether a still-valid `cf_clearance`
exists, with a recommendation — `proceed` (already cleared, don't re-solve),
`solve` (challenge up, no clearance), or `reissue` (clearance present but page
still blocks → IP/UA drifted, re-solve). Use it as a preflight to avoid
re-solving what you already cleared.
## Two ways to drive a page — and when to drop to `eval` ## Two ways to drive a page — and when to drop to `eval`
You have a **real Chrome with the user's DOM**. Two layers, mix them freely: You have a **real Chrome with the user's DOM**. Two layers, mix them freely:
@@ -207,7 +220,11 @@ assigned fresh on every snapshot.
For unstructured reading (no refs needed): For unstructured reading (no refs needed):
```bash ```bash
chrome-use get text @e1 # visible text of an element chrome-use get text # WHOLE PAGE — all frames by default (see below)
chrome-use get text @e1 # visible text of one element (or a CSS selector)
chrome-use get text --main # main content only — skip nav/header/sidebar
chrome-use get text --pierce # read through CLOSED shadow DOM (injected panels)
chrome-use frames # list every frame + where the text lives
chrome-use get html @e1 # innerHTML chrome-use get html @e1 # innerHTML
chrome-use get attr @e1 href # any attribute chrome-use get attr @e1 href # any attribute
chrome-use get value @e1 # input value chrome-use get value @e1 # input value
@@ -216,6 +233,27 @@ chrome-use get url # current URL
chrome-use get count ".item" # count matching elements chrome-use get count ".item" # count matching elements
``` ```
**Whole-page text is cross-frame by default.** `chrome-use get text` with no
selector aggregates visible text across **every** frame — top document plus
same-process child frames plus cross-origin iframes — so you never silently miss
content that lives in an iframe (Yahoo Auctions / Rakuten / Mercari shop
descriptions, embedded checkout/spec frames). Each child frame is delimited with
a `----- frame [kind] url -----` marker. You do **not** need to remember a flag —
the default already reads all frames. (`--all-frames` is still accepted as an
explicit alias.)
So: when text looks missing or wrong, you don't have to guess — just
`chrome-use get text` reads everything. To **see** the structure (which frame
holds what), run `chrome-use frames`. To **cut boilerplate** (global nav/header/
footer, "related items" sidebars), use `chrome-use get text --main`. If content
is lazy-loaded, `scroll` it into view first, then read.
**Closed shadow DOM.** Some injected UI (browser-extension debug panels, web
components) renders into a *closed* shadow root that `eval`/`innerText` cannot
read. `chrome-use get text --pierce` reads through closed shadow roots and child
documents via the CDP DOM tree — use it when content is clearly on screen (you
see it in a screenshot) but `get text`/`eval` come back empty.
## Interacting ## Interacting
```bash ```bash
@@ -243,11 +281,11 @@ chrome-use pick @e4 --option "Europe" # ANY combobox (react-select / ARIA /
# (no silent no-op). Use this for custom # (no silent no-op). Use this for custom
# dropdowns where `select` returns ✓ but # dropdowns where `select` returns ✓ but
# changes nothing. # changes nothing.
chrome-use upload @e5 file1.pdf # upload file(s) — NOTE: needs a --launch/direct-CDP chrome-use upload @e5 file1.pdf # upload file(s) — works over the extension relay too:
# session. Over the extension relay it CANNOT work # chrome.debugger forbids setFileInputFiles, so the
# (Chrome's chrome.debugger forbids it); chrome-use # file's bytes are streamed into the page and rebuilt as
# errors with a hint. Carry your login into a launched # a File there (chunked under native-messaging's 1 MiB cap).
# session via `cookies export` | `cookies set --curl`. # Works on file <input>s and drop/paste composers (e.g. X).
chrome-use scroll down 500 # scroll page (up/down/left/right) chrome-use scroll down 500 # scroll page (up/down/left/right)
chrome-use scrollintoview @e1 # scroll element into view chrome-use scrollintoview @e1 # scroll element into view
chrome-use drag @e1 @e2 # drag and drop chrome-use drag @e1 @e2 # drag and drop
@@ -309,12 +347,43 @@ detects this and prints a one-line hint. Drive them the screenshot way:
chrome-use screenshot /tmp/s.png # SEE the state (your only read path — chrome-use screenshot /tmp/s.png # SEE the state (your only read path —
# eval/get text return nothing useful) # eval/get text return nothing useful)
chrome-use click 640 360 # interact by viewport coordinate chrome-use click 640 360 # interact by viewport coordinate
chrome-use keydown d; sleep 0.6; chrome-use keyup d # hold-to-move chrome-use press d --hold 800 # hold-to-move, precise (timed in-daemon —
# NOT keydown+shell-sleep+keyup, which
# adds ~250ms jitter per round-trip)
chrome-use press Space # discrete actions (jump/attack/confirm) chrome-use press Space # discrete actions (jump/attack/confirm)
``` ```
Each command is a ~250ms round-trip, so this is fine for turn-based / canvas **Don't drive frame-by-frame with one CLI call per action** — that's the slowest,
*apps* but too slow to play a real-time 60fps action game frame-by-frame. lowest-fidelity way (each call is a process spawn + round-trip). Script a *timed
sequence in a single round-trip* with `batch` (it sends each step to the running
daemon; `press --hold` and `wait` block in-daemon, so timing is precise):
```bash
chrome-use batch "press d --hold 900" "press j" "press j" "wait 200" "press d --hold 500"
```
Also try reading real state instead of pixels: `eval` runs in the page's main
world, so for a framework/engine game you can often reach its globals (e.g. a
Phaser/PIXI/Three instance, a store, `window.__GAME__`) and read positions/score
directly — far better than guessing from a screenshot.
**For genuinely real-time driving, drop the CLI entirely and use the WebSocket.**
`chrome-use stream enable` opens a bidirectional WS (`stream status` prints the
`ws://127.0.0.1:<port>`). Connect once and you get a live ~60fps screencast AND
can send input on the same socket — no per-action process spawn, no round-trip,
works over the extension relay:
```js
// node (global WebSocket): live frames + locally-timed input
const ws = new WebSocket("ws://127.0.0.1:PORT")
ws.onmessage = e => { const m = JSON.parse(e.data); if (m.type==="frame") {/* base64 jpeg */} }
const k = (eventType,key,code,vk) => ws.send(JSON.stringify({type:"input_keyboard",eventType,key,code,windowsVirtualKeyCode:vk}))
k("keyDown"," ","Space",32); setTimeout(()=>k("keyUp"," ","Space",32), 80) // a jump
// also: {type:"input_mouse",eventType:"mousePressed",x,y,button:"left",clickCount:1}
```
This is the difference between watching a slideshow and playing the game. Reserve
screenshots for one-off checks; use the WS for any sustained real-time control.
## Waiting (read this) ## Waiting (read this)