Compare commits

...
85 Commits
Author SHA1 Message Date
leeguooooo 50b27ac0e0 feat(site): auto-sync + auto-suggest adapters (auto-trigger)
Release binaries / Build macOS ARM64 (push) Has been cancelled
Release binaries / Build macOS x64 (push) Has been cancelled
Release binaries / Build Linux ARM64 (push) Has been cancelled
Release binaries / Build Linux musl ARM64 (push) Has been cancelled
Release binaries / Build Linux musl x64 (push) Has been cancelled
Release binaries / Build Linux x64 (push) Has been cancelled
Release binaries / Build Windows x64 (push) Has been cancelled
Release binaries / Attach binaries to GitHub Release (push) Has been cancelled
Make `site` trigger itself so an agent doesn't have to know adapters exist.

Auto-sync: the pack refreshes on first use and on a TTL (default 7d), both in
the `site` command path (blocking, fast) and as a non-blocking background task
on daemon startup — so ~/.chrome-use/sites/.index.json is always populated with
zero added latency. Tune via AGENT_BROWSER_SITES_TTL_DAYS; disable with
AGENT_BROWSER_SITES_NO_AUTO_UPDATE=1. `update` now writes .last_update + a
domain→adapters .index.json (read-only adapters ordered first).

Auto-suggest: `open`/`navigate`/`snapshot` onto a domain with adapters attaches
`siteAdapters: {domain, commands}` to the response; the CLI prints a
`💡 site adapters for <domain>` hint (stderr) and the field rides along in --json.
SKILL.md tells the agent to prefer the listed `site <name>/<cmd>` over scraping.
This keeps the 'never auto-disrupt user tabs' guarantee — it suggests, the agent
decides; nothing auto-runs on navigation.

site.rs: needs_refresh/adapters_for_domain/write_domain_index + timestamp/index
in update(). daemon.rs: background bootstrap. actions.rs: with_site_hint on
navigate + snapshot. output.rs: hint render. Verified live: open github.com →
hint leads with read-only github/issues; --json carries siteAdapters.
2026-06-17 17:15:34 +09:00
leeguooooo d81bc01645 feat(site): bb-sites adapters — turn any site into a structured-data CLI
Release binaries / Build macOS ARM64 (push) Has been cancelled
Release binaries / Build macOS x64 (push) Has been cancelled
Release binaries / Build Linux ARM64 (push) Has been cancelled
Release binaries / Build Linux musl ARM64 (push) Has been cancelled
Release binaries / Build Linux musl x64 (push) Has been cancelled
Release binaries / Build Linux x64 (push) Has been cancelled
Release binaries / Build Windows x64 (push) Has been cancelled
Release binaries / Attach binaries to GitHub Release (push) Has been cancelled
Add `chrome-use site` — run community bb-sites adapters over chrome-use's
stealth transport. An adapter is a per-command JS function that calls a
site's own JSON API from inside your logged-in tab (your cookies, same-origin
fetch, the site's modules) and returns clean JSON — no clicking/scraping.

- site update   fetch the upstream bb-sites pack into ~/.chrome-use/sites
- site list     list installed adapters (name/cmd)
- site info     show an adapter's @meta (args, domain, capabilities)
- site <name>/<cmd> [args]  navigate to its domain (reuse tab if already there) + eval, return JSON

chrome-use ships zero adapter code; `site update` fetches epiral/bb-sites at
runtime (like a package manager). Adapters remain their authors' property.

cli/src/site.rs (load/parse/build_eval/list/update/map_args + tests), wired
via commands.rs (parse), actions.rs (handle_site), main.rs (CLI dispatch).
Docs in README, README.zh, skill-data/core. Verified live: github/issues
returned 30 real issues as JSON over the relay.
2026-06-17 16:02:40 +09:00
leeguooooo c667e0e704 docs(zh): bring Chinese README to parity with English
Translate the sections the zh README was missing: 为什么用扩展 (extension vs raw
debug port + comparison table), 命令名 (chrome-use/abs same binary), 自动化测试
(chrome-use test YAML suite), 自己验证 (verify-yourself detector list), 调参 (env
var table); fill out 反检测 (0% stealth explanation + --launch CreepJS caveat),
the install 'other ways' details, and the blog/X footer. All 12 top-level sections
now match EN.
2026-06-17 14:49:52 +09:00
leeguooooo 08cb8dbeb9 docs: link blog + X in README 2026-06-17 14:15:52 +09:00
leeguooooo fd2cdcde77 chore(release): 1.5.18 — issue-reporting guidance + clear data:-over-relay error
Release binaries / Build macOS ARM64 (push) Has been cancelled
Release binaries / Build macOS x64 (push) Has been cancelled
Release binaries / Build Linux ARM64 (push) Has been cancelled
Release binaries / Build Linux musl ARM64 (push) Has been cancelled
Release binaries / Build Linux musl x64 (push) Has been cancelled
Release binaries / Build Linux x64 (push) Has been cancelled
Release binaries / Build Windows x64 (push) Has been cancelled
Release binaries / Attach binaries to GitHub Release (push) Has been cancelled
2026-06-17 14:10:07 +09:00
leeguooooo 8e001e3d88 feat(dx): guide users to file issues; clear error for data: URLs over the relay
#5: the CLI never pointed users at the issue tracker. `--help` and `--version`
now print the issues URL (https://github.com/leeguooooo/chrome-use/issues), so
agents hitting a rough edge know where to report it (the skill already nudges).

#1: navigating to a `data:` URL over the extension relay fails with a cryptic
`net::ERR_ABORTED` on an about:blank tab (chrome.debugger/chrome.tabs can't drive
a top-frame data: navigation; it works fine under --launch). Detect that case and
explain it — use a real http(s)://file:// URL or --launch — instead of leaking
the raw code. (Surfaced while stress-testing 5 concurrent agents.)
2026-06-17 14:10:04 +09:00
leeguooooo eb2bc343a0 chore(release): 1.5.17 — complete multi-agent isolation (no foreign tab/pop-up adoption on click)
Release binaries / Build macOS ARM64 (push) Has been cancelled
Release binaries / Build macOS x64 (push) Has been cancelled
Release binaries / Build Linux ARM64 (push) Has been cancelled
Release binaries / Build Linux musl ARM64 (push) Has been cancelled
Release binaries / Build Linux musl x64 (push) Has been cancelled
Release binaries / Build Linux x64 (push) Has been cancelled
Release binaries / Build Windows x64 (push) Has been cancelled
Release binaries / Attach binaries to GitHub Release (push) Has been cancelled
2026-06-17 13:31:50 +09:00
leeguooooo d86c9c4be2 fix(relay): don't adopt foreign tabs/pop-ups on click — completes multi-agent isolation (#last hole)
After strict isolation (v1.5.16) a session's tracked set is only its OWN tabs, so
in `adopt_newly_opened` (run after every click to follow a pop-up) EVERY foreign
tab looks "new" relative to the session's `before` set and got adopted — a
click-heavy flow on a busy shared Chrome pulled other agents'/the user's tabs
(github, Lark, iphone-use) into the session mid-flow.

A pop-up the agent itself opened can't be told apart from a foreign tab over the
relay (the synthesized targetInfo carries no opener/window/group), so on the relay
`adopt_newly_opened` now adopts nothing: the agent drives only tabs it explicitly
created; pop-ups (OAuth/login windows) are the user's. Launched browsers (every
tab ours) still follow pop-ups. Verified live: a fresh click-heavy relay session
stays clean (only its own tabs), and 5 concurrent agents churning tabs show zero
cross-agent drift.
2026-06-17 13:31:49 +09:00
leeguooooo 32c25a6627 chore(release): 1.5.16 — strict multi-agent tab isolation on the relay
Release binaries / Build macOS ARM64 (push) Has been cancelled
Release binaries / Build macOS x64 (push) Has been cancelled
Release binaries / Build Linux ARM64 (push) Has been cancelled
Release binaries / Build Linux musl ARM64 (push) Has been cancelled
Release binaries / Build Linux musl x64 (push) Has been cancelled
Release binaries / Build Linux x64 (push) Has been cancelled
Release binaries / Build Windows x64 (push) Has been cancelled
Release binaries / Attach binaries to GitHub Release (push) Has been cancelled
2026-06-17 11:50:10 +09:00
leeguooooo a8ce3dd3f8 fix(relay): strict multi-agent isolation — a session owns only its own tabs
Several agents (and other tools opening tabs) share one real Chrome via the
relay. Previously every session adopted ALL tabs from Target.getTargets, so
another agent's tab churn polluted the list, dropped the tab being driven, and
drifted commands onto the wrong page (the W-8BEN tax tab vanished mid-flow when a
concurrent iphone-use agent opened tabs).

A tab group belongs to exactly one agent. On the relay a session now tracks and
drives ONLY the tabs it created (its own colored group) plus pop-ups its own
clicks open — it never adopts the user's or other agents' tabs:

- discover_and_attach (relay): create the session's own tab and pin it; do not
  adopt any existing foreign tab.
- resync_targets (relay): never adopt unknown targets; never prune the session's
  tabs on a single getTargets snapshot (multi-agent churn / cross-process-nav
  gaps routinely omit live tabs) — prune only after RELAY_PRUNE_MISSES
  consecutive absent snapshots (debounced), pinned active always protected.
- adopt_newly_opened: a tab that appears right after this session's action is a
  pop-up we opened — record it as owned.

Launched browsers (every tab ours) keep adopting all tabs. Adds debounced_prune_ids
+ unit tests for the churn tolerance.
2026-06-17 11:50:09 +09:00
leeguooooo 997373fd57 chore(release): 1.5.15 — trusted activation for in-iframe buttons (#39)
Release binaries / Build macOS ARM64 (push) Has been cancelled
Release binaries / Build macOS x64 (push) Has been cancelled
Release binaries / Build Linux ARM64 (push) Has been cancelled
Release binaries / Build Linux musl ARM64 (push) Has been cancelled
Release binaries / Build Linux musl x64 (push) Has been cancelled
Release binaries / Build Linux x64 (push) Has been cancelled
Release binaries / Build Windows x64 (push) Has been cancelled
Release binaries / Attach binaries to GitHub Release (push) Has been cancelled
2026-06-17 10:16:17 +09:00
leeguooooo 5a858af93f fix(iframe): trusted activation for in-iframe buttons — keyboard, not synthetic click (#39)
A DOM `.click()` is isTrusted:false, which security-sensitive embedded forms
reject — Google Payments' enabled `保存` button silently no-op'd, so a
cross-origin payment/checkout/KYC form could be read, scrolled, and typed into
but never submitted. A coordinate click can't help either: getBoxModel for a
sub-frame node returns frame-local coords that don't compose the iframe offset,
so it lands wrong (verified — the same-origin probe came back isTrusted:false
via the coordinate fallback).

Fix: click on an in-iframe ref now focuses the element in its own frame session
and dispatches a real Enter (Space for checkbox-like roles) on the page session.
Chrome routes the key to the focused element across frames (same mechanism as
`type --focused`), and Enter/Space on a focused button/link/checkbox fires a
trusted click. Non-activatable roles fall back to DOM .click().

Adds e2e_iframe_button_click_is_trusted (+ fixture): an in-iframe button records
event.isTrusted into its own text; the test asserts the ref-click delivers
isTrusted:true.
2026-06-17 10:16:15 +09:00
leeguooooo 58dc02bfdc chore(release): 1.5.14 — fix eval await regression (replMode) + default scroll; green CI (#36, #38)
Release binaries / Build macOS ARM64 (push) Has been cancelled
Release binaries / Build macOS x64 (push) Has been cancelled
Release binaries / Build Linux ARM64 (push) Has been cancelled
Release binaries / Build Linux musl ARM64 (push) Has been cancelled
Release binaries / Build Linux musl x64 (push) Has been cancelled
Release binaries / Build Linux x64 (push) Has been cancelled
Release binaries / Build Windows x64 (push) Has been cancelled
Release binaries / Attach binaries to GitHub Release (push) Has been cancelled
2026-06-17 02:34:55 +09:00
leeguooooo c47601bd7b fix(eval): replMode only for sync let/const decls, keep awaitPromise for async (#38)
replMode and awaitPromise are mutually exclusive in Chrome — under replMode a
returned promise serialises to {} instead of being awaited, which broke every
fetch/async eval (e2e_domain_filter, e2e_headers, e2e_react_tree all regressed).
Enable replMode only for synchronous scripts that declare a top-level let/const
(the #38 case); promise-returning scripts keep awaitPromise — restoring the
pre-#38 await behaviour while still fixing the let-redeclaration collision.
2026-06-17 02:08:11 +09:00
leeguooooo 0296bc7a88 fix(scroll): keep default scroll on window.scrollBy; wheel only for --at/--frame (#36)
The centered-wheel default no-op'd on some pages (headless e2e_hover_scroll_press
regressed). Restore window.scrollBy for plain page scroll; the coordinate wheel
stays opt-in via --at/--frame for cross-origin iframe content.
2026-06-17 02:01:23 +09:00
leeguooooo 32e203b908 style: cargo fmt (fixes the CI format-check failure) 2026-06-17 01:33:22 +09:00
leeguooooo fc51cd63ba chore(release): 1.5.13 — eval replMode (re-declarable let/const) + snapshot-first skill rule (#37, #38)
Release binaries / Build macOS ARM64 (push) Has been cancelled
Release binaries / Build macOS x64 (push) Has been cancelled
Release binaries / Build Linux ARM64 (push) Has been cancelled
Release binaries / Build Linux musl ARM64 (push) Has been cancelled
Release binaries / Build Linux musl x64 (push) Has been cancelled
Release binaries / Build Linux x64 (push) Has been cancelled
Release binaries / Build Windows x64 (push) Has been cancelled
Release binaries / Attach binaries to GitHub Release (push) Has been cancelled
2026-06-17 01:15:31 +09:00
leeguooooo f714c7920b fix(eval): replMode so successive evals can re-declare let/const; snapshot-first skill rule (#37, #38)
#38: `chrome-use eval` now runs with Runtime.evaluate replMode (like the DevTools
console) — top-level `let`/`const` no longer throw "already been declared" across
successive evals (independent `eval` steps in a `test` suite collided in the
page's shared lexical scope), and top-level await is allowed. Main-world and
completion-value semantics are unchanged.

#37: core skill gains a hard rule — snapshot-first, never screenshot+coordinates
to locate form fields/buttons; `snapshot -i` now pierces cross-origin iframes and
lists their elements by @ref; screenshots are for visual checks only, and a
full-page retina screenshot often exceeds an image reader's limits.
2026-06-17 01:15:31 +09:00
leeguooooo 1ac8ef7732 chore(release): 1.5.12 — relay-safe hover/dblclick/drag, deeper iframe snapshot, key-events typing (#37)
Release binaries / Build macOS ARM64 (push) Has been cancelled
Release binaries / Build macOS x64 (push) Has been cancelled
Release binaries / Build Linux ARM64 (push) Has been cancelled
Release binaries / Build Linux musl ARM64 (push) Has been cancelled
Release binaries / Build Linux musl x64 (push) Has been cancelled
Release binaries / Build Linux x64 (push) Has been cancelled
Release binaries / Build Windows x64 (push) Has been cancelled
Release binaries / Attach binaries to GitHub Release (push) Has been cancelled
2026-06-17 00:58:23 +09:00
leeguooooo 9f24e66033 fix(relay): DOM-dispatch hover/dblclick/drag; deeper iframe snapshot; key-events typing (#37)
Follow-up to #36 — make the whole interaction surface reach cross-origin OOPIFs
and stop coordinate events drifting onto the user's foreground tab over the relay.

- hover/dblclick/drag now DOM-dispatch over the relay or into an iframe (like
  click already did): a coordinate Input event isn't confined to the target tab
  on a busy real Chrome and can't map an OOPIF element's box to a top-viewport
  point. drag does an HTML5 DnD in the element's frame; cross-frame drag errors
  loudly instead of drifting.
- snapshot recurses iframes to MAX_IFRAME_DEPTH (3) instead of one level, so refs
  inside nested payment/checkout widgets get a frame_id and resolve into the
  right frame.
- relay tab adoption merges several Target.getTargets snapshots — a single flaky
  relay snapshot was dropping live tabs (a driven tab vanished after restart).
- `type --key-events` (alias --keys) sends real per-character keyDown/keyUp
  instead of Input.insertText, so autocomplete/combobox widgets that ignore the
  insertText input event fire (Google address postal lookup; commits Angular
  reactive forms so Save enables).
- SKILL: hard "snapshot-first, never default to screenshot+coordinates" rule;
  snapshot -i pierces cross-origin iframes since v1.5.12; cross-origin iframe
  driving guidance (#37).
2026-06-17 00:58:12 +09:00
leeguooooo 70ab38d35f chore(release): 1.5.11 — cross-origin iframe scroll/click + open auto-reattach (#35, #36)
Release binaries / Build macOS ARM64 (push) Has been cancelled
Release binaries / Build macOS x64 (push) Has been cancelled
Release binaries / Build Linux ARM64 (push) Has been cancelled
Release binaries / Build Linux musl ARM64 (push) Has been cancelled
Release binaries / Build Linux musl x64 (push) Has been cancelled
Release binaries / Build Linux x64 (push) Has been cancelled
Release binaries / Build Windows x64 (push) Has been cancelled
Release binaries / Attach binaries to GitHub Release (push) Has been cancelled
2026-06-16 18:05:15 +09:00
leeguooooo 6830df50ea fix(relay): reach cross-origin iframes; auto-reattach open (#35, #36)
#35: `open` auto-reattaches when the bound relay tab is gone — drops the dead
page, opens a fresh tab in the session's group, and navigates it, instead of
only `tab new` recovering.

#36: scroll and click now reach content inside cross-origin OOPIFs:
- scroll dispatches a real wheel at a viewport point (default center, --at x,y,
  or --frame n) so it scrolls the iframe under the pointer, which
  window.scrollBy on the top document silently no-ops on.
- over the extension relay, clicks always use DOM-dispatch instead of
  coordinate Input events — a coordinate event isn't confined to the target tab
  on a busy real Chrome (it drifted onto the foreground tab) and an OOPIF
  element's box can't be mapped to a top-viewport point.
2026-06-16 18:05:06 +09:00
leeguooooo cd47ec43d0 chore(release): 1.5.10 — warn on debug-port launch while relay is up (#32)
Release binaries / Build macOS ARM64 (push) Has been cancelled
Release binaries / Build macOS x64 (push) Has been cancelled
Release binaries / Build Linux ARM64 (push) Has been cancelled
Release binaries / Build Linux musl ARM64 (push) Has been cancelled
Release binaries / Build Linux musl x64 (push) Has been cancelled
Release binaries / Build Linux x64 (push) Has been cancelled
Release binaries / Build Windows x64 (push) Has been cancelled
Release binaries / Attach binaries to GitHub Release (push) Has been cancelled
2026-06-16 14:32:29 +09:00
leeguooooo 2cd361817d fix(launch): warn when launching a debug-port Chrome while the relay is up (#32)
The connect-mode diagnostic (1.5.5) proved the 'Allow remote debugging?' modal
is NOT Chrome 149 UX (my earlier hypothesis) — it's chrome-use launching a fresh
debug-port Chrome on session=default while the ab-connect relay is up (32 logged
CONSENT-MODAL-RISK launches), almost always from a stray --launch/--no-auto-connect.
A launch now warns loudly when the relay is available, naming the modal and how
to avoid it (drop --launch/--new, don't pass --no-auto-connect), so the modal is
self-explained and the offending caller is fixable.
2026-06-16 14:32:28 +09:00
leeguooooo 42f47c49aa chore(release): 1.5.9 — strip zero-width title unicode (#33) + screenshot --clip/element (#34)
Release binaries / Build macOS ARM64 (push) Has been cancelled
Release binaries / Build macOS x64 (push) Has been cancelled
Release binaries / Build Linux ARM64 (push) Has been cancelled
Release binaries / Build Linux musl ARM64 (push) Has been cancelled
Release binaries / Build Linux musl x64 (push) Has been cancelled
Release binaries / Build Linux x64 (push) Has been cancelled
Release binaries / Build Windows x64 (push) Has been cancelled
Release binaries / Attach binaries to GitHub Release (push) Has been cancelled
2026-06-16 14:26:41 +09:00
leeguooooo e29800df72 fix(tab-list): strip zero-width unicode from titles (#33); feat(screenshot): --clip pixel region + documented element capture (#34)
#33: some sites prepend runs of ZWJ/word-joiner/invisible-times/BOM to
document.title (badging/anti-scrape); left in, they polluted 'tab list', broke
text matching, and wrecked column alignment. sanitize_title() now strips
zero-width/bidi-format chars at every title ingestion point + get_title().

#34: 'screenshot <selector>' (element capture) already worked but was
undocumented; added 'screenshot --clip x,y,w,h' for an explicit pixel region
(CDP captureScreenshot clip), documented both in --help. Verified live.
2026-06-16 14:26:40 +09:00
leeguooooo e7e849ea39 chore(release): 1.5.8 — file upload over the extension relay (#13)
Release binaries / Build macOS ARM64 (push) Has been cancelled
Release binaries / Build macOS x64 (push) Has been cancelled
Release binaries / Build Linux ARM64 (push) Has been cancelled
Release binaries / Build Linux musl ARM64 (push) Has been cancelled
Release binaries / Build Linux musl x64 (push) Has been cancelled
Release binaries / Build Linux x64 (push) Has been cancelled
Release binaries / Build Windows x64 (push) Has been cancelled
Release binaries / Attach binaries to GitHub Release (push) Has been cancelled
2026-06-16 14:05:33 +09:00
leeguooooo ebb02c65c8 fix(relay): file upload now works over the extension relay (#13)
chrome.debugger forbids DOM.setFileInputFiles, so 'upload' used to hard-fail on
the relay and push users to a --launch/direct-CDP session. Now it falls back to
reconstructing the File entirely in the page (Playwright/Cypress-style: build a
File from the bytes, assign input.files = dataTransfer.files, fire input/change;
for drop/paste composers like X, dispatch synthetic paste+drop with the
DataTransfer). The bytes are streamed in <1 MiB base64 chunks because the relay
tunnels CDP through native messaging (1 MiB/message cap) — a whole image as one
arg closed the channel. Verified live over the relay: an 809 KB PNG lands intact
on a file input with change firing. No more direct-CDP needed for uploads.
2026-06-16 14:05:31 +09:00
leeguooooo 4317db636f chore(release): 1.5.7 — cf-status Cloudflare clearance preflight
Release binaries / Build macOS ARM64 (push) Has been cancelled
Release binaries / Build macOS x64 (push) Has been cancelled
Release binaries / Build Linux ARM64 (push) Has been cancelled
Release binaries / Build Linux musl ARM64 (push) Has been cancelled
Release binaries / Build Linux musl x64 (push) Has been cancelled
Release binaries / Build Linux x64 (push) Has been cancelled
Release binaries / Build Windows x64 (push) Has been cancelled
Release binaries / Attach binaries to GitHub Release (push) Has been cancelled
2026-06-16 12:19:26 +09:00
leeguooooo c99838a034 feat(cloudflare): cf-status preflight — skip re-solving when cf_clearance is still valid
Passing a Cloudflare challenge mints an HttpOnly cf_clearance cookie bound to
IP+UA. 'chrome-use cf-status' (aliases cf/cloudflare-status/clearance) reports
whether the active page is currently a CF challenge and whether a still-valid
cf_clearance exists (read via CDP — HttpOnly is invisible to document.cookie),
plus CF_VERIFIED_DEVICE trust, and a recommendation: proceed (already cleared,
don't re-solve) / solve (challenge up, no clearance) / reissue (clearance present
but page still blocks → IP/UA drifted). Lets an agent avoid re-solving what it
already cleared — the persistence optimization. Pure helpers unit-tested; live
-verified on a real cf_clearance.
2026-06-16 12:19:24 +09:00
leeguooooo dc2aa4cade chore(release): 1.5.6 — pin adopted tab against transient relay snapshots (#31)
Release binaries / Build macOS ARM64 (push) Has been cancelled
Release binaries / Build macOS x64 (push) Has been cancelled
Release binaries / Build Linux ARM64 (push) Has been cancelled
Release binaries / Build Linux musl ARM64 (push) Has been cancelled
Release binaries / Build Linux musl x64 (push) Has been cancelled
Release binaries / Build Linux x64 (push) Has been cancelled
Release binaries / Build Windows x64 (push) Has been cancelled
Release binaries / Attach binaries to GitHub Release (push) Has been cancelled
2026-06-15 17:00:42 +09:00
leeguooooo 7085f3bf36 fix(relay): don't prune the pinned target on a transient getTargets snapshot (#31)
Driving a busy real Chrome via the relay, a single Target.getTargets call
occasionally returns a different window's tabs ('tab list hops windows'). resync
pruned every tracked page absent from that snapshot — including the agent's
explicitly-adopted (pinned) tab — after which active-target resolution fell back
to active_page_index and eval/click/snapshot drifted onto a foreign tab
(about:blank / chrome-extension:// / the user's page), breaking any 3+ step flow.

prunable_target_ids() now protects the pinned active target from snapshot-based
pruning; a genuine close still arrives as Target.targetDestroyed (event drain) and
removes it properly. Unit-tested.
2026-06-15 17:00:38 +09:00
leeguooooo 29815ff5f3 chore(release): 1.5.5 — connect-mode diagnostic log for the remote-debugging consent modal (#31)
Release binaries / Build macOS ARM64 (push) Has been cancelled
Release binaries / Build macOS x64 (push) Has been cancelled
Release binaries / Build Linux ARM64 (push) Has been cancelled
Release binaries / Build Linux musl ARM64 (push) Has been cancelled
Release binaries / Build Linux musl x64 (push) Has been cancelled
Release binaries / Build Linux x64 (push) Has been cancelled
Release binaries / Build Windows x64 (push) Has been cancelled
Release binaries / Attach binaries to GitHub Release (push) Has been cancelled
2026-06-15 16:47:14 +09:00
leeguooooo 2a338d4c29 diag(connect): log CDP transport mode to detect 'Allow remote debugging?' modal source (#31)
The consent modal only appears on a raw remote-debugging attach or a browser we
launched with a debug port — never on the ab-connect extension relay. Append one
line per connection to ~/.chrome-use/connect-mode.log (relay | raw-port-attach |
launched | remote-ws), flagging 'CONSENT-MODAL-RISK' when a raw-port/launch path
runs while the relay was available. Lets us tell a code regression from Chrome's
own extension-debugger consent UX when the modal reappears. Best-effort, never
fails a connection. Verified: normal 'open' logs mode=relay (consent-free).
2026-06-15 16:47:12 +09:00
leeguooooo 6fcf52db60 chore(release): 1.5.4 — get text --pierce (closed shadow DOM, #30)
Release binaries / Build macOS ARM64 (push) Has been cancelled
Release binaries / Build macOS x64 (push) Has been cancelled
Release binaries / Build Linux ARM64 (push) Has been cancelled
Release binaries / Build Linux musl ARM64 (push) Has been cancelled
Release binaries / Build Linux musl x64 (push) Has been cancelled
Release binaries / Build Linux x64 (push) Has been cancelled
Release binaries / Build Windows x64 (push) Has been cancelled
Release binaries / Attach binaries to GitHub Release (push) Has been cancelled
2026-06-15 15:46:31 +09:00
leeguooooo af8823b27b feat(text): 'get text --pierce' reads through CLOSED shadow DOM (#30)
Some injected UI (browser-extension debug panels, web components) renders into a
CLOSED shadow root that eval/innerText cannot read. --pierce walks the CDP DOM
tree (DOM.getDocument depth:-1 pierce:true), which includes closed shadow roots
and child documents, and collects text nodes (skipping script/style/etc).

Review-safe: rides the per-tab debugger session already attached, no new Chrome
permission and no ab-connect/extension change — so it works in extension-relay
mode without touching the published extension. Verified live: a closed-shadow
panel that main-world eval reports HIDDEN is read in full via --pierce.

First slice of #30 (read extension/injected-panel content). Deeper extension
introspection (background SW / chrome.storage) stays a launch-mode / raw-CDP
concern, deliberately NOT done by expanding ab-connect's debugger powers.
2026-06-15 15:46:23 +09:00
leeguooooo c85e3faa82 chore(release): 1.5.3 — get text defaults to cross-frame; #29 (sessions/did-you-mean/tab liveness); CI changelog fix
Release binaries / Build macOS ARM64 (push) Has been cancelled
Release binaries / Build macOS x64 (push) Has been cancelled
Release binaries / Build Linux ARM64 (push) Has been cancelled
Release binaries / Build Linux musl ARM64 (push) Has been cancelled
Release binaries / Build Linux musl x64 (push) Has been cancelled
Release binaries / Build Linux x64 (push) Has been cancelled
Release binaries / Build Windows x64 (push) Has been cancelled
Release binaries / Attach binaries to GitHub Release (push) Has been cancelled
2026-06-15 13:42:58 +09:00
leeguooooo b25958946c feat(text): 'get text' (no selector) defaults to cross-frame whole-page read
So an agent never silently misses iframed content (listing descriptions etc.)
without having to know the --all-frames flag. Single-frame pages are unchanged
(identical to the old body read); multi-frame pages now include child frames —
a strict superset. Skill + help updated to make the default and 'frames'/--main
discoverable.
2026-06-15 13:42:48 +09:00
leeguooooo d4ff49caa8 ci(release): don't let an empty changelog section abort the release (bash -e)
The changelog step runs under 'bash -e'. section() returned non-zero when a
commit category was empty (grep no-match / empty [ -n ] test), aborting the
script before the closing heredoc delimiter — so any release whose range lacked
a whole category (e.g. 1.5.2: only feat, no fix) failed to attach binaries.
Add '|| true' + 'return 0' so section() always succeeds.
2026-06-15 13:42:48 +09:00
leeguooooo 4e949fffbf chore(release): 1.5.2 — sessions command + did-you-mean + honest tab-switch liveness (#29) 2026-06-15 13:34:26 +09:00
leeguooooo af46490812 feat(cli): sessions command + 'did you mean' suggestions + honest tab-switch liveness (#29)
- chrome-use sessions: top-level alias for the daemon inventory (the skill
  advertises sessions, so it's a natural guess that used to error).
- Unknown commands now suggest the nearest valid one (Levenshtein + prefix
  match), staying silent when nothing is close (e.g. 'clik' -> click,
  'sesions' -> sessions, 'xyzzy' -> no suggestion).
- tab <id>: probe the switched session and show a warning indicator instead of
  a green check when it isn't responding yet, so a switch onto a re-attaching
  (churned-tabId) session no longer reports false success. The #24 targetId
  recovery self-heals within ~6s, hence a warning rather than a hard error.
2026-06-15 13:25:16 +09:00
leeguooooo 57c52d6517 chore(release): 1.5.1 — frame-aware text extraction (get text --all-frames/--main, frames; #27) + ab-connect 0.4.9 targetId recovery (#24)
Release binaries / Build macOS ARM64 (push) Has been cancelled
Release binaries / Build macOS x64 (push) Has been cancelled
Release binaries / Build Linux ARM64 (push) Has been cancelled
Release binaries / Build Linux musl ARM64 (push) Has been cancelled
Release binaries / Build Linux musl x64 (push) Has been cancelled
Release binaries / Build Linux x64 (push) Has been cancelled
Release binaries / Build Windows x64 (push) Has been cancelled
Release binaries / Attach binaries to GitHub Release (push) Has been cancelled
2026-06-15 13:15:47 +09:00
leeguooooo 2707ceb1c4 feat(text): frame-aware text extraction — get text --all-frames / --main + frames (#27)
On listing/marketplace pages (Yahoo Auctions, Rakuten, Mercari shops) the
seller's description lives in a child frame or under a related-items sidebar,
so 'get text body' returned only header/nav boilerplate.

- get text --all-frames: aggregate visible text across every reachable frame.
  Same-process child frames are read via Page.createIsolatedWorld; OOPIFs via
  their auto-attached debugger session (iframe_sessions). Each non-top frame is
  labelled with a '----- frame [kind] url -----' separator.
- get text --main: readability-lite — prefer the densest <main>/<article>
  region over the whole body, dropping global header/nav/footer chrome.
- frames: enumerate frames (kind + url + per-frame text length) so an agent can
  see where a page's text actually lives and pick the right read.

Verified live: inline srcdoc frame text aggregated through --all-frames; Yahoo
Auctions <main> (2881 chars) extracted via --main, stripping the Yahoo header.
2026-06-15 12:51:59 +09:00
leeguooooo f7a657ac46 ci(release): group changelog by type ( Features / 🐛 Fixes / 🔧 Other)
Release notes were a flat list of commit subjects — hard to tell at a glance what
was added vs fixed (recurring '看不出改了什么'). Group by conventional-commit type
so every future release auto-shows scannable Features/Fixes sections.
2026-06-15 12:17:38 +09:00
leeguooooo 4e295ce139 fix(ab-connect): recover a churned-tabId session by stable CDP targetId (0.4.9, #24)
Live-reproduced #24 on 0.4.8 driving the Mercari signin token-exchange hop
(login.jp.mercari.com): the cross-process nav gives the tab a NEW Chrome tabId
while the CDP targetId stays the same. So cb-tab-<oldTabId> can't be recovered —
recoverSessionTab parsed the old tabId, chrome.tabs.get(oldTabId) failed (gone),
and it gave up → permanent 'stale sessionId ... its tab is gone' until the page
settled ~6s later and something re-attached. current/tab <targetId>/daemon
restart all failed because the relay still mapped the targetId to the dead
session.

Fix: remember each session's targetId across detach (sessionTargets map). When
recoverSessionTab can't recover by the encoded tabId, fall back to the STABLE
targetId — chrome.debugger.getTargets() to find the tab now hosting that target,
attach it, and ALIAS the dead cb-tab-<oldTabId> session to the live tab so the
daemon's session id keeps resolving. Longer retry window (~6s) since this hop
takes seconds to settle. Builds on 0.4.6/0.4.8 reattach; covers the tabId-churn
case those missed.

Needs dogfood on the real Mercari flow (can't repro the tabId churn synthetically).
2026-06-15 12:08:32 +09:00
leeguooooo 3d82f11ff2 chore(release): 1.5.0 — text-selector click + get text→body + tab --activate + click --follow/openedTab (#24); fill fires input/change/blur (#25); close <tab> wording + chrome-use current (#26)
Release binaries / Build macOS ARM64 (push) Has been cancelled
Release binaries / Build macOS x64 (push) Has been cancelled
Release binaries / Build Linux ARM64 (push) Has been cancelled
Release binaries / Build Linux musl ARM64 (push) Has been cancelled
Release binaries / Build Linux musl x64 (push) Has been cancelled
Release binaries / Build Linux x64 (push) Has been cancelled
Release binaries / Build Windows x64 (push) Has been cancelled
Release binaries / Attach binaries to GitHub Release (push) Has been cancelled
2026-06-15 11:32:30 +09:00
leeguooooo 33269adc1a fix(fill/tabs): dispatch real input/change/blur (#25); close <tab> wording + chrome-use current (#26)
#25 — fill() didn't fire the events framework inputs / site autocomplete need:
it set value directly (bypassing React's value-tracker) and typed via
Input.insertText, so controlled components and input/change/blur listeners (e.g.
Mercari's postal-code → 都道府県 lookup) never ran though the value showed. fill
now emulates a real edit: focus, set through the element's prototype value setter
(React _valueTracker registers), then dispatch input → input → change → blur/
focusout. SELECT and contenteditable handled too. type <sel> <text> remains for
per-keystroke sites. Verified live: an input wired with input/change/blur fired
'IICB' from one fill.

#26 (ergonomics):
- 'close <tab>' now closes just that tab and prints 'Tab [tN] closed'; bare
  'close' still closes the browser. Previously 'close t12' ran a browser close
  and alarmingly printed 'Browser closed'.
- new 'chrome-use current': prints the active tab's stable handle (tabId + CDP
  targetId + url/title), refreshed live — so an agent holds the targetId (which
  survives cross-process nav) instead of re-deriving 'which tab is live' from
  'tabs' every step. The deeper tab-id churn is the #21/#23 stable-targetId story.

Tests cover fill events (live), close tab-vs-browser parse, and current.
2026-06-15 11:26:34 +09:00
leeguooooo 9ab8753b48 feat(click): report (and optionally --follow) a tab opened by a click (#24-A)
A click on a target=_blank link / window.open opened a new tab, but the active
tab stayed put, so the post-click snapshot showed the OLD page — looking exactly
like the click failed. On the relay the new tab is discovered only via getTargets
(the relay doesn't push target events to the daemon), so it went unsurfaced.

handle_click now snapshots tracked targets before the click and, after, runs a
lightweight BrowserManager::adopt_newly_opened (one getTargets, attaches only the
new target — far cheaper than a full resync) to detect a freshly-opened tab. It's
reported as openedTab {tabId,url,title} in the response (and a '→ opened new tab
[tN] <url>' hint in text mode). Default keeps focus on the current tab (so
multi-tab flows aren't hijacked, per #7/#8.1); 'click <sel> --follow' switches to
the new tab. Verified live: clicking a _blank link prints
'→ opened new tab [t13] https://example.org/'.

Completes the #24 friction items (B/C/D shipped in 770708b).
2026-06-15 11:13:34 +09:00
leeguooooo 770708b8e6 fix(cli): text-selector click by visible label + get text→body + tab --activate (#24)
Three CLI gaps surfaced driving a Mercari signup→checkout flow:

- #24-B (correctness): a bare label like 'click 購入手続きへ' was fed straight to
  document.querySelector as CSS and failed as an invalid selector, even though
  snapshot listed the button by that exact name. build_find_element_js now tries
  CSS first, then falls back to matching an interactive element by visible text
  (exact then contains) — nested and non-ASCII labels resolve. 'text=<label>'
  forces the text path. CSS still wins when it matches.
- #24-D: 'get text' with no selector now returns the whole page (body).
- #24-C: 'tab <ref> --activate' (alias --front) switches to the tab AND raises it
  to the foreground — to surface a specific tab for the human.

Tests cover the text fallback / text= / xpath builder, body default, activate
flag. The core stale-sessionId-after-cross-process-nav bug is the #20/#23 class,
already fixed in ext 0.4.8 — needs that extension deployed.
2026-06-15 11:00:49 +09:00
leeguooooo 7c594820da docs(stream): document the bidirectional WS as the real-time driving path
Dogfooding (driving a canvas game) showed the slow, low-fidelity way — one
screenshot + one CLI call per action — when chrome-use already ships the right
tool: the session WebSocket is BIDIRECTIONAL. It streams ~60fps screencast frames
AND accepts input_keyboard/input_mouse/input_touch on the same socket, straight
to CDP Input.dispatch* — verified live over the extension relay (217 frames in
3.4s, ~64fps, and the input drove the game). But the inbound input protocol was
undocumented, so agents default to the CLI-per-action grind.

Document it in --help (stream) and the core skill: the frame + input message
schemas and the 'connect once, read frames, send timed input' loop, with a node
snippet. Reserve screenshots for one-off checks; use the WS for sustained
real-time control.
2026-06-14 00:54:30 +09:00
leeguooooo 81d18bbd2e feat(input): press --hold <ms> for precise timed key-holds + document timed-driving pattern
Dogfooding by driving a canvas game surfaced that per-action shell round-trips
(keydown; sleep; keyup) are the slowest, lowest-fidelity way to drive anything
timed — each is a process spawn + relay round-trip with ~250ms jitter, so a
'0.8s hold' is anything but.

- 'press <key> --hold <ms>': keyDown, wait, keyUp all inside the daemon, so the
  hold duration is precise and it's one round-trip. For games (hold-to-move/
  charge) and any press-and-hold.
- Documented the real driving pattern in the core skill + --help: script a timed
  sequence in ONE round-trip with 'batch "press d --hold 900" "press j" "wait 200"'
  (batch sends each step to the running daemon; --hold/wait block in-daemon), and
  prefer reading engine state via main-world 'eval' over guessing from pixels.

Parser test covers plain/held/missing-duration. Builds on the keydown/keyup full
descriptor fix.
2026-06-14 00:43:52 +09:00
leeguooooo d99a223d23 chore(release): 1.4.1 — hold-to-move (keydown/keyup full descriptor, #game) + expects ab-connect 0.4.8 (#23 reattach hardening)
Release binaries / Build macOS ARM64 (push) Has been cancelled
Release binaries / Build macOS x64 (push) Has been cancelled
Release binaries / Build Linux ARM64 (push) Has been cancelled
Release binaries / Build Linux musl ARM64 (push) Has been cancelled
Release binaries / Build Linux musl x64 (push) Has been cancelled
Release binaries / Build Linux x64 (push) Has been cancelled
Release binaries / Build Windows x64 (push) Has been cancelled
Release binaries / Attach binaries to GitHub Release (push) Has been cancelled
2026-06-14 00:04:08 +09:00
leeguooooo 4e7e80a596 fix(ab-connect): bind to stable tabId as the primary key + retry on mid-flight detach (0.4.8, #23)
claude-in-chrome completes the Rakuten cart→購入手続き→checkout flow that
chrome-use 1.2.3 couldn't, because it binds to the browser-level tabId (survives
renderer-process swaps) rather than a CDP target/sessionId (torn down by the
cross-origin OAuth/SSO nav). chrome-use's relay is already keyed to the stable
tabId (cb-tab-<tabId>, #17) and sends commands by {tabId} — the gap was purely
that the extension treated the session→tab map as the source of truth and only
reactively re-attached after a failed lookup.

Make the tabId the PRIMARY resolution path: derive it straight from the session
id (tabIdFromSession), ensure-attach with short retries across the swap window
(recoverSessionTab now loops), and route every send through sendCdpToTab, which
on a detached-style error drops the stale handle, re-attaches the stable tab, and
retries once. So a cross-process nav never surfaces as a hard error — there's no
'session gone' window, matching claude-in-chrome. Builds on 0.4.5/0.4.6 reattach;
makes it primary + bulletproof rather than a fallback.
2026-06-14 00:02:22 +09:00
leeguooooo 9bf79a4242 fix(keyboard): keydown/keyup send full key descriptor so hold-to-move works
`keydown`/`keyup` dispatched a minimal Input.dispatchKeyEvent carrying only
{key}, so games/handlers that read event.code ("KeyD", "ArrowRight") or
event.keyCode saw nothing — a held key set no movement flag and the player
barely moved (dogfood: Dead Cell). They now build the same descriptor `press`
uses (key + code + windows/nativeVirtualKeyCode + printable text on down) via a
shared interaction::dispatch_single_key. Verified live: holding a direction now
drives continuous movement (player ran into an enemy and took damage), where
before it nudged ~80px.
2026-06-13 23:58:30 +09:00
leeguooooo 5b4ffdb2bb chore(release): 1.4.0 — no-hijack open + tab adopt-by-targetId + keydown/keyup docs + canvas hint + all-component version coherence (doctor)
Release binaries / Build macOS ARM64 (push) Has been cancelled
Release binaries / Build macOS x64 (push) Has been cancelled
Release binaries / Build Linux ARM64 (push) Has been cancelled
Release binaries / Build Linux musl ARM64 (push) Has been cancelled
Release binaries / Build Linux musl x64 (push) Has been cancelled
Release binaries / Build Linux x64 (push) Has been cancelled
Release binaries / Build Windows x64 (push) Has been cancelled
Release binaries / Attach binaries to GitHub Release (push) Has been cancelled
2026-06-13 23:45:30 +09:00
leeguooooo 62e7229b47 feat(version): extension reports its version; doctor shows all-component coherence
The upgrade story spanned four parts (CLI, daemon, extension, skill) with no
single view and — worst — the extension was a total black box: nothing reported
which build was live, so a user could sit on a stale extension with zero signal.

- ext (0.4.7): on connect the extension sends a `hello` with
  chrome.runtime.getManifest().version; the native-messaging host records it to a
  `relay-ext-version` sidecar (next to relay-cdp-url, removed on exit).
- build.rs embeds the shipped extension version (AB_CONNECT_VERSION, read from the
  ext manifest at compile time) so the CLI knows what extension it expects.
- `chrome-use doctor` gains a Versions section: CLI (vs the cached latest from the
  background update check), extension (connected version vs the bundled expected —
  warns + tells you to reload it in Chrome if behind), and skill (bundled, version-
  locked; `skills add` copies may be stale). Daemon coherence was already covered.

So 'which of the four parts is on what version, and what needs upgrading' is now
one command. Verified: doctor warns on a simulated old extension and passes on a
current one; gracefully shows 'not connected / predates reporting' when the host
hasn't learned a version yet.
2026-06-13 23:41:16 +09:00
leeguooooo 23ab4ce68f fix(relay): don't hijack a user tab on open; surface keydown/keyup + canvas hint
Dogfooding a canvas game over the extension relay surfaced three issues:

1. (serious) A fresh relay session's first `open` navigated one of the USER's
   existing tabs instead of opening its own — in testing it replaced a
   half-filled form with the target site. On connect the daemon passively
   attaches to the user's tabs and pinned one as active; navigate() then drove
   it. Now: on the relay (agent_group set), if the active tab isn't one this
   session created, navigate() opens its own tab in the session's group first.
   Off the relay (a browser we launched) reusing the active tab stays correct.
   Pure helper active_index_is_owned() + regression tests.

2. (discoverability) `keydown <key>` / `keyup <key>` (hold-to-move, essential
   for games/shortcuts) already existed as commands+daemon handlers but were
   absent from --help and the skill, so they were undiscoverable. Documented in
   --help, the core skill, and the canvas-app hint.

3. (UX) Canvas/WebGL pages expose almost no a11y tree, so `snapshot` is empty
   and agents get stuck hunting refs. snapshot now detects a viewport-dominating
   canvas with a sparse tree and prints a hint pointing at the screenshot +
   coordinate-click + keydown/keyup path.

Verified live over the relay: `open` now lands the game in its own new tab with
the user's tabs (incl. the Rakuten recovery form) untouched; the canvas hint
fires on the game page; `close` cleans up only the session's own tab.
2026-06-13 23:27:17 +09:00
leeguooooo e272546b5c chore(release): 1.3.0 — daemon restart/status (#20.2) + live tab resync, adopt-by-targetId, open --reuse-tab (#21)
Release binaries / Build macOS ARM64 (push) Has been cancelled
Release binaries / Build macOS x64 (push) Has been cancelled
Release binaries / Build Linux ARM64 (push) Has been cancelled
Release binaries / Build Linux musl ARM64 (push) Has been cancelled
Release binaries / Build Linux musl x64 (push) Has been cancelled
Release binaries / Build Linux x64 (push) Has been cancelled
Release binaries / Build Windows x64 (push) Has been cancelled
Release binaries / Attach binaries to GitHub Release (push) Has been cancelled
2026-06-13 17:16:49 +09:00
leeguooooo c7de19b099 feat(tabs): live tab resync + adopt-by-targetId + open --reuse-tab (#21)
Multi-session over one relayed Chrome had a tab-identity fracture: each daemon
discovered targets ONCE at connect and assigned its own t<N> indices, so a tab
filled in session A was unreachable from session B — B saw a disjoint/blank set
and rebinding via 'open' piled up duplicate tabs. A stranded, still-filled tab
could not be finished from any other session.

- 'tab list' now re-syncs the live target set on every call: adopts tabs other
  sessions opened (or that re-attached after a cross-process nav), drops gone
  ones (clears phantom rows), and refreshes url/title from each live tab via
  Target.getTargetInfo (the relay only stamps target_info on attach, so it goes
  stale/blank after navigation — which made rows indistinguishable).
- 'tab list --full' now prints each tab's stable CDP targetId. Unlike t<N>
  (per-session, reassigned each connect), targetId is stable across every session
  on the relayed Chrome.
- 'tab <targetId>' adopts a specific pre-existing tab — including another
  session's — WITHOUT reloading, so a half-filled form survives. handle_tab_switch
  resyncs first, then resolves a raw targetId before falling back to t<N>/label.
- 'open <url> --reuse-tab' (alias --reuse) switches to an existing tab already on
  that URL (matched by origin+path, ignoring volatile query/fragment) instead of
  spawning a duplicate.

Verified live over the extension relay: a fresh session's 'tab list --full' lists
the user's real tabs with correct titles + full URLs + targetIds, and
'tab <targetId>' lands on and reads the exact stranded Rakuten account-recovery
form from the report. Unit tests cover URL normalization + --reuse-tab parsing;
full suite green. Docs: --help Tabs section + core skill multi-session guidance.
2026-06-13 17:11:38 +09:00
leeguooooo 6b9de10c73 fix(ab-connect): transparently re-attach a stale cb-tab session before failing (0.4.6, #20.1)
When a tab navigates across render processes (e.g. an SSO redirect to another
origin like login.account.rakuten.com), the debugger handle detaches and the
session drops out of the relay maps, so the next command dead-ends with
'stale sessionId ... its tab is gone' — even open/navigate, which should always
be able to drive the tab. But cb-tab-<tabId> encodes the STABLE Chrome tabId
(#17), and the tab itself usually survives the nav.

So before throwing, recoverSessionTab() parses the tabId out of the session,
checks the tab still exists + is eligible, and re-attaches (attachTab re-mints
the identical cb-tab-<tabId> session, keeping the daemon's binding valid), then
the in-flight command retries against the recovered tab. Complements the 0.4.5
onDetach proactive re-attach: that heals on the detach event, this heals lazily
on the next command if the event was missed. Falls back to the original error
only when the tab is genuinely gone (closed/restricted).
2026-06-13 16:50:07 +09:00
leeguooooo 63e0dd5921 docs(skill): document single-session relay limit — no cross-session tab reads (#20.3) 2026-06-13 16:43:21 +09:00
leeguooooo c0ee65d0d8 feat(cli): add 'chrome-use daemon restart|status' to reset stuck session state
A mid-session 'chrome-use upgrade' (or a crashed worker) can leave per-session
daemons holding stale/cross-leaked tab handles, and the only fix was hunting
PIDs with pgrep/kill. Add a first-class command:

- 'daemon restart' kills every session daemon worker (SIGTERM→SIGKILL +
  sidecar cleanup) but leaves the Chrome-launched __nm-host bridge alone, so
  the extension relay stays up — the next command spins a fresh, clean daemon
  against the same live Chrome. Closes no tabs.
- 'daemon status' lists running session daemons (pid + version) and relay state.

Wires connection::restart_all_daemons(), skips the command in the update-notify
nag, documents it in --help and the core skill. Unit tests cover the empty case
and a live-session kill (spawns a real child, asserts it's reaped + sidecars
cleaned). Issue #20.
2026-06-13 16:42:53 +09:00
leeguooooo 7601919a04 fix(ab-connect): auto-reattach on cross-process detach (Rakuten SSO #19 follow-up)
v1.2.3's stable per-tab session id (#17) fixed sessionId STABILITY, but nothing
re-attached when an origin swaps the render process (e.g. the
login.account.rakuten.com SSO redirect — full-page nav + OOPIF). chrome.debugger
detached, the tab survived, but only onUpdated('complete') could re-attach — and
for that flow it didn't, so the session went permanently stale (even
open/navigate failed, retries didn't recover).

onDetach now proactively re-attaches the surviving tab (retry w/ backoff for the
swapped-in process to settle; skips user/DevTools-initiated detaches), so the
stable cb-tab-<tabId> session is restored and commands self-heal. Extension
0.4.4 → 0.4.5; needs a Web Store republish + dogfood on the Rakuten flow.
2026-06-13 16:14:15 +09:00
leeguooooo 345c0d62a2 ci(release): checkout repo in the release job so the changelog isn't empty
The changelog step lived in the separate `release` job (needs: build), which
had no checkout — so git ran with no repo ('fatal: not a git repository') and
the body came out empty. Add a fetch-depth:0 checkout to that job; drop the
now-pointless fetch-depth:0 from the build job.
2026-06-13 15:55:46 +09:00
leeguooooo 0644fb2d0b chore(release): 1.2.3 — bringToFront command + tab list --full untruncated URLs (#19)
Release binaries / Build macOS ARM64 (push) Has been cancelled
Release binaries / Build macOS x64 (push) Has been cancelled
Release binaries / Build Linux ARM64 (push) Has been cancelled
Release binaries / Build Linux musl ARM64 (push) Has been cancelled
Release binaries / Build Linux musl x64 (push) Has been cancelled
Release binaries / Build Linux x64 (push) Has been cancelled
Release binaries / Build Windows x64 (push) Has been cancelled
Release binaries / Attach binaries to GitHub Release (push) Has been cancelled
2026-06-13 15:45:03 +09:00
leeguooooo 1ea6b1a2c5 fix(cli): add bringToFront command + tab list --full untruncated URLs (issue #19)
Two SPA-SSO debugging gaps:
- The core skill referenced `bringToFront` but the CLI parser never mapped it
  (the daemon handler existed) → 'Unknown command'. Wire up
  bringToFront / bring-to-front / bringtofront → the existing action.
- 'stale sessionId — re-open your target URL' recovery was impossible because
  `tab list` truncates long URLs with '…', cutting client_id/state out of SSO
  links. Add `tab list --full` (also `tab --full`) to print untruncated URLs;
  SKILL.md documents the recovery (full URL + re-open the stable entry URL).

The stale-session itself auto-recovers via the stable per-tab relay session id
(#17, extension 0.4.4). Parse tests for both new forms; verified live.
2026-06-13 15:44:25 +09:00
leeguooooo 7bb50d54b3 ci(release): fetch tags before building changelog (was empty)
v1.2.2's auto-changelog came out empty: in a detached-HEAD tag checkout the
tag refs git describe/git log need aren't reliably present even with
fetch-depth:0. Fetch them explicitly first.
2026-06-12 22:55:47 +09:00
leeguooooo e8864c96e2 chore(release): 1.2.2 — non-blocking 'update available' notice + release changelogs
Release binaries / Build macOS ARM64 (push) Has been cancelled
Release binaries / Build macOS x64 (push) Has been cancelled
Release binaries / Build Linux ARM64 (push) Has been cancelled
Release binaries / Build Linux musl ARM64 (push) Has been cancelled
Release binaries / Build Linux musl x64 (push) Has been cancelled
Release binaries / Build Linux x64 (push) Has been cancelled
Release binaries / Build Windows x64 (push) Has been cancelled
Release binaries / Attach binaries to GitHub Release (push) Has been cancelled
- feat(cli): non-blocking update-available notice (stderr, once/day, opt-out) so users learn to upgrade
- ci(release): auto-generate changelog from commit log on every release
2026-06-12 22:47:03 +09:00
leeguooooo 8432f6cd69 feat(cli): non-blocking 'update available' notice so users know to upgrade
The CLI ships as a GitHub Release binary with a manual `chrome-use upgrade`,
but nothing told users a newer version existed — so releases didn't reach them.

Add a lightweight update check: each run reads a cached latest-version and, if
it's newer than the running binary, prints a one-line hint to STDERR (never
stdout, so --json stays clean): "⚠ chrome-use X.Y.Z is available — run
chrome-use upgrade". The cache is refreshed at most once a day by a DETACHED
`__update-check` worker (curl → GitHub latest release), so the current command
never waits on the network. Skipped for meta commands (upgrade/install/doctor/
__*/--version/--help), in CI, in daemon mode, and via
CHROME_USE_NO_UPDATE_CHECK / AGENT_BROWSER_NO_UPDATE_CHECK.

Verified: nag shows for a newer cached version, suppressed by the opt-out env +
on meta commands + when up-to-date; the detached worker writes the real latest
tag from the GitHub API.
2026-06-12 22:46:44 +09:00
leeguooooo a8089310a6 ci(release): build changelog from commit log (not PR-only notes)
GitHub's generate_release_notes only lists merged PRs — near-empty for this
commit-to-main repo, so releases still showed nothing. Render the
conventional-commit subjects since the previous tag instead, and full-clone
(fetch-depth:0) so the diff is available.
2026-06-12 18:18:11 +09:00
leeguooooo ab92d2590b ci(release): auto-generate release changelog (commits + merged PRs since last tag)
GitHub Releases had an empty body — you couldn't tell what changed between
versions. Add generate_release_notes:true so every release ships an
auto-generated changelog.
2026-06-12 18:15:18 +09:00
leeguooooo c1417c3c70 chore(release): 1.2.1 — relay tab-drift pin on open (#14/#18) + stable per-tab relay session (#17)
Release binaries / Build macOS ARM64 (push) Has been cancelled
Release binaries / Build macOS x64 (push) Has been cancelled
Release binaries / Build Linux ARM64 (push) Has been cancelled
Release binaries / Build Linux musl ARM64 (push) Has been cancelled
Release binaries / Build Linux musl x64 (push) Has been cancelled
Release binaries / Build Linux x64 (push) Has been cancelled
Release binaries / Build Windows x64 (push) Has been cancelled
Release binaries / Attach binaries to GitHub Release (push) Has been cancelled
- fix(relay): pin active target on open so commands don't drift tabs (#14, #18)
- fix(connect): stable per-tab relay session id — re-attach auto-recovers (#17)
- docs: chrome-use test in README
2026-06-12 18:08:49 +09:00
郭立lee 7cb69bd444 fix(relay): pin active target on open so commands don't drift tabs (#14) (#18)
When connected to the user's real Chrome via the extension relay, sequential
commands could land on the wrong tab: `get url` returned x.com/home, then with
no navigation in between `eval` executed against x.com/notifications — so it
read the wrong page and returned nothing.

Root cause: the session's anti-drift anchor is `active_target_id` (pinned by
stable target_id), documented to be set "on every explicit open". But `open`
runs through `navigate()`, which never pinned. On the relay path `open` reuses
an existing tab via `navigate` rather than `add_page` (the only "explicit" path
that pins), so `active_target_id` stayed `None` and the session rode the fragile
`active_page_index`. A later passive tab close/reorder (drained before every
command) then drifted `eval`/`get url`/`snapshot` onto a foreign tab.

Fix:
- `navigate()` now syncs the index to the resolved active page and pins it by
  target_id after a successful navigation — restoring the "pin on explicit open"
  invariant for the relay path.
- `ensure_page()` pins its freshly-created tab too (matches `add_page`).
- Extract the pin-vs-index resolution into a pure `resolve_active_index()` and
  cover the invariant with unit tests (pin beats stale index; falls back when
  the pin is gone; survives passive background-tab discovery).

cargo fmt + clippy -D warnings clean; full suite 816 passed.
2026-06-12 17:02:21 +08:00
leeguooooo fb27835ebc fix(connect): stable per-tab relay session id — re-attach auto-recovers (#17)
When a tab's chrome.debugger session was torn down and re-established
(cross-process navigation, MV3 service-worker restart wiping the in-memory
maps, DevTools stealing the debugger), the extension minted a brand-new
monotonic `cb-tab-N` for the same tab. The daemon stays bound to the old id and
the relay consumes attach/detach events without telling it to rebind, so the
session was orphaned permanently → `stale sessionId / tab is gone`, and re-open
never recovered.

Derive the session id from the STABLE Chrome tabId (`cb-tab-<tabId>`) instead.
Any re-attach of the same tab now restores the SAME session the daemon already
holds, so eval/snapshot transparently follow the new page after a navigation.
Extension 0.4.3 → 0.4.4. Adds a relay unit test for the detach→reattach-same-
session recovery contract.
2026-06-12 17:43:48 +09:00
leeguooooo 2859da7b7c docs: document chrome-use test in README + point core skill at it 2026-06-12 17:33:24 +09:00
leeguooooo 9ba43e0cbd chore(release): 1.2.0 — chrome-use test (browser test suites)
Release binaries / Build macOS ARM64 (push) Has been cancelled
Release binaries / Build macOS x64 (push) Has been cancelled
Release binaries / Build Linux ARM64 (push) Has been cancelled
Release binaries / Build Linux musl ARM64 (push) Has been cancelled
Release binaries / Build Linux musl x64 (push) Has been cancelled
Release binaries / Build Linux x64 (push) Has been cancelled
Release binaries / Build Windows x64 (push) Has been cancelled
Release binaries / Attach binaries to GitHub Release (push) Has been cancelled
2026-06-12 17:27:38 +09:00
leeguooooo d740884299 feat(test): chrome-use test <suite.yaml> — re-runnable browser test suites
Turn repetitive browser checks into unit-test-style YAML suites. Steps reuse
chrome-use's own commands; assertions (url/visible/hidden/text/count/eval)
compile to a single truthy `eval`. The runner re-invokes the binary per step
(inherits all flag/launch/daemon/ref semantics; the daemon stays up so each
step is a fast socket call), launches an isolated browser by default, captures a
screenshot on failure, and exits non-zero for CI. `setup: account:` injects a
cookie-use login. Ships a `test` skill (skills get test). Unit-tested step/assert
compilation.
2026-06-12 17:27:38 +09:00
leeguooooo db484f2ac9 fix(cli): absolute screenshot path (#16) + show relay in session list (#15)
#16: handle_screenshot now returns a canonicalized ABSOLUTE path, so the
`✓ Screenshot saved to …` line is the same regardless of process cwd and the
agent can read the file without guessing the cwd.

#15: `session list` now reflects the extension-relay connection — when the relay
is up it shows the active session as `(relay/extension → live Chrome)` instead
of "No active sessions", and the --json output gains a `relay` bool. Stops agents
misjudging a live relay connection as down.
2026-06-12 16:43:39 +09:00
leeguooooo b475038e25 fix(upload): actionable error when file upload hits the extension-relay limit (#13)
DOM.setFileInputFiles is forbidden by Chrome's chrome.debugger API, so upload
always fails over the extension relay with an opaque -32000 "Not allowed". Map
it to a clear message: file upload needs a --launch/direct-CDP session, and
point at the cookies export|set --curl workaround. Note the limit in the core
skill upload line too.
2026-06-12 16:09:55 +09:00
leeguooooo 545e2545b4 fix(cookies): drop needless return in transfer arm (clippy -D warnings, CI red)
The cookies transfer arm's tail `return Ok(...)` tripped clippy::needless_return,
failing the CI lint gate (-D warnings). Make it a tail expression.
2026-06-12 15:49:16 +09:00
leeguooooo 5f342e34a2 docs(store): rewrite submission guide for rename-existing-item flow
The CWS rename to chrome-use updates the EXISTING item (knfcmbam…) with a
key-stripped package, not a new key-locked item — existing users auto-update and
reviews are kept. Native host allow_origins already lists both ids so the relay
doesn't break. Also point the icon/screenshots section at the generated assets.
2026-06-12 15:43:43 +09:00
leeguooooo 4106a151a1 feat(connect): install + recognize BOTH native-messaging host names (staged extension migration)
Stage 1 of the agent-browser → chrome-use extension migration: the CLI now
writes a host manifest under both com.agent_browser.connect (extension ≤0.4.2)
AND com.leeguoo.chrome_use (the rebrand 0.5.0+), both pointing at the same
launcher, and host_installed()/uninstall recognize both. So the relay works no
matter which extension version a user has, with no forced re-install — which
lets the store roll 0.4.3 (cosmetic name only, host unchanged) and later 0.5.0
(new host) without ever breaking the relay or re-popping the consent dialog.
2026-06-12 14:50:42 +09:00
leeguooooo eb60053183 fix(launch): serialize concurrent same-profile launches (issue #11)
N parallel `open --profile <same>` (e.g. chatgpt-imagegen's web backend firing
3 image gens at once) collided on the profile-copy disk I/O and Chrome's profile
lock: every candidate burned its full ~30s launch timeout and ALL failed (0
success), because the loser instances hung without writing DevToolsActivePort.

ProfileLaunchLock takes a cross-process flock on a per-resolved-profile lock
file, held across the copy + launch until Chrome is up, so concurrent
same-profile launches queue instead of colliding — the storm becomes
all-succeed-serially instead of all-fail. The kernel releases the lock when the
holder exits, so a crash can't wedge the queue; acquisition is best-effort
(launch proceeds unlocked if it can't be taken). Uncontended single launches
are unaffected.
2026-06-12 14:40:03 +09:00
leeguooooo 2aa216dd7a fix(config): brand-compat config dir (~/.chrome-use ⇄ ~/.agent-browser) so the relay survives the rename
After the agent-browser → chrome-use rename, the new binary used ~/.chrome-use
+ host com.leeguoo.chrome_use and couldn't find the relay that the still-old
native-messaging host wrote to ~/.agent-browser → it fell back to raw
--remote-debugging-port and re-popped 'Allow remote debugging?'.

- config_home()/config_dir_basename(): decide once per run — prefer the new
  .chrome-use, but keep using an existing .agent-browser install if that's the
  only one present; fresh installs get .chrome-use. get_socket_dir() routes
  through it so sockets/state are consistent within a run.
- relay_url_path(): the relay-cdp-url is a cross-binary handoff (host writes,
  CLI reads), so read from whichever brand dir actually has the file
  (~/.chrome-use OR ~/.agent-browser).

Combined with keeping HOST_NAME=com.agent_browser.connect (b6febbe), the renamed
chrome-use binary now relays through the existing ab-connect 0.4.2 extension
with zero dialog. Verified live: chrome-use found ~/.agent-browser/relay-cdp-url
and listed the user's real tabs, no consent dialog.
2026-06-12 14:29:21 +09:00
leeguooooo b6febbef39 fix(connect): keep native-messaging host as com.agent_browser.connect (no relay break)
Reverting the host-name rename from the chrome-use rebrand. The host name is
invisible internal plumbing (lives only in NativeMessagingHosts/*.json and the
extension), so renaming it to com.leeguoo.chrome_use bought nothing user-facing
but broke the relay for every existing user: the new chrome-use binary couldn't
find a matching host/extension, silently fell back to raw --remote-debugging-port,
and re-popped the 'Allow remote debugging?' consent dialog.

Keeping com.agent_browser.connect means the renamed chrome-use binary keeps
working with the already-installed host json and the live ab-connect 0.4.2
extension — zero relay break, no dialog, and the store republish becomes an
OPTIONAL cosmetic display-name update (manifest bumped 0.5.0 → 0.4.3, name stays
chrome-use). Only the binary/command name changed for users.
2026-06-12 14:07:10 +09:00
37 changed files with 6059 additions and 254 deletions
+55 -1
View File
@@ -116,6 +116,16 @@ jobs:
permissions:
contents: write
steps:
# The release job is separate from the build matrix and has no repo by
# default — check it out (full history + tags) so the changelog step has a
# git repo to diff. Without this, `git` failed with "not a git repository"
# and the changelog came out empty.
- name: Checkout
uses: actions/checkout@v6
with:
ref: ${{ github.event.inputs.tag || github.ref }}
fetch-depth: 0
- name: Download all artifacts
uses: actions/download-artifact@v8
with:
@@ -125,6 +135,47 @@ jobs:
- name: List assets
run: ls -la dist
# Build the changelog from conventional-commit subjects since the previous
# tag. GitHub's built-in generate_release_notes only lists merged PRs,
# which is near-empty for this commit-to-main workflow — so we render the
# commit log ourselves and every release shows what actually changed.
- name: Generate changelog
id: changelog
run: |
# fetch-depth:0 gets history, but the tag refs the changelog needs
# aren't always present in a detached-HEAD tag checkout — pull them in.
git fetch --tags --force --quiet origin 2>/dev/null || true
TAG="${{ github.event.inputs.tag || github.ref_name }}"
PREV="$(git describe --tags --abbrev=0 "${TAG}^" 2>/dev/null || true)"
RANGE="${TAG}"
[ -n "$PREV" ] && RANGE="${PREV}..${TAG}"
# Group commit subjects by conventional-commit type so the notes are
# scannable ("what's new / what's fixed") instead of a flat dev log.
LOG="$(git log "$RANGE" --no-merges --pretty='%s' | grep -v '^chore(release)' || true)"
# NOTE: the job runs under `bash -e`. grep returning 1 (no match) and
# the `[ -n "$body" ]` test returning 1 (empty section) must NOT abort
# the script — otherwise a release whose commit range lacks a whole
# category (e.g. only `feat`, no `fix`) dies before writing the closing
# heredoc delimiter and the whole release step fails. `|| true` +
# `return 0` keep section() always-succeeding.
section() { # $1=header $2=grep-pattern
local body; body="$(printf '%s\n' "$LOG" | grep -E "$2" | sed 's/^/- /' || true)"
[ -n "$body" ] && printf '\n### %s\n%s\n' "$1" "$body"
return 0
}
{
echo "notes<<__NOTES_EOF__"
echo "## What changed"
section "✨ Features" '^feat'
section "🐛 Fixes" '^fix'
section "🔧 Other" '^(perf|refactor|docs|build|ci|test|style|revert)'
if [ -n "$PREV" ]; then
echo ""
echo "**Full changelog**: https://github.com/${{ github.repository }}/compare/${PREV}...${TAG}"
fi
echo "__NOTES_EOF__"
} >> "$GITHUB_OUTPUT"
- name: Attach to release
uses: softprops/action-gh-release@v3
with:
@@ -133,5 +184,8 @@ jobs:
dist/*.tar.gz
dist/*.tar.gz.sha256
fail_on_unmatched_files: true
# keep existing release notes if the release was created beforehand
# The commit-based changelog so every release shows what changed. The
# first matrix job to run creates the release with these notes;
# append_body:false keeps later platform jobs from duplicating them.
body: ${{ steps.changelog.outputs.notes }}
append_body: false
+1
View File
@@ -75,3 +75,4 @@ out/
# extension signing key (never commit) + local-only id record
.secrets/
*.pem
/cu-test-artifacts
+88
View File
@@ -205,6 +205,91 @@ chrome-use --launch --profile auto open https://x.com/home
In CI environments, standalone mode is used automatically.
## Site adapters — turn a website into a structured-data CLI
Most "read GitHub issues" / "search Reddit" / "get my Bilibili feed" tasks don't
need clicking and screenshotting at all — the site already has a JSON API behind
its own login. A **site adapter** is a tiny JS function that calls that API *from
inside your logged-in tab* (your cookies, same-origin `fetch`, the site's own
modules) and returns clean JSON. The site can't tell it apart from you, because it
*is* you.
chrome-use ships none of these adapters — `site update` fetches the community
[**bb-sites**](https://github.com/epiral/bb-sites) pack at runtime (like a package
manager pulling a dependency), then runs them over chrome-use's stealth transport:
```bash
chrome-use site update # fetch the adapter pack (~145 commands)
chrome-use site list # github/issues, reddit/search, bilibili/feed, …
chrome-use site info github/issues # see an adapter's args + domain
# Run one — navigates to the site (reusing the tab if you're already there) and returns JSON
chrome-use site github/issues epiral/bb-browser --json
chrome-use site reddit/search "rust async" --json
chrome-use site bilibili/feed --json # works because it's your logged-in session
```
Positional args fill the adapter's declared args in order; `--key value` overrides
by name. Adapters are authored by the bb-sites community and remain their authors'
property — chrome-use just runs them.
**Auto-sync + auto-suggest.** You rarely type `site update` yourself: chrome-use
syncs the pack on first use and refreshes it weekly in the background (tune with
`AGENT_BROWSER_SITES_TTL_DAYS`, disable with `AGENT_BROWSER_SITES_NO_AUTO_UPDATE=1`).
And when you `open`/`snapshot` a page whose domain has adapters, chrome-use surfaces
them right in the output — a `💡 site adapters for <domain>` line, plus a
`siteAdapters` field under `--json` — so an agent reaches for the structured-data
adapter instead of scraping the DOM:
```text
$ chrome-use open https://github.com
💡 site adapters for github.com — prefer these for structured data:
github/issues, github/me, github/repo, …
e.g. chrome-use site github/issues --json
✓ GitHub
```
## Automated testing (`chrome-use test`)
Turn the repetitive "open it, click around, check it's right" work into a
**re-runnable suite** — unit tests for the frontend. Write cases in YAML; steps
reuse chrome-use's own commands and assertions compile to a single check:
```yaml
# smoke.yaml
suite: chatgpt smoke
setup:
- account: chatgpt/huayue # inject a cookie-use login (optional)
cases:
- name: home loads logged in
steps:
- open: https://chatgpt.com/
- wait: { load: networkidle }
assert:
- url: { contains: chatgpt.com }
- visible: "#prompt-textarea"
```
```bash
chrome-use test smoke.yaml # launches an isolated browser, runs cases
chrome-use test smoke.yaml --session default # …or against your connected Chrome
```
```
suite: chatgpt smoke (session cu-test)
✓ home loads logged in 1.2s
✗ composer takes text 0.8s
assert text "#prompt-textarea" contains "hi" → got ""
↳ cu-test-artifacts/composer-takes-text.png
2 cases · 1 passed · 1 failed
```
Exit code is non-zero if any case fails (drop it into CI), and failed cases save
a screenshot. Assertions: `url` · `visible` · `hidden` · `text` · `count` ·
`eval`. Steps: `open` · `click` · `fill` · `type` · `press` · `wait` · `scroll`
· `eval`. Full guide: `chrome-use skills get test`. Found a regression? Add a
case — the suite gets more valuable the more you use it.
## Anti-detection
<img src="assets/shield.png" alt="stealth shield" width="320" align="right" />
@@ -285,3 +370,6 @@ We deliberately **don't ship our own bot detector** — the strongest, most hone
## License
Apache-2.0
---
> Built by **leeguooooo** — field notes on AI agents, reverse engineering & Cloudflare Workers at **[blog.misonote.com](https://blog.misonote.com)** · follow on **[X @leeguooooo](https://x.com/leeguooooo)**
+138 -1
View File
@@ -63,6 +63,25 @@ chrome-use 让**任意** agentClaude Code、Cursor、Codex、你自己的脚
每个 `--session` 拿到**自己的彩色标签组**,多个 agent 共用同一个真实浏览器、互不干扰,也不动你自己的标签页。
## 为什么用扩展(而非裸调试端口)
其他本地工具走裸 `--remote-debugging-port`CDP)驱动 Chrome。从 **Chrome 136** 起,每次这样连接都会弹出一个阻塞式的 **"Allow remote debugging?"** 同意框 —— 而且端口得提前开好。我们的扩展改用原生消息:**装一次,之后零确认。**
| | **chrome-use**(本扩展) | web-access(裸 CDP 端口) | Claude in Chromechrome.debugger |
|---|---|---|---|
| 连接方式 | 原生消息 —— 无端口、无 token | `--remote-debugging-port` | `chrome.debugger` |
| **"Allow remote debugging?" 弹框** | **从不** ✅ | **每次连都弹** 🔴 | 无 |
| 复用你的真实登录 | 是 | 是 | 是 |
| `Runtime.enable`CDP)泄漏¹ | **默认关闭 → 干净** ✅ | 域已启用 | 不适用 |
| CreepJS 隐身分² | **0% stealth · 0% headless** ✅ | 真实 Chrome | 真实 Chrome |
| 每会话标签组 / 并发 agent | **支持** ✅ | 无 | 无 |
| 为 chrome-use CLI 打造 | 是 | 独立代理 | 单 app 助手 |
> ¹ 对 [rebrowser-bot-detector](https://bot-detector.rebrowser.net/) 实测:我们的中继报 `runtimeEnableLeak: 🟢 No leak`、`navigatorWebdriver: 🟢`。
> ² 对 [CreepJS](https://abrahamjuliot.github.io/creepjs/) 在「连接真实 Chrome」路径上实测 —— 见 [反检测](#反检测)。
>
> 同意框不是假想:裸端口工具**每次** attach 都会弹(Chrome 136+ 安全策略)。扩展路径从不弹。
## 安装
```bash
@@ -71,6 +90,15 @@ curl -fsSL https://raw.githubusercontent.com/leeguooooo/chrome-use/main/install.
从最新的 [GitHub Release](https://github.com/leeguooooo/chrome-use/releases) 下载对应平台的预编译二进制,安装 `chrome-use`(以及 `abs` 别名)。无需 npm,无需 token。
<details>
<summary>其他安装方式</summary>
- **锁定版本:** `AGENT_BROWSER_VERSION=v0.27.0-fork.12 curl -fsSL https://raw.githubusercontent.com/leeguooooo/chrome-use/main/install.sh | sh`
- **自定义路径:** `AGENT_BROWSER_BIN_DIR=$HOME/bin curl -fsSL … | sh`
- **Windows** 从 [Releases 页](https://github.com/leeguooooo/chrome-use/releases) 下载 `chrome-use-win32-x64.tar.gz`,把 `chrome-use.exe` 放进 PATH。
- **npm(旧渠道):** `npm install -g chrome-use` —— 仍在发布,但 GitHub Releases 现在是主渠道。
</details>
### 安装 AI agent skills
```bash
@@ -79,6 +107,10 @@ npx skills add leeguooooo/chrome-use
`skills/chrome-use` 拉进当前项目,让你的 AI agent 拿到正确的用法和预授权的 bash 权限。
## 命令名
`chrome-use``chrome-use``abs` 是**同一个二进制** —— `abs` 只是短别名。没有单独的「隐身可执行文件」;隐身是**运行时行为**(见下方 [反检测](#反检测)),根据你是连接真实 Chrome 还是 `--launch` 全新实例自动启用。
## 连接你的 Chrome
**推荐 —— 浏览器扩展(一键,无弹窗)。** 从 Chrome 应用商店安装 [**chrome-use** 扩展](https://chromewebstore.google.com/detail/chrome-use/knfcmbamhjmaonkfnjhldjedeobeafmk),再注册一次本地桥:
@@ -128,8 +160,83 @@ chrome-use --launch open https://example.com
# 保留登录:用你真实的 Chrome profile 启动
chrome-use --launch --profile auto open https://x.com/home
# 或显式指定:--profile Default / --profile "Profile 1"
```
## 站点适配器 —— 把一个网站变成「结构化数据 CLI」
大多数「读 GitHub issue」「搜 Reddit」「拉我的 B 站动态」这类任务,根本不需要点击 +
截图 —— 网站登录态背后本来就有 JSON 接口。**站点适配器**就是一小段 JS 函数,它在你
**已登录的标签页内**调用那个接口(用你的 cookie、同源 `fetch`、网站自己的模块),返回
干净的 JSON。网站分辨不出它和你的区别,因为它**就是你**。
chrome-use 本身不附带任何适配器 —— `site update` 会在运行时拉取社区的
[**bb-sites**](https://github.com/epiral/bb-sites) 适配器包(就像包管理器拉依赖),
然后在 chrome-use 的隐身通道上运行它们:
```bash
chrome-use site update # 拉取适配器包(约 145 条命令)
chrome-use site list # github/issues、reddit/search、bilibili/feed…
chrome-use site info github/issues # 查看某个适配器的参数 + 域名
# 运行一个 —— 会导航到对应站点(已在该站点则复用当前标签页)并返回 JSON
chrome-use site github/issues epiral/bb-browser --json
chrome-use site reddit/search "rust async" --json
chrome-use site bilibili/feed --json # 能用,因为走的是你的登录态
```
位置参数按适配器声明的参数顺序填入;`--key value` 按名覆盖。适配器由 bb-sites 社区编写、
版权归各自作者所有 —— chrome-use 只负责运行它们。
**自动同步 + 自动提示。** 你基本不用手动 `site update`:chrome-use 首次使用时自动拉取,
之后每周后台刷新一次(`AGENT_BROWSER_SITES_TTL_DAYS` 调周期,`AGENT_BROWSER_SITES_NO_AUTO_UPDATE=1`
关闭)。而当你 `open`/`snapshot` 一个有适配器的域名时,chrome-use 会在输出里直接把可用命令
亮出来 —— 一行 `💡 site adapters for <域名>`,`--json` 下则是 `siteAdapters` 字段 —— 这样
agent 会直接改用结构化适配器,而不是去扒 DOM:
```text
$ chrome-use open https://github.com
💡 site adapters for github.com — prefer these for structured data:
github/issues, github/me, github/repo, …
e.g. chrome-use site github/issues --json
✓ GitHub
```
## 自动化测试(`chrome-use test`
把反复的「打开它、点一圈、看对不对」变成**可重跑的测试套件** —— 前端的单元测试。用 YAML 写用例;步骤复用 chrome-use 自己的命令,断言编译成一次检查:
```yaml
# smoke.yaml
suite: chatgpt smoke
setup:
- account: chatgpt/huayue # 注入一个 cookie-use 登录(可选)
cases:
- name: home loads logged in
steps:
- open: https://chatgpt.com/
- wait: { load: networkidle }
assert:
- url: { contains: chatgpt.com }
- visible: "#prompt-textarea"
```
```bash
chrome-use test smoke.yaml # 启动隔离浏览器,跑用例
chrome-use test smoke.yaml --session default # …或对你已连接的 Chrome 跑
```
```
suite: chatgpt smoke (session cu-test)
✓ home loads logged in 1.2s
✗ composer takes text 0.8s
assert text "#prompt-textarea" contains "hi" → got ""
↳ cu-test-artifacts/composer-takes-text.png
2 cases · 1 passed · 1 failed
```
任一用例失败时退出码非零(可直接丢进 CI),失败用例会存截图。断言:`url` · `visible` · `hidden` · `text` · `count` · `eval`。步骤:`open` · `click` · `fill` · `type` · `press` · `wait` · `scroll` · `eval`。完整指南:`chrome-use skills get test`。发现回归?加个用例 —— 用得越多,套件越值钱。
## 反检测
连接你真实 Chrome 时,我们**零** JS 注入 —— 浏览器指纹完全是真的。指导原则是 **native CDP/Chrome 覆盖优先于 JS 谎言**:被重定义的 getter 本身可被检测,原生覆盖则不会。
@@ -146,7 +253,9 @@ chrome-use --launch --profile auto open https://x.com/home
| [rebrowser-bot-detector](https://bot-detector.rebrowser.net/) | `runtimeEnableLeak` 🟢 · `pwInitScripts` 🟢 |
| [bot.sannysoft.com](https://bot.sannysoft.com) | 全绿 |
`--launch` 独立模式下会改用一整套隐身补丁,同样过上述检测。
CreepJS 上的 `0% stealth` 是关键数字:因为连接路径**什么都不打补丁**,根本没有可供说谎检测器抓的 override。(读 `navigator.languages` 顺序或 IP 地理位置的面板可能给个软性的「navigator」/「location」标记 —— 那反映的是*你真实 Chrome* 的语言列表和网络,不是自动化破绽。)
`--launch` 独立模式(全新浏览器)会改用一整套隐身补丁,也能过上述检测 —— 唯一例外:CreepJS 报 **~20% stealth**,因为 srcdoc-iframe 的 `contentWindow` 补丁触发了它的 `hasIframeProxy` 探测(用来藏自动化的 proxy 本身成了破绽)。其余全干净(`0% headless`、sannysoft/browserscan 全绿、Cloudflare 通过)。设 **`AGENT_BROWSER_DISABLE_IFRAME_PROXY=1`** 去掉那个补丁即可拿到干净的 **0% stealth**(代价是放弃小众的 srcdoc-iframe 遮蔽)。**扩展连接路径**(你的真实 Chrome)零 JS 注入、不受影响 —— 它才是货真价实的 0% 路径。
### 类人输入(行为隐身)
@@ -167,6 +276,30 @@ chrome-use --launch --profile auto open https://x.com/home
操作你的真实 Chrome 不该打断你的工作。agent **全程在后台操作**:新标签后台打开(在自己的彩色会话标签组里),**从不强制把标签拽到前台**,并用 `Emulation.setFocusEmulationEnabled` 让每个 agent 标签照常渲染、`document.hasFocus()` / `visibilityState` 仍报 `visible`。于是截图正常、页面不被降频,"标签全程隐藏"也不会变成新的机器人信号。你在自己的标签里照常工作,agent 在旁边默默干活。(想置顶某个标签仍可显式调用命令。)
### 自己验证
别光听我们说 —— 把你连接的 Chrome 指向最硬的公开检测器,自己对比:
- **[CreepJS](https://abrahamjuliot.github.io/creepjs/)** —— 最全面的指纹 / 说谎检测器
- **[bot.incolumitas.com](https://bot.incolumitas.com/)** —— 行为 + 指纹打分,方法公开
- **[BrowserScan](https://www.browserscan.net/bot-detection)** —— Webdriver / User-Agent / CDP / Navigator
- **[bot.sannysoft.com](https://bot.sannysoft.com)** —— 经典自动化特征清单
- **[pixelscan.net](https://pixelscan.net/)** · **[iphey.com](https://iphey.com/)** —— 一致性与身份
我们故意**不自带 bot 检测器** —— 最强、最诚实的基准,就是拿市面上最好的检测器去测你的真实浏览器。
### 调参(环境变量)
| 变量 | 默认 | 作用 |
|---|---|---|
| `AGENT_BROWSER_CAPTURE_CONSOLE` | 关 | 启用 `Runtime` 域,让 `console` / `errors` 捕获页面输出。关闭可保持最隐身的画像。 |
| `AGENT_BROWSER_HUMANIZE` | 关 | 类人输入动作:`off`(瞬时)、`fast`(轻量缓动轨迹)、`human`(全套曲线轨迹 + 落点抖动 + 击键节奏 + 缓动滚动/拖拽)。也可用 `--humanize`。默认 `off`;自适应检测器会把 Akamai/PerimeterX/DataDome 守护的页面自动升到 `human`。 |
| `AGENT_BROWSER_TIMEZONE` | 未设 | 仅 `--launch`。IANA id(如 `Asia/Tokyo`)原生设置时区(Intl + Date 跟随,无 JS 谎言)以匹配代理;`auto` 按 locale 推导。 |
| `AGENT_BROWSER_BLOCK_WEBRTC` | auto | 仅 `--launch`。设了代理时自动强制 WebRTC 走代理(不泄漏真实 IP)。`1` 无代理时也隐藏本地 IP;`0` 退出。 |
| `AGENT_BROWSER_HIDE_CANVAS` | 关 | 仅 `--launch`。加入会话稳定的 canvas/audio 指纹噪声。默认关(噪声本身就是一种「谎言」)。 |
| `AGENT_BROWSER_ADAPTIVE_REF` | 开 | 当保存的 `@ref` 移动且 role/name 重查失败时,按指纹相似度重定位(需高分 + 明显领先,否则明确报错)。`0` 关闭。 |
| `AGENT_BROWSER_CLICK_MODE` | _(auto)_ | 点击策略。默认先滚动入视、派发坐标点击,若被浮层遮挡则回退 DOM `.click()``dom` 始终用 `.click()`(适合 blur 即关的自动补全/菜单项);`coord` 严格只用坐标(遮挡时硬失败)。 |
## chrome-use 的独特之处
- **默认 auto-connect** —— `chrome-use open` 连你现有的 Chrome 而非启新的
@@ -181,3 +314,7 @@ chrome-use --launch --profile auto open https://x.com/home
## License
Apache-2.0
---
> 由 **leeguooooo** 打造 —— AI agent、逆向工程与 Cloudflare Workers 的实战笔记见 **[blog.misonote.com](https://blog.misonote.com)** · 关注 **[X @leeguooooo](https://x.com/leeguooooo)**
+21 -1
View File
@@ -290,7 +290,7 @@ checksum = "613afe47fcd5fac7ccf1db93babcb082c5994d996f20b8b159f2ad1658eb5724"
[[package]]
name = "chrome-use"
version = "1.1.0"
version = "1.5.20"
dependencies = [
"aes",
"aes-gcm",
@@ -312,6 +312,7 @@ dependencies = [
"rust-embed",
"serde",
"serde_json",
"serde_yaml",
"sha1",
"sha2",
"similar",
@@ -1982,6 +1983,19 @@ dependencies = [
"serde",
]
[[package]]
name = "serde_yaml"
version = "0.9.34+deprecated"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "6a8b1a1a2ebf674015cc02edccce75287f1a0130d394307b36743c2f5d504b47"
dependencies = [
"indexmap",
"itoa",
"ryu",
"serde",
"unsafe-libyaml",
]
[[package]]
name = "sha1"
version = "0.10.6"
@@ -2419,6 +2433,12 @@ dependencies = [
"subtle",
]
[[package]]
name = "unsafe-libyaml"
version = "0.2.11"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "673aac59facbab8a9007c7f6108d11f63b603f7cabff99fabf650fea5c32b861"
[[package]]
name = "untrusted"
version = "0.9.0"
+2 -1
View File
@@ -1,6 +1,6 @@
[package]
name = "chrome-use"
version = "1.1.0"
version = "1.5.20"
edition = "2021"
description = "Fast browser automation CLI for AI agents"
license = "Apache-2.0"
@@ -45,6 +45,7 @@ sha1 = "0.10"
chrono = "0.4"
urlencoding = "2"
rust-embed = "8"
serde_yaml = "0.9"
[target.'cfg(unix)'.dependencies]
libc = "0.2"
+18
View File
@@ -3,6 +3,23 @@ use std::env;
use std::fs;
use std::path::Path;
/// Embed the version of the `ab-connect` extension this CLI ships alongside, so
/// `doctor` can tell a connected extension "you're older than what this CLI
/// expects, update it." Read from the extension manifest at build time so it
/// stays in sync with whatever extension version is in the same checkout/release
/// (the ext is on its own 0.4.x line, separate from the CLI version). Falls back
/// to "unknown" if the manifest can't be read.
fn embed_extension_version() {
let manifest = Path::new("../extensions/ab-connect/manifest.json");
println!("cargo:rerun-if-changed=../extensions/ab-connect/manifest.json");
let version = fs::read_to_string(manifest)
.ok()
.and_then(|s| serde_json::from_str::<serde_json::Value>(&s).ok())
.and_then(|v| v.get("version").and_then(|x| x.as_str()).map(String::from))
.unwrap_or_else(|| "unknown".to_string());
println!("cargo:rustc-env=AB_CONNECT_VERSION={}", version);
}
/// Ensure `packages/dashboard/out/` exists so `rust-embed` doesn't fail during
/// Rust-only dev builds where the dashboard hasn't been built. The placeholder
/// `index.html` is only written when the directory is completely absent.
@@ -20,6 +37,7 @@ fn ensure_dashboard_dir() {
fn main() {
ensure_dashboard_dir();
embed_extension_version();
let protocol_dir = Path::new("cdp-protocol");
let out_dir = env::var("OUT_DIR").unwrap();
+635 -49
View File
@@ -30,12 +30,105 @@ pub enum ParseError {
InvalidSessionName { name: String },
}
/// Top-level commands an agent is likely to mistype, used for "did you mean"
/// suggestions on an unknown command (issue #29). Not exhaustive — just the
/// common verbs plus a few known wrong-guesses mapped to the real command.
const KNOWN_COMMANDS: &[&str] = &[
"open",
"navigate",
"click",
"fill",
"type",
"press",
"snapshot",
"screenshot",
"eval",
"get",
"text",
"html",
"frames",
"find",
"wait",
"scroll",
"hover",
"select",
"check",
"uncheck",
"tab",
"tabs",
"close",
"back",
"forward",
"reload",
"sessions",
"status",
"daemon",
"doctor",
"upgrade",
"connect",
"cookies",
"mouse",
"keyboard",
"stream",
"frame",
"profiles",
"title",
"url",
"is",
"drag",
"dialog",
"upload",
"site",
];
/// Levenshtein distance, capped — small inputs only (command names).
fn edit_distance(a: &str, b: &str) -> usize {
let a: Vec<char> = a.chars().collect();
let b: Vec<char> = b.chars().collect();
let mut prev: Vec<usize> = (0..=b.len()).collect();
let mut curr = vec![0usize; b.len() + 1];
for (i, &ca) in a.iter().enumerate() {
curr[0] = i + 1;
for (j, &cb) in b.iter().enumerate() {
let cost = if ca == cb { 0 } else { 1 };
curr[j + 1] = (prev[j + 1] + 1).min(curr[j] + 1).min(prev[j] + cost);
}
std::mem::swap(&mut prev, &mut curr);
}
prev[b.len()]
}
/// Closest known command within a small edit distance, or a prefix/substring
/// match — `None` if nothing is close enough to suggest confidently.
fn nearest_command(input: &str) -> Option<String> {
let lower = input.to_lowercase();
// Exact prefix/substring hits first (e.g. "session" -> "sessions").
if let Some(c) = KNOWN_COMMANDS
.iter()
.find(|c| c.starts_with(&lower) || lower.starts_with(**c))
{
return Some(c.to_string());
}
// Tolerance scales with length: short words get distance 1, longer get 2.
let max_dist = if lower.len() <= 4 { 1 } else { 2 };
KNOWN_COMMANDS
.iter()
.map(|c| (*c, edit_distance(&lower, c)))
.filter(|(_, d)| *d <= max_dist)
.min_by_key(|(_, d)| *d)
.map(|(c, _)| c.to_string())
}
impl ParseError {
pub fn format(&self) -> String {
match self {
ParseError::UnknownCommand { command } => {
format!("Unknown command: {}", command)
}
ParseError::UnknownCommand { command } => match nearest_command(command) {
Some(suggestion) => format!(
"Unknown command: {}\nDid you mean: chrome-use {}?",
command, suggestion
),
None => format!("Unknown command: {}", command),
},
ParseError::UnknownSubcommand {
subcommand,
valid_options,
@@ -370,6 +463,12 @@ fn parse_command_inner(args: &[String], flags: &Flags) -> Result<Value, ParseErr
if flags.provider.is_some() {
nav_cmd["waitUntil"] = json!("none");
}
// `--reuse-tab`: adopt an existing tab already on this URL instead of
// navigating/spawning a new one (issue #21 — avoids duplicate tabs on
// rebind, preserves in-page state).
if rest.iter().any(|a| *a == "--reuse-tab" || *a == "--reuse") {
nav_cmd["reuseTab"] = json!(true);
}
// Explicit readiness override (issue #10): SPAs whose `load` event
// never fires (a long-lived XHR/websocket holds it open) hang out the
// load-event wait. `--wait-until domcontentloaded` returns as soon as
@@ -408,10 +507,19 @@ fn parse_command_inner(args: &[String], flags: &Flags) -> Result<Value, ParseErr
"back" => Ok(json!({ "id": id, "action": "back" })),
"forward" => Ok(json!({ "id": id, "action": "forward" })),
"reload" => Ok(json!({ "id": id, "action": "reload" })),
// Explicit opt-in to raise the active tab to the foreground (the core
// skill references it; the daemon handler existed but the CLI didn't map
// it — issue #19). Accept the documented camelCase + kebab/lowercase.
"bringToFront" | "bring-to-front" | "bringtofront" => {
Ok(json!({ "id": id, "action": "bringtofront" }))
}
// === Core Actions ===
"click" => {
let new_tab = rest.contains(&"--new-tab");
// `--follow`: if the click opens a new tab, switch the active tab to
// it (default reports the opened tab but stays put) (issue #24-A).
let follow = rest.contains(&"--follow");
// Coordinate click as a first-class form (issue #8.4): when the only
// handle is a pixel position, no element/selector is needed.
// click <x> <y> e.g. click 449 320
@@ -420,23 +528,27 @@ fn parse_command_inner(args: &[String], flags: &Flags) -> Result<Value, ParseErr
let coord_args: Vec<&str> = rest
.iter()
.copied()
.filter(|a| *a != "--new-tab" && *a != "--coords")
.filter(|a| !a.starts_with("--"))
.collect();
if let Some((x, y)) = parse_coords(&coord_args) {
return Ok(json!({ "id": id, "action": "click", "x": x, "y": y }));
}
let sel = rest
.iter()
.find(|arg| **arg != "--new-tab")
.find(|arg| !arg.starts_with("--"))
.ok_or_else(|| ParseError::MissingArguments {
context: "click".to_string(),
usage: "click <selector> | click <x> <y> | click --coords <x>,<y> [--new-tab]",
usage:
"click <selector> | click <x> <y> | click --coords <x>,<y> [--new-tab] [--follow]",
})?;
let mut cmd = json!({ "id": id, "action": "click", "selector": sel });
if new_tab {
Ok(json!({ "id": id, "action": "click", "selector": sel, "newTab": true }))
} else {
Ok(json!({ "id": id, "action": "click", "selector": sel }))
cmd["newTab"] = json!(true);
}
if follow {
cmd["follow"] = json!(true);
}
Ok(cmd)
}
"dblclick" => {
let sel = rest.first().ok_or_else(|| ParseError::MissingArguments {
@@ -453,20 +565,31 @@ fn parse_command_inner(args: &[String], flags: &Flags) -> Result<Value, ParseErr
Ok(json!({ "id": id, "action": "fill", "selector": sel, "value": rest[1..].join(" ") }))
}
"type" => {
// `--key-events` (alias `--keys`): send real per-character keystrokes
// instead of Input.insertText, so autocomplete/combobox widgets that
// only react to key events fire (e.g. Google address postal lookup).
let key_events = rest.iter().any(|a| *a == "--key-events" || *a == "--keys");
let rest: Vec<&str> = rest
.iter()
.copied()
.filter(|a| *a != "--key-events" && *a != "--keys")
.collect();
// `type --focused <text>` types into whatever element currently has
// focus (no selector) — for custom widgets that move focus to a hidden
// input after you open them.
if rest.first() == Some(&"--focused") {
return Ok(json!({
"id": id, "action": "type", "focused": true,
"text": rest[1..].join(" "),
"text": rest[1..].join(" "), "keyEvents": key_events,
}));
}
let sel = rest.first().ok_or_else(|| ParseError::MissingArguments {
context: "type".to_string(),
usage: "type <selector> <text> (or: type --focused <text>)",
usage: "type <selector> <text> (or: type --focused <text>) [--key-events]",
})?;
Ok(json!({ "id": id, "action": "type", "selector": sel, "text": rest[1..].join(" ") }))
Ok(
json!({ "id": id, "action": "type", "selector": sel, "text": rest[1..].join(" "), "keyEvents": key_events }),
)
}
"pick" => {
// pick <selector|@ref> --option "<text>" — atomic combobox select:
@@ -571,11 +694,27 @@ fn parse_command_inner(args: &[String], flags: &Flags) -> Result<Value, ParseErr
// === Keyboard ===
"press" | "key" => {
let key = rest.first().ok_or_else(|| ParseError::MissingArguments {
context: "press".to_string(),
usage: "press <key>",
let key = rest.iter().find(|a| !a.starts_with("--")).ok_or_else(|| {
ParseError::MissingArguments {
context: "press".to_string(),
usage: "press <key> [--hold <ms>]",
}
})?;
Ok(json!({ "id": id, "action": "press", "key": key }))
let mut c = json!({ "id": id, "action": "press", "key": key });
// `--hold <ms>`: hold the key down for <ms> then release, timed inside
// the daemon (one round-trip, no shell-sleep jitter) — for games and
// hold-to-charge where keydown+sleep+keyup over 3 round-trips is too
// imprecise.
if let Some(i) = rest.iter().position(|a| *a == "--hold") {
let ms = rest.get(i + 1).and_then(|s| s.parse::<u64>().ok()).ok_or(
ParseError::MissingArguments {
context: "press --hold".to_string(),
usage: "press <key> --hold <ms>",
},
)?;
c["hold"] = json!(ms);
}
Ok(c)
}
"keydown" => {
let key = rest.first().ok_or_else(|| ParseError::MissingArguments {
@@ -641,10 +780,57 @@ fn parse_command_inner(args: &[String], flags: &Flags) -> Result<Value, ParseErr
} else {
return Err(ParseError::MissingArguments {
context: "scroll --selector".to_string(),
usage: "scroll [direction] [amount] [--selector <sel>]",
usage: "scroll [direction] [amount] [--selector <sel>] [--at <x,y>] [--frame <n>]",
});
}
}
"--at" => {
// `--at x,y`: dispatch the wheel at this viewport pixel, so it
// scrolls whatever element/iframe is under the pointer — including
// cross-origin iframes that `window.scrollBy` can't reach (#36).
let val = rest.get(i + 1).ok_or(ParseError::MissingArguments {
context: "scroll --at".to_string(),
usage: "scroll [direction] [amount] --at <x,y>",
})?;
let mut parts = val.split(',');
match (
parts.next().and_then(|s| s.trim().parse::<f64>().ok()),
parts.next().and_then(|s| s.trim().parse::<f64>().ok()),
) {
(Some(x), Some(y)) => {
obj.insert("at".to_string(), json!([x, y]));
}
_ => {
return Err(ParseError::InvalidValue {
message: format!("scroll --at: invalid coordinate `{}`", val),
usage:
"scroll [direction] [amount] --at <x,y> (e.g. --at 640,400)",
})
}
}
i += 1;
}
"--frame" => {
// `--frame n`: scroll the n-th frame from `chrome-use frames` by
// dispatching the wheel at that frame's center — reaches content in
// a cross-origin iframe without needing a selector into it (#36).
let val = rest.get(i + 1).ok_or(ParseError::MissingArguments {
context: "scroll --frame".to_string(),
usage: "scroll [direction] [amount] --frame <n>",
})?;
match val.trim().parse::<usize>() {
Ok(n) => {
obj.insert("frame".to_string(), json!(n));
}
Err(_) => {
return Err(ParseError::InvalidValue {
message: format!("scroll --frame: invalid index `{}`", val),
usage: "scroll [direction] [amount] --frame <n> (index from `chrome-use frames`)",
})
}
}
i += 1;
}
arg if arg.starts_with('-') => {}
_ => {
match positional_index {
@@ -819,17 +1005,41 @@ fn parse_command_inner(args: &[String], flags: &Flags) -> Result<Value, ParseErr
// selector: @ref or CSS selector
// path: file path (contains / or . or ends with known extension)
let mut full_page = false;
let positional: Vec<&str> = rest
.iter()
.filter(|arg| match **arg {
"--full" | "-f" => {
full_page = true;
false
let mut clip: Option<Value> = None;
let mut positional: Vec<&str> = Vec::new();
let mut i = 0;
while i < rest.len() {
match rest[i] {
"--full" | "-f" => full_page = true,
// `--clip x,y,w,h` captures a pixel region (issue #34).
"--clip" => {
let raw = rest
.get(i + 1)
.ok_or_else(|| ParseError::MissingArguments {
context: "screenshot --clip".to_string(),
usage: "screenshot --clip <x,y,w,h> [path]",
})?;
let nums: Vec<f64> = raw
.split(',')
.filter_map(|n| n.trim().parse::<f64>().ok())
.collect();
if nums.len() != 4 {
return Err(ParseError::InvalidValue {
message: format!(
"--clip expects 'x,y,w,h' (4 numbers), got '{raw}'"
),
usage: "screenshot --clip <x,y,w,h> [path]",
});
}
clip = Some(json!({
"x": nums[0], "y": nums[1], "width": nums[2], "height": nums[3]
}));
i += 1;
}
_ => true,
})
.copied()
.collect();
other => positional.push(other),
}
i += 1;
}
let (selector, path) = match (positional.first(), positional.get(1)) {
(Some(first), Some(second)) => {
// Two args: first is selector, second is path
@@ -860,6 +1070,9 @@ fn parse_command_inner(args: &[String], flags: &Flags) -> Result<Value, ParseErr
"path": path, "selector": selector,
"fullPage": full_page, "annotate": flags.annotate
});
if let Some(c) = clip {
cmd["clip"] = c;
}
if let Some(ref fmt) = flags.screenshot_format {
cmd["format"] = json!(fmt);
}
@@ -981,6 +1194,47 @@ fn parse_command_inner(args: &[String], flags: &Flags) -> Result<Value, ParseErr
Ok(json!({ "id": id, "action": "evaluate", "script": script }))
}
"site" => {
// `site <name>/<command> [positional...] [--key value]`. The
// `update`/`list`/`info` subcommands are handled CLI-side (main.rs)
// and never reach here — by this point `rest[0]` is a `name/cmd`
// adapter spec. Load it, map the args onto the adapter's declared
// `args`, and emit a `site` action: the daemon navigates to the
// adapter's @meta.domain (reusing the tab if already there) and evals
// the adapter function in the site's own logged-in page.
let spec = rest.first().ok_or(ParseError::InvalidValue {
message: "site requires <name>/<command> (run `chrome-use site list`)".to_string(),
usage: "site <name>/<command> [args]",
})?;
let adapter =
crate::site::load_adapter(spec).map_err(|e| ParseError::InvalidValue {
message: e,
usage: "site <name>/<command> [args]",
})?;
let domain = adapter
.domain()
.ok_or(ParseError::InvalidValue {
message: format!("site: adapter `{spec}` @meta is missing a \"domain\""),
usage: "site <name>/<command>",
})?
.to_string();
// Split remaining args: `--key value` → named, everything else → positional.
let mut positional: Vec<String> = Vec::new();
let mut named: Vec<(String, String)> = Vec::new();
let mut it = rest[1..].iter();
while let Some(a) = it.next() {
if let Some(key) = a.strip_prefix("--") {
let val = it.next().map(|s| s.to_string()).unwrap_or_default();
named.push((key.to_string(), val));
} else {
positional.push(a.to_string());
}
}
let mapped = crate::site::map_args(&adapter, &positional, &named);
let script = crate::site::build_eval(&adapter, &mapped);
Ok(json!({ "id": id, "action": "site", "domain": domain, "script": script }))
}
// === Stealth self-check ===
"stealth" => {
// `stealth [status]` — local stealth self-check: mode, live probes
@@ -990,8 +1244,31 @@ fn parse_command_inner(args: &[String], flags: &Flags) -> Result<Value, ParseErr
Ok(json!({ "id": id, "action": "stealth_status" }))
}
// `cf-status` — Cloudflare challenge/clearance preflight: is the page
// currently a CF challenge, and is there a still-valid cf_clearance (the
// HttpOnly persistence cookie)? Lets an agent SKIP re-solving when already
// cleared, and know when it must solve. Persistence optimization.
"cf-status" | "cf" | "cloudflare-status" | "clearance" => {
Ok(json!({ "id": id, "action": "cf_status" }))
}
// === Close ===
"close" | "quit" | "exit" => Ok(json!({ "id": id, "action": "close" })),
"close" | "quit" | "exit" => {
// `close <tab>` closes only that tab (and the output says "Tab
// closed"); bare `close` closes the browser/session. `close --all` is
// intercepted earlier in the dispatcher. Previously `close t12` still
// ran a browser close and alarmingly printed "Browser closed" (#26).
if let Some(tab_ref) = rest.iter().find(|a| !a.starts_with("--")) {
Ok(json!({ "id": id, "action": "tab_close", "tabId": tab_ref }))
} else {
Ok(json!({ "id": id, "action": "close" }))
}
}
// The active tab's stable handle — `targetId` survives cross-process
// navigation and is reusable across sessions, so an agent can hold it
// instead of re-deriving "which tab is live" from `tabs` each step (#26).
"current" => Ok(json!({ "id": id, "action": "current" })),
// === Inspect ===
"inspect" => Ok(json!({ "id": id, "action": "inspect" })),
@@ -1246,6 +1523,11 @@ fn parse_command_inner(args: &[String], flags: &Flags) -> Result<Value, ParseErr
// === Get ===
"get" => parse_get(&rest, &id),
// List every frame the session can reach (top + same-process child
// frames + out-of-process iframes), with a text-length per frame so you
// can see where a listing's description actually lives (issue #27).
"frames" => Ok(json!({ "id": id, "action": "frames" })),
// Top-level shortcuts for `get <x>` status reads — users naturally type
// `chrome-use url` / `cdp-url` / `title` without the `get` prefix
// (and expect `cdp-url`/`cdp_url` to work interchangeably).
@@ -1328,11 +1610,11 @@ fn parse_command_inner(args: &[String], flags: &Flags) -> Result<Value, ParseErr
usage: "cookies transfer --from <profile> [--domain <domain>]",
});
}
return Ok(json!({
Ok(json!({
"id": id,
"action": "cookies_set",
"cookies": cookies,
}));
}))
}
"set" => {
// --curl <file> mode: import cookies from a JSON array,
@@ -1496,7 +1778,12 @@ fn parse_command_inner(args: &[String], flags: &Flags) -> Result<Value, ParseErr
// `tabs` (plural) is a natural guess for the `tab` subcommand tree —
// alias it so `tabs` / `tabs list` / `tabs new` all work (issue #8.4).
"tab" | "tabs" => {
match rest.first().copied() {
// `--full` makes `tab list` emit untruncated URLs (needed to re-open
// a long SSO/redirect URL after a stale session — issue #19). Pick
// the subcommand as the first non-flag arg so the flag can appear
// anywhere (`tab --full`, `tab list --full`).
let full = rest.contains(&"--full");
match rest.iter().find(|a| !a.starts_with("--")).copied() {
Some("new") => {
// Accepted forms:
// tab new [url]
@@ -1528,7 +1815,13 @@ fn parse_command_inner(args: &[String], flags: &Flags) -> Result<Value, ParseErr
}
Ok(cmd)
}
Some("list") => Ok(json!({ "id": id, "action": "tab_list" })),
Some("list") => {
let mut cmd = json!({ "id": id, "action": "tab_list" });
if full {
cmd["full"] = json!(true);
}
Ok(cmd)
}
Some("close") => {
let mut cmd = json!({ "id": id, "action": "tab_close" });
if let Some(tab_ref) = rest.get(1) {
@@ -1536,12 +1829,23 @@ fn parse_command_inner(args: &[String], flags: &Flags) -> Result<Value, ParseErr
}
Ok(cmd)
}
Some(tab_ref) => Ok(json!({
"id": id,
"action": "tab_switch",
"tabId": tab_ref,
})),
None => Ok(json!({ "id": id, "action": "tab_list" })),
Some(tab_ref) => {
// `tab <ref> --activate` (alias `--front`) switches to the tab
// AND raises it to the foreground — for handing a specific tab
// to the human (SMS code, captcha) (issue #24-C).
let mut cmd = json!({ "id": id, "action": "tab_switch", "tabId": tab_ref });
if rest.iter().any(|a| *a == "--activate" || *a == "--front") {
cmd["activate"] = json!(true);
}
Ok(cmd)
}
None => {
let mut cmd = json!({ "id": id, "action": "tab_list" });
if full {
cmd["full"] = json!(true);
}
Ok(cmd)
}
}
}
@@ -2316,11 +2620,42 @@ fn parse_get(rest: &[&str], id: &str) -> Result<Value, ParseError> {
match rest.first().copied() {
Some("text") => {
let sel = rest.get(1).ok_or_else(|| ParseError::MissingArguments {
context: "get text".to_string(),
usage: "get text <selector>",
})?;
Ok(json!({ "id": id, "action": "gettext", "selector": sel }))
// `get text --all-frames` aggregates visible text across every
// frame, including out-of-process iframes invisible to the top
// document (issue #27). The selector is ignored in this mode.
let all_frames = rest[1..]
.iter()
.any(|a| matches!(*a, "--all-frames" | "--frames" | "-a"));
if all_frames {
return Ok(json!({ "id": id, "action": "gettext", "allFrames": true }));
}
// `get text --pierce` reads through CLOSED shadow DOM / child docs
// via the CDP DOM tree — content eval/innerText can't reach, e.g. an
// extension's injected panel in a closed shadow root (issue #30).
let pierce = rest[1..]
.iter()
.any(|a| matches!(*a, "--pierce" | "--shadow" | "--deep"));
if pierce {
return Ok(json!({ "id": id, "action": "gettext", "pierce": true }));
}
// `get text --main` returns the main-content region (readability),
// skipping header/nav/footer/sidebar boilerplate (issue #27).
let main = rest[1..]
.iter()
.any(|a| matches!(*a, "--main" | "--readable" | "-m"));
if main {
return Ok(json!({ "id": id, "action": "gettext", "main": true }));
}
// `get text` with no selector reads the WHOLE PAGE and now defaults
// to cross-frame aggregation, so an agent gets a page's iframed
// content (listing descriptions etc.) without having to know about
// `--all-frames` (#27). On a single-frame page this is identical to
// the old body read; multi-frame pages get the child frames too —
// a strict superset. An explicit selector stays element-scoped.
match rest.iter().skip(1).find(|a| !a.starts_with("--")).copied() {
Some(sel) => Ok(json!({ "id": id, "action": "gettext", "selector": sel })),
None => Ok(json!({ "id": id, "action": "gettext", "allFrames": true })),
}
}
Some("html") => {
let sel = rest.get(1).ok_or_else(|| ParseError::MissingArguments {
@@ -3552,6 +3887,40 @@ mod tests {
assert_eq!(cmd["url"], "https://example.com");
}
#[test]
fn test_press_plain_and_hold() {
let cmd = parse_command(&args("press d"), &default_flags()).unwrap();
assert_eq!(cmd["action"], "press");
assert_eq!(cmd["key"], "d");
assert!(cmd.get("hold").is_none());
let held = parse_command(&args("press d --hold 800"), &default_flags()).unwrap();
assert_eq!(held["key"], "d");
assert_eq!(held["hold"], 800);
// Missing/invalid duration is an error, not a silent no-hold.
assert!(parse_command(&args("press d --hold"), &default_flags()).is_err());
assert!(parse_command(&args("press d --hold abc"), &default_flags()).is_err());
}
#[test]
fn test_navigate_reuse_tab_flag() {
let cmd = parse_command(
&args("open https://example.com --reuse-tab"),
&default_flags(),
)
.unwrap();
assert_eq!(cmd["action"], "navigate");
assert_eq!(cmd["reuseTab"], true);
// Alias.
let cmd2 =
parse_command(&args("open https://example.com --reuse"), &default_flags()).unwrap();
assert_eq!(cmd2["reuseTab"], true);
// Absent by default.
let cmd3 = parse_command(&args("open https://example.com"), &default_flags()).unwrap();
assert!(cmd3.get("reuseTab").is_none());
}
#[test]
fn test_navigate_with_headers() {
let mut flags = default_flags();
@@ -3700,6 +4069,21 @@ mod tests {
assert!(cmd.get("x").is_none());
}
#[test]
fn test_click_follow_flag() {
// `--follow` sets the flag; the selector is still found even with the flag
// before it (issue #24-A).
let cmd = parse_command(&args("click @e5 --follow"), &default_flags()).unwrap();
assert_eq!(cmd["selector"], "@e5");
assert_eq!(cmd["follow"], true);
let cmd2 = parse_command(&args("click --follow @e5"), &default_flags()).unwrap();
assert_eq!(cmd2["selector"], "@e5");
assert_eq!(cmd2["follow"], true);
// Absent by default.
let plain = parse_command(&args("click @e5"), &default_flags()).unwrap();
assert!(plain.get("follow").is_none());
}
#[test]
fn test_tabs_alias_lists() {
assert_eq!(
@@ -3716,6 +4100,30 @@ mod tests {
);
}
#[test]
fn test_tab_list_full_flag() {
// issue #19: `--full` → untruncated URLs; works as `tab list --full`,
// `tab --full`, and `tabs --full`. Plain list has no `full`.
for inv in ["tab list --full", "tab --full", "tabs --full"] {
let cmd = parse_command(&args(inv), &default_flags()).unwrap();
assert_eq!(cmd["action"], "tab_list", "{inv}");
assert_eq!(cmd["full"], true, "{inv}");
}
let plain = parse_command(&args("tab list"), &default_flags()).unwrap();
assert_eq!(plain["action"], "tab_list");
assert!(plain.get("full").is_none());
}
#[test]
fn test_bring_to_front_aliases() {
// issue #19: the documented `bringToFront` (+ kebab/lowercase) maps to
// the existing daemon action.
for inv in ["bringToFront", "bring-to-front", "bringtofront"] {
let cmd = parse_command(&args(inv), &default_flags()).unwrap();
assert_eq!(cmd["action"], "bringtofront", "{inv}");
}
}
#[test]
fn test_get_text_hyphen_and_underscore_aliases() {
for verb in ["get-text", "get_text"] {
@@ -3801,6 +4209,28 @@ mod tests {
assert_eq!(cmd["action"], "type");
assert_eq!(cmd["selector"], "#input");
assert_eq!(cmd["text"], "some text");
assert_eq!(cmd["keyEvents"], false);
}
#[test]
fn test_type_key_events() {
// --key-events sends real keystrokes (for autocomplete/combobox) and must
// not be swallowed into the typed text.
let cmd = parse_command(
&args("type #postal 201-0001 --key-events"),
&default_flags(),
)
.unwrap();
assert_eq!(cmd["action"], "type");
assert_eq!(cmd["selector"], "#postal");
assert_eq!(cmd["text"], "201-0001");
assert_eq!(cmd["keyEvents"], true);
let focused =
parse_command(&args("type --focused 201-0001 --keys"), &default_flags()).unwrap();
assert_eq!(focused["focused"], true);
assert_eq!(focused["text"], "201-0001");
assert_eq!(focused["keyEvents"], true);
}
#[test]
@@ -3884,6 +4314,28 @@ mod tests {
assert_eq!(cmd["tabId"], "docs");
}
#[test]
fn test_close_tab_vs_browser() {
// `close <tab>` closes that tab (says "Tab closed"); bare `close` closes
// the browser (#26).
let tab = parse_command(&args("close t12"), &default_flags()).unwrap();
assert_eq!(tab["action"], "tab_close");
assert_eq!(tab["tabId"], "t12");
let browser = parse_command(&args("close"), &default_flags()).unwrap();
assert_eq!(browser["action"], "close");
// `quit`/`exit` aliases still browser-close.
assert_eq!(
parse_command(&args("quit"), &default_flags()).unwrap()["action"],
"close"
);
}
#[test]
fn test_current_command() {
let cmd = parse_command(&args("current"), &default_flags()).unwrap();
assert_eq!(cmd["action"], "current");
}
#[test]
fn test_tab_sends_string_tab_id() {
let cmd = parse_command(&args("tab t2"), &default_flags()).unwrap();
@@ -4063,6 +4515,24 @@ mod tests {
assert_eq!(cmd["fullPage"], true);
}
#[test]
fn test_screenshot_clip() {
// `--clip x,y,w,h` captures a pixel region (issue #34); the path still parses.
let cmd = parse_command(
&args("screenshot --clip 10,20,200,40 out.png"),
&default_flags(),
)
.unwrap();
assert_eq!(cmd["action"], "screenshot");
assert_eq!(cmd["clip"]["x"], 10.0);
assert_eq!(cmd["clip"]["y"], 20.0);
assert_eq!(cmd["clip"]["width"], 200.0);
assert_eq!(cmd["clip"]["height"], 40.0);
assert_eq!(cmd["path"], "out.png");
// Bad clip is a clear error, not silent.
assert!(parse_command(&args("screenshot --clip 1,2,3"), &default_flags()).is_err());
}
#[test]
fn test_screenshot_with_ref() {
let cmd = parse_command(&args("screenshot @e1"), &default_flags()).unwrap();
@@ -4583,12 +5053,99 @@ mod tests {
}
#[test]
fn test_get_text_missing_selector() {
let result = parse_command(&args("get text"), &default_flags());
assert!(result.is_err());
let err = result.unwrap_err();
assert!(matches!(err, ParseError::MissingArguments { .. }));
assert!(err.format().contains("get text"));
fn test_get_text_defaults_to_all_frames() {
// `get text` with no selector now reads the whole page across ALL frames
// by default (#27), so iframed content isn't silently missed. (Was: a
// top-frame `body` read, #24-D.)
let cmd = parse_command(&args("get text"), &default_flags()).unwrap();
assert_eq!(cmd["action"], "gettext");
assert_eq!(cmd["allFrames"], true);
assert!(cmd.get("selector").is_none());
// An explicit selector still wins and stays element-scoped.
let cmd2 = parse_command(&args("get text h1"), &default_flags()).unwrap();
assert_eq!(cmd2["selector"], "h1");
assert!(cmd2.get("allFrames").is_none());
// `text` top-level shortcut behaves the same.
let cmd3 = parse_command(&args("text"), &default_flags()).unwrap();
assert_eq!(cmd3["allFrames"], true);
}
#[test]
fn test_get_text_all_frames() {
// `--all-frames` switches to whole-page, cross-frame aggregation and
// drops the selector (issue #27).
for variant in ["get text --all-frames", "get text --frames", "text -a"] {
let cmd = parse_command(&args(variant), &default_flags()).unwrap();
assert_eq!(cmd["action"], "gettext", "{variant}");
assert_eq!(cmd["allFrames"], true, "{variant}");
assert!(cmd.get("selector").is_none(), "{variant}");
}
// A flag mixed with a selector still triggers all-frames.
let cmd = parse_command(&args("get text body --all-frames"), &default_flags()).unwrap();
assert_eq!(cmd["allFrames"], true);
// Without the flag, a leading flag-like token is skipped for the selector.
let cmd = parse_command(&args("get text main"), &default_flags()).unwrap();
assert_eq!(cmd["selector"], "main");
assert!(cmd.get("allFrames").is_none());
}
#[test]
fn test_frames_command() {
let cmd = parse_command(&args("frames"), &default_flags()).unwrap();
assert_eq!(cmd["action"], "frames");
}
#[test]
fn test_nearest_command_suggestions() {
assert_eq!(nearest_command("sesions").as_deref(), Some("sessions"));
assert_eq!(nearest_command("session").as_deref(), Some("sessions"));
assert_eq!(nearest_command("clik").as_deref(), Some("click"));
assert_eq!(
nearest_command("screenshits").as_deref(),
Some("screenshot")
);
// Nonsense with no close match stays silent.
assert_eq!(nearest_command("xyzzy"), None);
// The unknown-command error embeds the suggestion.
let err = ParseError::UnknownCommand {
command: "sesions".to_string(),
};
assert!(err.format().contains("Did you mean: chrome-use sessions?"));
}
#[test]
fn test_get_text_main() {
for variant in ["get text --main", "get text --readable", "text -m"] {
let cmd = parse_command(&args(variant), &default_flags()).unwrap();
assert_eq!(cmd["action"], "gettext", "{variant}");
assert_eq!(cmd["main"], true, "{variant}");
assert!(cmd.get("selector").is_none(), "{variant}");
}
}
#[test]
fn test_get_text_pierce() {
for variant in ["get text --pierce", "get text --shadow", "text --deep"] {
let cmd = parse_command(&args(variant), &default_flags()).unwrap();
assert_eq!(cmd["action"], "gettext", "{variant}");
assert_eq!(cmd["pierce"], true, "{variant}");
assert!(cmd.get("selector").is_none(), "{variant}");
}
}
#[test]
fn test_tab_activate_flag() {
let plain = parse_command(&args("tab t3"), &default_flags()).unwrap();
assert_eq!(plain["action"], "tab_switch");
assert!(plain.get("activate").is_none());
let act = parse_command(&args("tab t3 --activate"), &default_flags()).unwrap();
assert_eq!(act["action"], "tab_switch");
assert_eq!(act["tabId"], "t3");
assert_eq!(act["activate"], true);
// `--front` alias.
let front = parse_command(&args("tab t3 --front"), &default_flags()).unwrap();
assert_eq!(front["activate"], true);
}
// === Protocol alignment tests ===
@@ -5482,6 +6039,35 @@ mod tests {
assert_eq!(cmd["selector"], ".sidebar");
}
#[test]
fn test_scroll_at_coordinate() {
// `--at x,y` carries a [x, y] array for a wheel dispatched at that pixel
// (issue #36: cross-origin iframe scroll).
let cmd = parse_command(&args("scroll down 700 --at 640,400"), &default_flags()).unwrap();
assert_eq!(cmd["action"], "scroll");
assert_eq!(cmd["direction"], "down");
assert_eq!(cmd["amount"], 700);
assert_eq!(cmd["at"], json!([640.0, 400.0]));
}
#[test]
fn test_scroll_at_rejects_garbage() {
assert!(parse_command(&args("scroll --at nope"), &default_flags()).is_err());
assert!(parse_command(&args("scroll --at 1"), &default_flags()).is_err());
}
#[test]
fn test_scroll_frame_index() {
let cmd = parse_command(&args("scroll down 700 --frame 2"), &default_flags()).unwrap();
assert_eq!(cmd["action"], "scroll");
assert_eq!(cmd["frame"], 2);
}
#[test]
fn test_scroll_frame_rejects_non_integer() {
assert!(parse_command(&args("scroll --frame two"), &default_flags()).is_err());
}
#[test]
fn test_scroll_selector_before_positional() {
let cmd =
+131 -27
View File
@@ -16,8 +16,18 @@ use std::io::Write;
use std::path::PathBuf;
/// Native-messaging host name; must match `HOST_NAME` in the extension and the
/// manifest filename.
pub const HOST_NAME: &str = "com.leeguoo.chrome_use";
/// manifest filename. `com.agent_browser.connect` is the original name, used by
/// every shipped extension up to ab-connect 0.4.2.
pub const HOST_NAME: &str = "com.agent_browser.connect";
/// Alternate host name for the chrome-use rebrand era (ab-connect 0.5.0+). We
/// install AND recognize both names so the relay works regardless of which
/// extension version a user has — old (0.4.2) or new — with no forced
/// re-install. See [`install_native_host`] / [`host_installed`].
pub const HOST_NAME_ALT: &str = "com.leeguoo.chrome_use";
/// Every native-messaging host name this CLI installs and accepts.
pub const HOST_NAMES: &[&str] = &[HOST_NAME, HOST_NAME_ALT];
/// Stable id of the `ab-connect` extension, pinned by the `key` in its
/// manifest.json (and the signing key of the published `.crx`). Chrome only lets
@@ -182,18 +192,9 @@ fn install_native_host() -> Result<Vec<String>, String> {
let _ = std::fs::set_permissions(&launcher, std::fs::Permissions::from_mode(0o755));
}
let manifest = serde_json::json!({
"name": HOST_NAME,
"description": "chrome-use connect — native messaging host",
"path": launcher.display().to_string(),
"type": "stdio",
"allowed_origins": [
format!("chrome-extension://{EXTENSION_ID}/"),
format!("chrome-extension://{STORE_EXTENSION_ID}/"),
],
});
let body = serde_json::to_string_pretty(&manifest).map_err(|e| e.to_string())?;
// Write a manifest under EVERY accepted host name (both point to the same
// launcher + allowed extensions), so any extension version's
// `connectNative(<its host name>)` finds a matching host json.
let mut written = Vec::new();
for dir in native_messaging_dirs() {
if let Some(parent) = dir.parent() {
@@ -202,9 +203,22 @@ fn install_native_host() -> Result<Vec<String>, String> {
}
}
std::fs::create_dir_all(&dir).map_err(|e| e.to_string())?;
let path = dir.join(format!("{HOST_NAME}.json"));
std::fs::write(&path, &body).map_err(|e| e.to_string())?;
written.push(path.display().to_string());
for host in HOST_NAMES {
let manifest = serde_json::json!({
"name": host,
"description": "chrome-use connect — native messaging host",
"path": launcher.display().to_string(),
"type": "stdio",
"allowed_origins": [
format!("chrome-extension://{EXTENSION_ID}/"),
format!("chrome-extension://{STORE_EXTENSION_ID}/"),
],
});
let body = serde_json::to_string_pretty(&manifest).map_err(|e| e.to_string())?;
let path = dir.join(format!("{host}.json"));
std::fs::write(&path, &body).map_err(|e| e.to_string())?;
written.push(path.display().to_string());
}
}
if written.is_empty() {
return Err("no Chrome/Chromium NativeMessagingHosts directory found".into());
@@ -289,9 +303,11 @@ fn remove_force_install_profile() -> bool {
fn remove_host_manifests() -> usize {
let mut n = 0;
for dir in native_messaging_dirs() {
let path = dir.join(format!("{HOST_NAME}.json"));
if path.exists() && std::fs::remove_file(&path).is_ok() {
n += 1;
for host in HOST_NAMES {
let path = dir.join(format!("{host}.json"));
if path.exists() && std::fs::remove_file(&path).is_ok() {
n += 1;
}
}
}
n
@@ -334,7 +350,7 @@ fn native_messaging_dirs() -> Vec<PathBuf> {
fn host_manifest_path_for_chrome() -> Option<PathBuf> {
native_messaging_dirs()
.into_iter()
.map(|d| d.join(format!("{HOST_NAME}.json")))
.flat_map(|d| HOST_NAMES.iter().map(move |h| d.join(format!("{h}.json"))))
.find(|p| p.exists())
.or_else(|| {
native_messaging_dirs()
@@ -352,9 +368,11 @@ fn host_manifest_path_for_chrome() -> Option<PathBuf> {
/// service worker; this manifest is the durable signal that the extension is
/// the chosen path.
pub fn host_installed() -> bool {
native_messaging_dirs()
.into_iter()
.any(|d| d.join(format!("{HOST_NAME}.json")).exists())
native_messaging_dirs().into_iter().any(|d| {
HOST_NAMES
.iter()
.any(|h| d.join(format!("{h}.json")).exists())
})
}
fn report(json: bool, ok: bool, msg: &str) {
@@ -399,10 +417,23 @@ fn random_guid() -> String {
}
/// Where the daemon/CLI reads the relay's CDP WebSocket URL (perms 600).
///
/// Cross-binary handoff: the native-messaging *host* writes it and the CLI reads
/// it, but the two may be different binaries under different brand dirs after
/// the agent-browser → chrome-use rename. Read from whichever brand dir actually
/// has the file (an old `agent-browser` host writes `~/.agent-browser`; a
/// `chrome-use` host writes `~/.chrome-use`); default to [`config_home`].
fn relay_url_path() -> PathBuf {
dirs::home_dir()
.map(|h| h.join(".chrome-use").join("relay-cdp-url"))
.unwrap_or_else(|| PathBuf::from("/tmp/ab-relay-cdp-url"))
if let Some(home) = dirs::home_dir() {
for base in [".chrome-use", ".agent-browser"] {
let p = home.join(base).join("relay-cdp-url");
if p.exists() {
return p;
}
}
return crate::connection::config_home().join("relay-cdp-url");
}
PathBuf::from("/tmp/ab-relay-cdp-url")
}
/// The live relay CDP WebSocket URL, if the native-messaging host is running
@@ -418,6 +449,69 @@ pub fn relay_url() -> Option<String> {
}
}
/// Append a one-line record of how a CDP connection was established, to
/// `~/.chrome-use/connect-mode.log`. This is the smoking-gun detector for the
/// "Allow remote debugging?" consent modal: that modal ONLY appears on a raw
/// remote-debugging attach / a browser we launched with a debug port — NEVER on
/// the extension relay. When the modal reappears, this log says which session
/// took which path and when, so we can tell a code regression (`raw-port` /
/// `launched` while the relay was up) from Chrome's own extension-debugger
/// consent UX. Low volume (one line per connection); best-effort, never fails a
/// connection.
pub fn log_connect_mode(ws_url: &str, launched: bool, session: &str) {
let relay = relay_url();
let relay_up = relay.is_some();
let mode = if launched {
"launched(debug-port)"
} else if relay.as_deref() == Some(ws_url) {
"relay"
} else if ws_url.contains("127.0.0.1") || ws_url.contains("localhost") {
"raw-port-attach"
} else {
"remote-ws"
};
// A raw-port attach or a self-launch while the relay was available is the
// exact thing that pops the consent modal — flag it loudly in the line.
let suspect = (mode == "raw-port-attach" || launched) && relay_up;
let line = format!(
"session={session} mode={mode} relay_up={relay_up}{} ws={ws_url}\n",
if suspect { " CONSENT-MODAL-RISK" } else { "" }
);
if let Some(home) = dirs::home_dir() {
let path = home.join(".chrome-use").join("connect-mode.log");
use std::io::Write;
if let Ok(mut f) = std::fs::OpenOptions::new()
.create(true)
.append(true)
.open(&path)
{
let _ = f.write_all(line.as_bytes());
}
}
}
/// Sidecar recording the connected extension's version, written by the host when
/// it receives the extension's `hello` (sibling of `relay-cdp-url`). Lets
/// `doctor` surface which extension build is live without a CDP round-trip.
fn relay_ext_version_path() -> PathBuf {
relay_url_path().with_file_name("relay-ext-version")
}
/// Version of the connected `ab-connect` extension, if the host learned it from
/// the extension's `hello`. `None` when no extension has connected since the
/// host started, or the extension predates version reporting.
pub fn relay_ext_version() -> Option<String> {
let s = std::fs::read_to_string(relay_ext_version_path())
.ok()?
.trim()
.to_string();
if s.is_empty() {
None
} else {
Some(s)
}
}
/// Hidden `__nm-host` mode: launched by Chrome for the ab-connect extension.
///
/// Bridges the extension (native-messaging stdio, envelope protocol) to a local
@@ -539,6 +633,15 @@ async fn nm_host_main() {
Ok(v) => v,
Err(_) => continue,
};
// Extension version handshake: record it next to the relay URL so
// `doctor` can report which extension build is live (and whether it's
// behind). Best-effort; the message carries no CDP payload.
if v.get("method").and_then(|m| m.as_str()) == Some("hello") {
if let Some(ver) = v.get("version").and_then(|x| x.as_str()) {
let _ = std::fs::write(relay_ext_version_path(), ver);
}
continue;
}
let outs = {
let mut s = state.lock().await;
s.handle_ext_message(&v, "")
@@ -571,6 +674,7 @@ async fn nm_host_main() {
}
nm_log("[nm-host] stdin EOF — Chrome closed the port");
let _ = std::fs::remove_file(relay_url_path());
let _ = std::fs::remove_file(relay_ext_version_path());
}
#[allow(clippy::too_many_arguments)]
+101 -5
View File
@@ -88,8 +88,39 @@ impl Connection {
}
}
/// Brand-compat config directory basename. The project renamed
/// `agent-browser` → `chrome-use`, but this dotfile dir is invisible internal
/// plumbing: it's shared with the native-messaging host (the `relay-cdp-url`
/// handoff) and holds saved auth/daemon state. Renaming it would break existing
/// installs and re-pop the "Allow remote debugging?" dialog when the relay
/// can't be located. So decide ONCE per run: prefer the new `.chrome-use`, but
/// keep using an existing `.agent-browser` install if that's the only one
/// present; fresh installs get `.chrome-use`. `dotted` picks the home-dir form
/// (`.chrome-use`) vs the XDG/tmp subdir form (`chrome-use`); both agree.
pub fn config_dir_basename(dotted: bool) -> &'static str {
let prefer_old = dirs::home_dir()
.map(|h| !h.join(".chrome-use").exists() && h.join(".agent-browser").exists())
.unwrap_or(false);
match (prefer_old, dotted) {
(true, true) => ".agent-browser",
(true, false) => "agent-browser",
(false, true) => ".chrome-use",
(false, false) => "chrome-use",
}
}
/// The home-based config dir (`~/.chrome-use`, or `~/.agent-browser` on an
/// existing install — see [`config_dir_basename`]). Single source of truth so
/// sockets, auth, and the relay handoff all agree within one run.
pub fn config_home() -> PathBuf {
match dirs::home_dir() {
Some(home) => home.join(config_dir_basename(true)),
None => env::temp_dir().join(config_dir_basename(false)),
}
}
/// Get the base directory for socket/pid files.
/// Priority: AGENT_BROWSER_SOCKET_DIR > XDG_RUNTIME_DIR > ~/.chrome-use > tmpdir
/// Priority: AGENT_BROWSER_SOCKET_DIR > XDG_RUNTIME_DIR > config_home() > tmpdir
pub fn get_socket_dir() -> PathBuf {
// 1. Explicit override (ignore empty string)
if let Ok(dir) = env::var("AGENT_BROWSER_SOCKET_DIR") {
@@ -101,17 +132,17 @@ pub fn get_socket_dir() -> PathBuf {
// 2. XDG_RUNTIME_DIR (Linux standard, ignore empty string)
if let Ok(runtime_dir) = env::var("XDG_RUNTIME_DIR") {
if !runtime_dir.is_empty() {
return PathBuf::from(runtime_dir).join("chrome-use");
return PathBuf::from(runtime_dir).join(config_dir_basename(false));
}
}
// 3. Home directory fallback (like Docker Desktop's ~/.docker/run/)
if let Some(home) = dirs::home_dir() {
return home.join(".chrome-use");
if dirs::home_dir().is_some() {
return config_home();
}
// 4. Last resort: temp dir
env::temp_dir().join("chrome-use")
env::temp_dir().join(config_dir_basename(false))
}
#[cfg(unix)]
@@ -611,6 +642,22 @@ fn kill_stale_daemon(session: &str) {
cleanup_stale_files(session);
}
/// Kill every per-session daemon worker (SIGTERM→SIGKILL + sidecar cleanup),
/// leaving the Chrome-launched `__nm-host` native-messaging bridge alone — it's
/// not a tracked session daemon, so the extension relay stays up. Returns the
/// session names that were stopped. Powers `chrome-use daemon restart`, which
/// clears corrupted/cross-leaked daemon state (e.g. after a version-mismatch
/// restart) without the user resorting to `pgrep`/`kill` (issue #20).
pub fn restart_all_daemons() -> Vec<String> {
let inventory = walk_daemons();
let mut stopped = Vec::new();
for session in &inventory.sessions {
kill_stale_daemon(&session.name);
stopped.push(session.name.clone());
}
stopped
}
pub fn ensure_daemon(session: &str, opts: &DaemonOptions) -> Result<DaemonResult, String> {
// Socket connectivity is the sole liveness check — no PID check — so
// callers in a different PID namespace (e.g. unshare) can still reuse
@@ -1151,6 +1198,55 @@ mod tests {
let _ = fs::remove_dir(&dir);
}
#[test]
fn test_restart_all_daemons_empty_dir() {
let dir = std::env::temp_dir().join("ab-test-restart-empty");
let _ = fs::create_dir_all(&dir);
let _guard = EnvGuard::new(&["AGENT_BROWSER_SOCKET_DIR", "XDG_RUNTIME_DIR"]);
_guard.set("AGENT_BROWSER_SOCKET_DIR", dir.to_str().unwrap());
// No daemons registered → nothing to stop, and it must not blow up.
assert!(restart_all_daemons().is_empty());
let _ = fs::remove_dir(&dir);
}
#[cfg(unix)]
#[test]
fn test_restart_all_daemons_kills_live_session() {
let dir = std::env::temp_dir().join("ab-test-restart-live");
let _ = fs::create_dir_all(&dir);
let _guard = EnvGuard::new(&["AGENT_BROWSER_SOCKET_DIR", "XDG_RUNTIME_DIR"]);
_guard.set("AGENT_BROWSER_SOCKET_DIR", dir.to_str().unwrap());
// Spawn a real, killable child and register it as a session daemon.
let mut child = Command::new("sleep")
.arg("30")
.spawn()
.expect("spawn sleep");
let pid = child.id();
let _ = fs::write(dir.join("rktest.pid"), pid.to_string());
let _ = fs::write(get_socket_path("rktest"), b"");
let stopped = restart_all_daemons();
assert!(
stopped.contains(&"rktest".to_string()),
"stopped: {:?}",
stopped
);
// Reap the killed child first — until the parent waits, it lingers as a
// zombie that still answers `kill(pid, 0)`, so is_pid_alive would lie.
let _ = child.wait();
assert!(!is_pid_alive(pid));
// Sidecars are cleaned up.
assert!(!dir.join("rktest.pid").exists());
assert!(!get_socket_path("rktest").exists());
let _ = fs::remove_dir(&dir);
}
#[test]
fn test_cleanup_stale_files_removes_version() {
let dir = std::env::temp_dir().join("ab-test-cleanup-version");
+2
View File
@@ -18,6 +18,7 @@ mod launch;
mod network;
mod providers;
mod security;
mod versions;
use serde_json::{json, Value};
@@ -97,6 +98,7 @@ pub fn run_doctor(opts: DoctorOptions) -> i32 {
let mut fixed: Vec<String> = Vec::new();
environment::check(&mut checks);
versions::check(&mut checks);
chrome::check(&mut checks);
daemon::check(&mut checks);
config::check(&mut checks);
+89
View File
@@ -0,0 +1,89 @@
//! Version-coherence checks across all four moving parts: the CLI binary, the
//! per-session daemons (covered by `daemon.rs`), the connected `ab-connect`
//! extension, and the bundled skill. The extension was previously a black box —
//! nothing reported which build was live — so a user could sit on an old
//! extension with no signal. The extension now reports its version over the
//! relay (`hello`), the host records it, and this surfaces it in one place.
use super::{Check, Status};
use crate::{connect, upgrade};
pub(super) fn check(checks: &mut Vec<Check>) {
let category = "Versions";
let cli_version = env!("CARGO_PKG_VERSION");
// CLI — compare against the latest seen by the background update check.
match upgrade::cached_latest_version() {
Some(latest) if upgrade::version_is_newer(&latest, cli_version) => {
checks.push(
Check::new(
"versions.cli",
category,
Status::Warn,
format!("CLI {cli_version} (newer available: {latest})"),
)
.with_fix("chrome-use upgrade".to_string()),
);
}
_ => {
checks.push(Check::new(
"versions.cli",
category,
Status::Pass,
format!("CLI {cli_version}"),
));
}
}
// Extension — the build this CLI shipped alongside (embedded at compile time
// from the extension manifest) is what we expect to be running.
let expected_ext = env!("AB_CONNECT_VERSION");
match connect::relay_ext_version() {
Some(ext) if upgrade::version_is_newer(expected_ext, &ext) => {
checks.push(
Check::new(
"versions.extension",
category,
Status::Warn,
format!("extension {ext} is behind the bundled {expected_ext}"),
)
.with_fix(
"update ab-connect in Chrome: chrome://extensions \u{2192} reload \
(or wait for the Web Store auto-update)"
.to_string(),
),
);
}
Some(ext) => {
checks.push(Check::new(
"versions.extension",
category,
Status::Pass,
format!("extension {ext}"),
));
}
None => {
checks.push(Check::new(
"versions.extension",
category,
Status::Info,
format!(
"extension not connected (or it predates version reporting — \
expected {expected_ext})"
),
));
}
}
// Skill — ships inside the same release artifact as the binary, so it's
// version-locked here. Copies made elsewhere via `skills add` aren't.
checks.push(Check::new(
"versions.skill",
category,
Status::Info,
format!(
"skills bundled with this CLI ({cli_version}); copies made via `skills add` \
elsewhere may be stale re-run to refresh"
),
));
}
+260 -5
View File
@@ -10,7 +10,9 @@ mod flags;
mod install;
mod native;
mod output;
mod site;
mod skills;
mod test_runner;
#[cfg(test)]
mod test_utils;
mod upgrade;
@@ -28,8 +30,8 @@ use windows_sys::Win32::System::Threading::OpenProcess;
use commands::{gen_id, parse_command, ParseError};
use connection::{
cleanup_stale_files, ensure_daemon, get_socket_dir, is_pid_alive, send_command, walk_daemons,
DaemonOptions,
cleanup_stale_files, ensure_daemon, get_socket_dir, is_pid_alive, restart_all_daemons,
send_command, walk_daemons, DaemonOptions,
};
use flags::{clean_args, parse_flags, Flags};
use install::run_install;
@@ -269,13 +271,19 @@ fn run_session(args: &[String], session: &str, json_mode: bool) {
.into_iter()
.map(|s| s.name)
.collect();
// The extension relay drives the user's live Chrome but isn't always
// registered as a launched daemon session — without surfacing it,
// `session list` says "No active sessions" while open/tab work fine,
// and agents misjudge the connection as down (issue #15).
let relay_up = connect::relay_url().is_some();
if json_mode {
println!(
r#"{{"success":true,"data":{{"sessions":{}}}}}"#,
serde_json::to_string(&sessions).unwrap_or_default()
r#"{{"success":true,"data":{{"sessions":{},"relay":{}}}}}"#,
serde_json::to_string(&sessions).unwrap_or_default(),
relay_up
);
} else if sessions.is_empty() {
} else if sessions.is_empty() && !relay_up {
println!("No active sessions");
} else {
println!("Active sessions:");
@@ -287,6 +295,14 @@ fn run_session(args: &[String], session: &str, json_mode: bool) {
};
println!("{} {}", marker, s);
}
if relay_up && !sessions.iter().any(|s| s == session) {
println!(
"{} {} {}",
color::cyan(""),
session,
color::dim("(relay/extension → live Chrome)")
);
}
}
}
None | Some(_) => {
@@ -305,6 +321,94 @@ fn run_session(args: &[String], session: &str, json_mode: bool) {
}
}
/// `chrome-use daemon <restart|status>` — manage the per-session daemon workers
/// without resorting to `pgrep`/`kill`. `restart` clears corrupted or
/// cross-leaked daemon state (e.g. after a mid-session `chrome-use upgrade`
/// where stale tab handles bleed across sessions, issue #20) by killing every
/// session worker. The Chrome-launched `__nm-host` native-messaging bridge is
/// NOT a tracked session daemon, so the extension relay survives a restart —
/// the next command spins up a fresh, clean daemon against the same live Chrome.
fn run_daemon(args: &[String], json_mode: bool) {
match args.get(1).map(|s| s.as_str()) {
Some("restart") => {
let stopped = restart_all_daemons();
let relay_up = connect::relay_url().is_some();
if json_mode {
print_json_value(json!({
"success": true,
"data": { "stopped": stopped, "count": stopped.len(), "relay": relay_up },
}));
} else if stopped.is_empty() {
println!("No session daemons running — nothing to restart.");
if relay_up {
println!(
"{}",
color::dim("Extension relay still up; next command starts a fresh daemon.")
);
}
} else {
for s in &stopped {
println!("{} Stopped daemon: {}", color::green(""), s);
}
println!(
"{}",
color::dim(if relay_up {
"Extension relay (__nm-host) left running; next command starts a fresh daemon."
} else {
"Next command starts a fresh daemon."
})
);
}
}
Some("status") | Some("list") => {
let inventory = walk_daemons();
let relay_up = connect::relay_url().is_some();
if json_mode {
let sessions: Vec<_> = inventory
.sessions
.iter()
.map(|s| json!({ "name": s.name, "pid": s.pid, "version": s.version }))
.collect();
print_json_value(json!({
"success": true,
"data": { "sessions": sessions, "relay": relay_up },
}));
} else if inventory.sessions.is_empty() {
println!("No session daemons running.");
if relay_up {
println!("{}", color::dim("Extension relay (__nm-host): up"));
}
} else {
println!("Session daemons:");
for s in &inventory.sessions {
let ver = s
.version
.as_deref()
.map(|v| format!(" {}", color::dim(&format!("(v{})", v))))
.unwrap_or_default();
println!(" {} pid {}{}", s.name, s.pid, ver);
}
if relay_up {
println!("{}", color::dim("Extension relay (__nm-host): up"));
}
}
}
other => {
eprintln!(
"{} usage: chrome-use daemon <restart|status>",
color::error_indicator()
);
if let Some(unknown) = other {
eprintln!(
"{}",
color::dim(&format!(" unknown subcommand: {}", unknown))
);
}
exit(2);
}
}
}
fn get_dashboard_pid_path() -> std::path::PathBuf {
get_socket_dir().join("dashboard.pid")
}
@@ -571,6 +675,17 @@ fn main() {
return;
}
// Hidden update-check worker, spawned detached by maybe_notify_update() to
// refresh the cached latest version without blocking a real command.
if env::args().nth(1).as_deref() == Some("__update-check") {
upgrade::run_update_check();
return;
}
// Non-blocking "update available" hint (stderr only; self-skips meta
// commands, daemon mode, CI, and the opt-out env vars).
upgrade::maybe_notify_update();
// Native daemon mode: when AGENT_BROWSER_DAEMON is set, run as the daemon process
if env::var("AGENT_BROWSER_DAEMON").is_ok() {
// Ignore SIGPIPE so the daemon isn't killed when the parent drops
@@ -707,6 +822,118 @@ fn main() {
return;
}
// Handle `test <suite.yaml>`: run a browser test suite. It orchestrates by
// re-invoking this binary per step, so it lives outside the normal dispatch.
if clean.first().map(|s| s.as_str()) == Some("test") {
let Some(suite) = clean.get(1) else {
eprintln!(
"{} usage: chrome-use test <suite.yaml> [--launch | --session <name>]",
color::error_indicator()
);
exit(2);
};
exit(test_runner::run_test(suite, &flags));
}
// Handle `site`: site adapters — turn a website into a structured-data CLI by
// running a per-command JS adapter inside your logged-in tab. `update`/`list`/
// `info` are CLI-side (download/filesystem); `site <name>/<cmd> [args]` falls
// through to the daemon dispatch below (navigate to the adapter's domain + eval).
if clean.first().map(|s| s.as_str()) == Some("site") {
// Auto-sync the adapter pack on first use and periodically (TTL, default
// 7d) so adapters stay fresh without a manual `site update`. Skipped for an
// explicit `update` (full sync below). Best-effort: offline → cached pack.
// Disable with AGENT_BROWSER_SITES_NO_AUTO_UPDATE=1.
if clean.get(1).map(|s| s.as_str()) != Some("update") && site::needs_refresh() {
let rt = tokio::runtime::Runtime::new().expect("Failed to create tokio runtime");
match rt.block_on(site::update()) {
Ok(n) => {
eprintln!(
"{}",
color::dim(&format!("site: synced {n} adapters (auto)"))
)
}
Err(e) => eprintln!(
"{}",
color::dim(&format!(
"site: auto-sync skipped ({e}); using cached adapters"
))
),
}
}
match clean.get(1).map(|s| s.as_str()) {
Some("update") => {
let rt = tokio::runtime::Runtime::new().expect("Failed to create tokio runtime");
match rt.block_on(site::update()) {
Ok(n) if flags.json => {
println!("{}", json!({ "success": true, "adapters": n }))
}
Ok(n) => println!(
"{} synced {} site adapters → ~/.chrome-use/sites (run `chrome-use site list`)",
color::success_indicator(),
n
),
Err(e) => {
eprintln!("{} {}", color::error_indicator(), e);
exit(1);
}
}
return;
}
Some("list") => {
match site::list_adapters() {
Ok(list) if flags.json => {
println!("{}", json!({ "success": true, "adapters": list }))
}
Ok(list) if list.is_empty() => {
println!("no site adapters installed — run `chrome-use site update`")
}
Ok(list) => {
for a in &list {
println!("{a}");
}
eprintln!(
"{}",
color::dim(&format!(
"{} adapters · run: chrome-use site <name>/<cmd> [args]",
list.len()
))
);
}
Err(e) => {
eprintln!("{} {}", color::error_indicator(), e);
exit(1);
}
}
return;
}
Some("info") => {
let spec = clean.get(2).cloned().unwrap_or_default();
match site::load_adapter(&spec) {
Ok(a) => println!(
"{}",
serde_json::to_string_pretty(&a.meta).unwrap_or_default()
),
Err(e) => {
eprintln!("{} {}", color::error_indicator(), e);
exit(1);
}
}
return;
}
// `site <name>/<cmd> [args]` → fall through to the daemon dispatch.
Some(spec) if spec.contains('/') => {}
_ => {
eprintln!(
"{} usage: chrome-use site <name>/<cmd> [args] | site update | site list | \
site info <name>/<cmd>",
color::error_indicator()
);
exit(2);
}
}
}
// Handle skills command (doesn't need daemon)
if clean.first().map(|s| s.as_str()) == Some("skills") {
skills::run_skills(&clean, flags.json);
@@ -760,6 +987,20 @@ fn main() {
return;
}
// Handle daemon management (doesn't talk to a daemon — it manages them).
if clean.first().map(|s| s.as_str()) == Some("daemon") {
run_daemon(&clean, flags.json);
return;
}
// `sessions` is a natural top-level guess for "list my sessions" (the skill
// advertises sessions as a feature) — route it to the daemon inventory the
// same way `daemon status` does (issue #29).
if clean.first().map(|s| s.as_str()) == Some("sessions") {
run_daemon(&["sessions".to_string(), "status".to_string()], flags.json);
return;
}
// Handle close --all: close all active sessions
if matches!(
clean.first().map(|s| s.as_str()),
@@ -1228,6 +1469,20 @@ fn main() {
&& flags.provider.is_none()
&& (flags.force_launch || !flags.auto_connect)
{
// Launching a debug-port Chrome pops Chrome's "Allow remote debugging?"
// consent modal (Chrome 136+). When the ab-connect relay is already up,
// this is almost always unintended — the relay drives the user's real
// Chrome with NO modal. Warn so the modal is self-explained and the
// caller (often a stray --launch / --no-auto-connect) is fixable (#32).
if !flags.json && connect::relay_url().is_some() {
eprintln!(
"{} launching a new Chrome with a debug port — this pops Chrome's \
\"Allow remote debugging?\" modal.\n The ab-connect relay is up; \
drop --launch/--new (and don't pass --no-auto-connect) to drive your \
real Chrome with no modal.",
color::warning_indicator()
);
}
let mut launch_cmd = json!({
"id": gen_id(),
"action": "launch",
+667 -33
View File
@@ -1315,6 +1315,7 @@ pub async fn execute_command(cmd: &Value, state: &mut DaemonState) -> Value {
"title" => handle_title(state).await,
"content" => handle_content(state).await,
"evaluate" => handle_evaluate(cmd, state).await,
"site" => handle_site(cmd, state).await,
"close" => handle_close(state).await,
"stealth_status" => handle_stealth_status(state).await,
"snapshot" => handle_snapshot(cmd, state).await,
@@ -1332,6 +1333,7 @@ pub async fn execute_command(cmd: &Value, state: &mut DaemonState) -> Value {
"uncheck" => handle_uncheck(cmd, state).await,
"wait" => handle_wait(cmd, state).await,
"gettext" => handle_gettext(cmd, state).await,
"frames" => handle_frames(cmd, state).await,
"getattribute" => handle_getattribute(cmd, state).await,
"isvisible" => handle_isvisible(cmd, state).await,
"isenabled" => handle_isenabled(cmd, state).await,
@@ -1340,6 +1342,7 @@ pub async fn execute_command(cmd: &Value, state: &mut DaemonState) -> Value {
"forward" => handle_forward(state).await,
"reload" => handle_reload(state).await,
"cookies_get" => handle_cookies_get(cmd, state).await,
"cf_status" => handle_cf_status(cmd, state).await,
"cookies_set" => handle_cookies_set(cmd, state).await,
"cookies_clear" => handle_cookies_clear(state).await,
"storage_get" => handle_storage_get(cmd, state).await,
@@ -1364,7 +1367,7 @@ pub async fn execute_command(cmd: &Value, state: &mut DaemonState) -> Value {
"recording_stop" => handle_recording_stop(state).await,
"recording_restart" => handle_recording_restart(cmd, state).await,
"pdf" => handle_pdf(cmd, state).await,
"tab_list" => handle_tab_list(state).await,
"tab_list" => handle_tab_list(cmd, state).await,
"tab_new" => handle_tab_new(cmd, state).await,
"tab_switch" => handle_tab_switch(cmd, state).await,
"tab_close" => handle_tab_close(cmd, state).await,
@@ -1395,6 +1398,7 @@ pub async fn execute_command(cmd: &Value, state: &mut DaemonState) -> Value {
"count" => handle_count(cmd, state).await,
"styles" => handle_styles(cmd, state).await,
"bringtofront" => handle_bringtofront(state).await,
"current" => handle_current(state).await,
"timezone" => handle_timezone(cmd, state).await,
"locale" => handle_locale(cmd, state).await,
"geolocation" => handle_geolocation(cmd, state).await,
@@ -2474,7 +2478,10 @@ async fn handle_navigate(cmd: &Value, state: &mut DaemonState) -> Result<Value,
wb.navigate(url).await?;
let new_url = wb.get_url().await.unwrap_or_else(|_| url.to_string());
let title = wb.get_title().await.unwrap_or_default();
return Ok(json!({ "url": new_url, "title": title }));
return Ok(with_site_hint(
json!({ "url": new_url, "title": title }),
url,
));
}
}
@@ -2531,11 +2538,52 @@ async fn handle_navigate(cmd: &Value, state: &mut DaemonState) -> Result<Value,
state.ref_map.clear();
state.iframe_sessions.clear();
state.active_frame_id = None;
// `--reuse-tab`: if a tab already shows this URL (same origin+path), switch
// to it instead of navigating — preserves any in-page state and stops
// re-`open` from piling up duplicate tabs on rebind (issue #21).
if cmd
.get("reuseTab")
.and_then(|v| v.as_bool())
.unwrap_or(false)
{
if let Ok(Some(switched)) = mgr.reuse_tab_for_url(url).await {
return Ok(with_site_hint(switched, url));
}
}
let result = mgr.navigate(url, wait_until).await?;
// Adaptive humanize: sample the freshly loaded page for known behavioural
// anti-bot vendors and escalate this session to Human if any are present.
detect_and_set_humanize(mgr).await;
Ok(result)
Ok(with_site_hint(result, url))
}
/// Annotate a navigation/snapshot result with the `site` adapters available for
/// the page's domain (auto-trigger): when you land on e.g. github.com, the
/// response carries `siteAdapters: { domain, commands: ["github/issues", …] }` so
/// the agent reaches for a structured-data adapter instead of scraping. No-op
/// when nothing matches or the pack isn't synced yet.
fn with_site_hint(mut result: Value, fallback_url: &str) -> Value {
let url = result
.get("url")
.and_then(|v| v.as_str())
.unwrap_or(fallback_url);
let host = url::Url::parse(url)
.ok()
.and_then(|u| u.host_str().map(String::from));
if let Some(host) = host {
let adapters = crate::site::adapters_for_domain(&host);
if !adapters.is_empty() {
if let Some(obj) = result.as_object_mut() {
obj.insert(
"siteAdapters".to_string(),
json!({ "domain": host, "commands": adapters }),
);
}
}
}
result
}
/// After navigation, probe the page for known anti-bot vendor fingerprints
@@ -2681,6 +2729,47 @@ async fn handle_evaluate(cmd: &Value, state: &DaemonState) -> Result<Value, Stri
Ok(json!({ "result": result, "origin": url }))
}
/// Run a site adapter: navigate to its `@meta.domain` (only if we're not already
/// there — the point is to run as you, in the page that's already open) and eval
/// the adapter function in the site's own logged-in page. The CLI/commands.rs has
/// already loaded the adapter and built the `script`; here we just place the page
/// and evaluate. Never disrupts the user's foreground tab — navigation happens on
/// the daemon's own tab (same as every other command on the relay).
async fn handle_site(cmd: &Value, state: &mut DaemonState) -> Result<Value, String> {
let domain = cmd
.get("domain")
.and_then(|v| v.as_str())
.ok_or("site: missing 'domain'")?
.to_string();
let script = cmd
.get("script")
.and_then(|v| v.as_str())
.ok_or("site: missing 'script'")?
.to_string();
let current = match state.browser.as_ref() {
Some(mgr) => mgr.get_url().await.unwrap_or_default(),
None => String::new(),
};
let on_domain = url::Url::parse(&current)
.ok()
.and_then(|u| u.host_str().map(|h| h.to_string()))
.map(|h| h == domain || h.ends_with(&format!(".{domain}")))
.unwrap_or(false);
if !on_domain {
let nav = json!({ "url": format!("https://{domain}/") });
handle_navigate(&nav, state).await?;
}
let eval_cmd = json!({ "script": script });
let out = handle_evaluate(&eval_cmd, state).await?;
Ok(json!({
"result": out.get("result").cloned().unwrap_or(Value::Null),
"origin": out.get("origin").cloned().unwrap_or(Value::Null),
"domain": domain,
}))
}
/// Local stealth self-check: reports the active mode, live fingerprint probes,
/// and the list of applied overrides — so an agent (or human) can confirm
/// stealth is working without driving an external detector, and audit exactly
@@ -2874,7 +2963,48 @@ async fn handle_snapshot(cmd: &Value, state: &mut DaemonState) -> Result<Value,
})
.collect();
Ok(json!({ "snapshot": tree, "origin": url, "refs": refs }))
let ref_count = refs.len();
let mut out = json!({ "snapshot": tree, "origin": url, "refs": refs });
// Auto-trigger: if this domain has site adapters, surface them so the agent
// pulls structured data instead of walking the tree. `with_site_hint` reads
// the `url` field, so pass it under that key.
if let Some(hint) = with_site_hint(json!({ "url": url }), &url).get("siteAdapters") {
out["siteAdapters"] = hint.clone();
}
// Canvas/WebGL apps (games, map/3D viewers, drawing tools) paint to a
// <canvas> and expose almost no accessibility tree, so `snapshot` comes back
// near-empty and agents get stuck looking for refs that will never exist
// (dogfood: the Dead Cell game). When the tree is sparse but a canvas
// dominates the viewport, tell them to switch to the screenshot-driven path.
if ref_count < 3 {
let canvas_js =
"(() => { const c = document.querySelector('canvas'); if (!c) return false; \
const r = c.getBoundingClientRect(); \
return r.width * r.height > innerWidth * innerHeight * 0.5; })()";
if let Ok(v) = mgr.evaluate(canvas_js, None).await {
if v.as_bool() == Some(true) {
out["note"] = json!(
"This page renders to a <canvas> (game / WebGL / editor) and exposes almost no \
accessibility tree refs won't help. Use `screenshot` to see it, coordinate \
`click <x> <y>` to interact, and `keydown`/`keyup`/`press` for keyboard \
(hold-to-move: `keydown d` `keyup d`)."
);
}
}
}
Ok(out)
}
/// Resolve a (possibly relative) saved-file path to an absolute one so the CLI
/// echoes a path the agent can read regardless of the process cwd (issue #16).
/// Falls back to the original string if the file can't be canonicalized.
fn absolutize_saved_path(p: &str) -> String {
std::fs::canonicalize(p)
.map(|c| c.to_string_lossy().into_owned())
.unwrap_or_else(|_| p.to_string())
}
async fn handle_screenshot(cmd: &Value, state: &mut DaemonState) -> Result<Value, String> {
@@ -2902,7 +3032,7 @@ async fn handle_screenshot(cmd: &Value, state: &mut DaemonState) -> Result<Value
.map_err(|e| format!("Base64 decode error: {}", e))?;
std::fs::write(p, bytes)
.map_err(|e| format!("Failed to write screenshot: {}", e))?;
return Ok(json!({ "path": p }));
return Ok(json!({ "path": absolutize_saved_path(p) }));
}
let tmp = format!(
"/tmp/screenshot-{}.png",
@@ -2949,6 +3079,14 @@ async fn handle_screenshot(cmd: &Value, state: &mut DaemonState) -> Result<Value
.get("screenshotDir")
.and_then(|v| v.as_str())
.map(String::from),
clip: cmd.get("clip").and_then(|c| {
Some((
c.get("x")?.as_f64()?,
c.get("y")?.as_f64()?,
c.get("width")?.as_f64()?,
c.get("height")?.as_f64()?,
))
}),
};
if annotate {
@@ -2976,7 +3114,7 @@ async fn handle_screenshot(cmd: &Value, state: &mut DaemonState) -> Result<Value
)
.await?;
let mut response = json!({ "path": result.path });
let mut response = json!({ "path": absolutize_saved_path(&result.path) });
if !result.annotations.is_empty() {
response["annotations"] = serde_json::to_value(&result.annotations)
.map_err(|e| format!("Failed to serialize annotations: {}", e))?;
@@ -3068,6 +3206,15 @@ async fn handle_click(cmd: &Value, state: &mut DaemonState) -> Result<Value, Str
let button = cmd.get("button").and_then(|v| v.as_str()).unwrap_or("left");
let click_count = cmd.get("clickCount").and_then(|v| v.as_i64()).unwrap_or(1) as i32;
let follow = cmd.get("follow").and_then(|v| v.as_bool()).unwrap_or(false);
// Snapshot tracked targets so we can tell if this click opened a NEW tab
// (target=_blank link / window.open). On the relay the new tab is discovered
// passively and doesn't steal focus (#7/#8.1), so without surfacing it the
// post-click snapshot shows the OLD page and looks like the click failed
// (issue #24-A).
let before: std::collections::HashSet<String> =
mgr.pages_list().into_iter().map(|p| p.target_id).collect();
interaction::click(
&mgr.client,
@@ -3080,7 +3227,26 @@ async fn handle_click(cmd: &Value, state: &mut DaemonState) -> Result<Value, Str
)
.await?;
Ok(json!({ "clicked": selector }))
// Give a just-opened tab a moment to register, then look for it.
let mgr = state.browser.as_mut().ok_or("Browser not launched")?;
tokio::time::sleep(std::time::Duration::from_millis(150)).await;
let opened = mgr.adopt_newly_opened(&before).await;
let mut out = json!({ "clicked": selector });
if let Some(page) = opened {
let tab_id = super::browser::format_tab_id(page.tab_id);
out["openedTab"] = json!({ "tabId": tab_id, "url": page.url, "title": page.title });
// `--follow`: switch the active tab to the newly-opened one (default is
// to report it but stay put, so multi-tab flows aren't hijacked).
if follow {
state.ref_map.clear();
state.iframe_sessions.clear();
state.active_frame_id = None;
let _ = mgr.tab_switch_by_id(page.tab_id).await;
out["followed"] = json!(true);
}
}
Ok(out)
}
async fn handle_dblclick(cmd: &Value, state: &mut DaemonState) -> Result<Value, String> {
@@ -3138,6 +3304,14 @@ async fn handle_type(cmd: &Value, state: &mut DaemonState) -> Result<Value, Stri
let mgr = state.browser.as_ref().ok_or("Browser not launched")?;
let session_id = mgr.active_session_id()?.to_string();
// `--key-events`: dispatch real per-character keyDown/keyUp instead of
// Input.insertText, so autocomplete/combobox widgets that only react to key
// events fire (e.g. Google's address postal-code lookup) (issue #4/#36).
let key_events = cmd
.get("keyEvents")
.and_then(|v| v.as_bool())
.unwrap_or(false);
// `type --focused <text>`: type into the currently-focused element without a
// selector (custom widgets that move focus to a hidden input on open).
if cmd
@@ -3149,7 +3323,14 @@ async fn handle_type(cmd: &Value, state: &mut DaemonState) -> Result<Value, Stri
.get("text")
.and_then(|v| v.as_str())
.ok_or("Missing 'text' parameter")?;
interaction::type_text_into_active_context(&mgr.client, &session_id, text, None).await?;
interaction::type_text_into_active_context(
&mgr.client,
&session_id,
text,
None,
key_events,
)
.await?;
return Ok(json!({ "typed": text, "focused": true }));
}
@@ -3173,6 +3354,7 @@ async fn handle_type(cmd: &Value, state: &mut DaemonState) -> Result<Value, Stri
clear,
delay,
&state.iframe_sessions,
key_events,
)
.await?;
Ok(json!({ "typed": text }))
@@ -3286,6 +3468,16 @@ async fn handle_press(cmd: &Value, state: &mut DaemonState) -> Result<Value, Str
// Parse modifier+key chords like "Control+a", "Shift+Enter", "Control+Shift+a"
let (actual_key, modifiers) = parse_key_chord(key);
// `--hold <ms>`: keyDown, wait, keyUp — all inside the daemon so the hold
// duration is precise (no shell-sleep / round-trip jitter). For games
// (hold-to-move/charge) and any press-and-hold interaction.
if let Some(ms) = cmd.get("hold").and_then(|v| v.as_u64()) {
interaction::dispatch_single_key(&mgr.client, &session_id, &actual_key, "keyDown").await?;
tokio::time::sleep(std::time::Duration::from_millis(ms)).await;
interaction::dispatch_single_key(&mgr.client, &session_id, &actual_key, "keyUp").await?;
return Ok(json!({ "pressed": key, "heldMs": ms }));
}
interaction::press_key_with_modifiers(&mgr.client, &session_id, &actual_key, modifiers).await?;
Ok(json!({ "pressed": key }))
}
@@ -3370,17 +3562,182 @@ async fn handle_scroll(cmd: &Value, state: &mut DaemonState) -> Result<Value, St
}
}
// An explicit `--selector` keeps the precise element-scroll path (scrollBy on
// the resolved node, same-origin only).
if let Some(sel) = selector {
interaction::scroll(
&mgr.client,
&session_id,
&state.ref_map,
Some(sel),
dx,
dy,
&state.iframe_sessions,
)
.await?;
return Ok(json!({ "scrolled": true, "via": "selector" }));
}
// `--at x,y` / `--frame n`: dispatch a real (isTrusted) wheel at a viewport
// coordinate. This hits the compositor and scrolls whatever scroll container
// is under the pointer — including cross-origin iframes that `window.scrollBy`
// on the top document silently no-ops on (issue #36).
if cmd.get("at").is_some() || cmd.get("frame").is_some() {
let (x, y, via) = if let Some(at) = cmd.get("at").and_then(|v| v.as_array()) {
let x = at.first().and_then(|v| v.as_f64()).unwrap_or(0.0);
let y = at.get(1).and_then(|v| v.as_f64()).unwrap_or(0.0);
(x, y, "at")
} else {
let n = cmd.get("frame").and_then(|v| v.as_u64()).unwrap_or(0);
let (x, y) = frame_center(mgr, &session_id, &state.iframe_sessions, n as usize).await?;
(x, y, "frame")
};
dispatch_wheel(&mgr.client, &session_id, x, y, dx, dy).await?;
return Ok(json!({ "scrolled": true, "via": via, "at": [x, y] }));
}
// Default (no selector/at/frame): scroll the page with `window.scrollBy`. This
// is the reliable path for ordinary page scrolling; a coordinate wheel at the
// viewport centre is NOT a dependable substitute (it no-ops on some pages,
// e.g. headless), so the wheel stays opt-in via `--at`/`--frame` for the
// cross-origin-iframe case (issue #36).
interaction::scroll(
&mgr.client,
&session_id,
&state.ref_map,
selector,
None,
dx,
dy,
&state.iframe_sessions,
)
.await?;
Ok(json!({ "scrolled": true }))
Ok(json!({ "scrolled": true, "via": "page" }))
}
/// Viewport center in CSS pixels, used as the default wheel landing point for
/// `scroll` (issue #36). Falls back to a sane 640×400 center if the page can't
/// be evaluated (e.g. a restricted document).
async fn viewport_center(mgr: &BrowserManager, session_id: &str) -> Result<(f64, f64), String> {
let dims = mgr
.client
.send_command_typed::<_, Value>(
"Runtime.evaluate",
&super::cdp::types::EvaluateParams {
expression: "[window.innerWidth, window.innerHeight]".to_string(),
return_by_value: Some(true),
await_promise: Some(false),
},
Some(session_id),
)
.await
.ok();
let arr = dims
.as_ref()
.and_then(|v| v.get("result"))
.and_then(|v| v.get("value"))
.and_then(|v| v.as_array());
let w = arr
.and_then(|a| a.first())
.and_then(|v| v.as_f64())
.filter(|w| *w > 0.0)
.unwrap_or(1280.0);
let h = arr
.and_then(|a| a.get(1))
.and_then(|v| v.as_f64())
.filter(|h| *h > 0.0)
.unwrap_or(800.0);
Ok((w / 2.0, h / 2.0))
}
/// Center of the `n`-th frame (as listed by `chrome-use frames`) in top-viewport
/// CSS pixels, so `scroll --frame n` lands its wheel inside a cross-origin iframe
/// without needing a selector into it (issue #36). Resolves the frame's owning
/// `<iframe>` element box via `DOM.getFrameOwner` + `DOM.getBoxModel` — exact for
/// a frame nested directly under the top document; for a deeper nesting the box is
/// relative to the intermediate frame, so prefer `--at x,y` from a screenshot.
async fn frame_center(
mgr: &BrowserManager,
session_id: &str,
iframe_sessions: &HashMap<String, String>,
n: usize,
) -> Result<(f64, f64), String> {
let frames =
super::element::collect_all_frames_text(&mgr.client, session_id, iframe_sessions).await?;
let frame = frames.get(n).ok_or_else(|| {
format!(
"frame index {} out of range (run `chrome-use frames`: {} frame(s))",
n,
frames.len()
)
})?;
if n == 0 {
// Frame 0 is the top document — there's no owner element; scroll its center.
return viewport_center(mgr, session_id).await;
}
let owner = mgr
.client
.send_command_typed::<_, Value>(
"DOM.getFrameOwner",
&json!({ "frameId": frame.frame_id }),
Some(session_id),
)
.await
.map_err(|e| format!("can't locate frame {}'s owner element: {}", n, e))?;
let backend_node_id = owner
.get("backendNodeId")
.and_then(|v| v.as_i64())
.ok_or_else(|| format!("frame {} has no owner <iframe> element", n))?;
let box_model = mgr
.client
.send_command_typed::<_, Value>(
"DOM.getBoxModel",
&json!({ "backendNodeId": backend_node_id }),
Some(session_id),
)
.await
.map_err(|e| format!("can't measure frame {}'s box: {}", n, e))?;
let content = box_model
.get("model")
.and_then(|m| m.get("content"))
.and_then(|c| c.as_array())
.ok_or_else(|| format!("frame {} box model has no content quad", n))?;
let coord = |i: usize| content.get(i).and_then(|v| v.as_f64()).unwrap_or(0.0);
// content quad is [x1,y1, x2,y2, x3,y3, x4,y4]; opposite corners are 0 and 2.
let cx = (coord(0) + coord(4)) / 2.0;
let cy = (coord(1) + coord(5)) / 2.0;
Ok((cx, cy))
}
/// Dispatch a trusted mouse wheel at `(x, y)`, humanized like `handle_wheel`.
async fn dispatch_wheel(
client: &super::cdp::client::CdpClient,
session_id: &str,
x: f64,
y: f64,
delta_x: f64,
delta_y: f64,
) -> Result<(), String> {
let level = humanize::active_level();
let seed = humanize::next_seed();
for (dx, dy, delay) in humanize::scroll_segments(delta_x, delta_y, level, seed) {
client
.send_command(
"Input.dispatchMouseEvent",
Some(json!({
"type": "mouseWheel",
"x": x,
"y": y,
"deltaX": dx,
"deltaY": dy,
})),
Some(session_id),
)
.await?;
if !delay.is_zero() {
tokio::time::sleep(delay).await;
}
}
Ok(())
}
async fn handle_select(cmd: &Value, state: &mut DaemonState) -> Result<Value, String> {
@@ -3507,6 +3864,56 @@ async fn handle_wait(cmd: &Value, state: &mut DaemonState) -> Result<Value, Stri
async fn handle_gettext(cmd: &Value, state: &mut DaemonState) -> Result<Value, String> {
let mgr = state.browser.as_ref().ok_or("Browser not launched")?;
let session_id = mgr.active_session_id()?.to_string();
// `get text --all-frames` aggregates visible text across every frame the
// session can reach — including out-of-process iframes that never show up
// in the top document (#27: Yahoo/Rakuten/Mercari listing descriptions).
if cmd.get("allFrames").and_then(|v| v.as_bool()) == Some(true) {
let frames = super::element::collect_all_frames_text(
&mgr.client,
&session_id,
&state.iframe_sessions,
)
.await?;
let mut combined = String::new();
let mut frame_count = 0usize;
for f in &frames {
let t = f.text.trim();
if t.is_empty() {
continue;
}
frame_count += 1;
if f.kind != "top" {
combined.push_str(&format!("\n\n----- frame [{}] {} -----\n", f.kind, f.url));
}
combined.push_str(t);
}
let url = mgr.get_url().await.unwrap_or_default();
return Ok(json!({
"text": combined,
"origin": url,
"frames": frame_count,
"allFrames": true,
}));
}
// `get text --pierce` reads text through CLOSED shadow roots and child
// documents via the CDP DOM tree — content `innerText`/`eval` can't see,
// e.g. an extension's injected panel in a closed shadow DOM (#30).
if cmd.get("pierce").and_then(|v| v.as_bool()) == Some(true) {
let text = super::element::get_pierced_text(&mgr.client, &session_id).await?;
let url = mgr.get_url().await.unwrap_or_default();
return Ok(json!({ "text": text, "origin": url, "pierce": true }));
}
// `get text --main` returns the page's main-content region (readability-lite),
// skipping global header/nav/footer/sidebar boilerplate (#27).
if cmd.get("main").and_then(|v| v.as_bool()) == Some(true) {
let text = super::element::get_main_content_text(&mgr.client, &session_id).await?;
let url = mgr.get_url().await.unwrap_or_default();
return Ok(json!({ "text": text, "origin": url, "main": true }));
}
let selector = cmd
.get("selector")
.and_then(|v| v.as_str())
@@ -3524,6 +3931,29 @@ async fn handle_gettext(cmd: &Value, state: &mut DaemonState) -> Result<Value, S
Ok(json!({ "text": text, "origin": url }))
}
async fn handle_frames(_cmd: &Value, state: &mut DaemonState) -> Result<Value, String> {
let mgr = state.browser.as_ref().ok_or("Browser not launched")?;
let session_id = mgr.active_session_id()?.to_string();
let frames =
super::element::collect_all_frames_text(&mgr.client, &session_id, &state.iframe_sessions)
.await?;
let list: Vec<Value> = frames
.iter()
.enumerate()
.map(|(i, f)| {
json!({
"index": i,
"kind": f.kind,
"url": f.url,
"frameId": f.frame_id,
"textLen": f.text.trim().chars().count(),
})
})
.collect();
let url = mgr.get_url().await.unwrap_or_default();
Ok(json!({ "frames": list, "count": list.len(), "origin": url }))
}
async fn handle_getattribute(cmd: &Value, state: &mut DaemonState) -> Result<Value, String> {
let mgr = state.browser.as_ref().ok_or("Browser not launched")?;
let session_id = mgr.active_session_id()?.to_string();
@@ -3958,6 +4388,111 @@ async fn handle_cookies_clear(state: &DaemonState) -> Result<Value, String> {
Ok(json!({ "cleared": true }))
}
// Detect whether the active page is *currently* a Cloudflare challenge
// (the full-page "Just a moment…" / "正在进行安全验证" interstitial), so an agent
// knows whether it must solve or can proceed. Runs in the top frame main world.
const CF_CHALLENGE_JS: &str = r#"(function(){
var t = document.title || '';
var challenged =
/just a moment|attention required|checking (your|if)|verify you are human|||||/i.test(t) ||
!!document.querySelector('#challenge-form, #challenge-running, #cf-challenge-running, [id^="cf-chl"], script[src*="/cdn-cgi/challenge-platform/"]');
var turnstile = !!document.querySelector('.cf-turnstile, [data-sitekey]');
return JSON.stringify({ title: t, challenged: challenged, turnstile: turnstile, readyState: document.readyState });
})()"#;
/// Recommendation for a Cloudflare-gated page, from the current challenge state
/// and whether a still-valid `cf_clearance` exists. Pure so it's unit-testable.
/// - not challenged → "proceed" (the page is cleared/loaded)
/// - challenged, valid cookie → "reissue" (clearance present but page still
/// blocks → it's stale or the IP/UA no longer matches what it was issued for)
/// - challenged, no cookie → "solve"
fn cf_recommendation(challenged: bool, clearance_valid: bool) -> &'static str {
if !challenged {
"proceed"
} else if clearance_valid {
"reissue"
} else {
"solve"
}
}
/// `cf_clearance` validity for a cookie's expiry (epoch seconds; <=0 = session
/// cookie, treated as non-expiring). Returns (present, expired). Pure.
fn clearance_state(expires: Option<f64>, now: f64) -> (bool, bool) {
match expires {
None => (false, false),
Some(e) if e <= 0.0 => (true, false), // session cookie: no expiry
Some(e) => (true, e < now),
}
}
async fn handle_cf_status(_cmd: &Value, state: &mut DaemonState) -> Result<Value, String> {
let mgr = state.browser.as_ref().ok_or("Browser not launched")?;
let session_id = mgr.active_session_id()?.to_string();
let url = mgr.get_url().await.unwrap_or_default();
// 1. Is the page a Cloudflare challenge right now?
let probe_raw = mgr
.evaluate(CF_CHALLENGE_JS, None)
.await
.unwrap_or(Value::Null);
let probe = parse_json_string(probe_raw, "cf challenge probe").unwrap_or(Value::Null);
let challenged = probe
.get("challenged")
.and_then(|v| v.as_bool())
.unwrap_or(false);
let turnstile = probe
.get("turnstile")
.and_then(|v| v.as_bool())
.unwrap_or(false);
let title = probe
.get("title")
.and_then(|v| v.as_str())
.unwrap_or("")
.to_string();
// 2. Persistence artifacts: cf_clearance (HttpOnly → must read via CDP, not
// document.cookie) + CF_VERIFIED_DEVICE. Scope to the current URL.
let urls = if url.is_empty() {
None
} else {
Some(vec![url.clone()])
};
let cookies = super::cookies::get_cookies(&mgr.client, &session_id, urls)
.await
.unwrap_or_default();
let clearance = cookies.iter().find(|c| c.name == "cf_clearance");
let now = std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.map(|d| d.as_secs_f64())
.unwrap_or(0.0);
let (present, expired) = clearance_state(clearance.map(|c| c.expires), now);
let expires_in = clearance
.filter(|_| present && !expired)
.map(|c| (c.expires - now).max(0.0) as i64);
let device_verified = cookies
.iter()
.any(|c| c.name.starts_with("CF_VERIFIED_DEVICE"));
let clearance_valid = present && !expired;
let recommendation = cf_recommendation(challenged, clearance_valid);
Ok(json!({
"url": url,
"title": title,
"challenged": challenged,
"turnstile": turnstile,
"clearance": {
"present": present,
"expired": expired,
"expiresIn": expires_in,
"httpOnly": clearance.map(|c| c.http_only).unwrap_or(false),
},
"deviceVerified": device_verified,
"recommendation": recommendation,
}))
}
async fn handle_storage_get(cmd: &Value, state: &DaemonState) -> Result<Value, String> {
let mgr = state.browser.as_ref().ok_or("Browser not launched")?;
let session_id = mgr.active_session_id()?.to_string();
@@ -4295,8 +4830,18 @@ async fn handle_keyboard(cmd: &Value, state: &DaemonState) -> Result<Value, Stri
.get("text")
.and_then(|v| v.as_str())
.ok_or("Missing 'text' parameter")?;
interaction::type_text_into_active_context(&mgr.client, &session_id, text, None)
.await?;
let key_events = cmd
.get("keyEvents")
.and_then(|v| v.as_bool())
.unwrap_or(false);
interaction::type_text_into_active_context(
&mgr.client,
&session_id,
text,
None,
key_events,
)
.await?;
return Ok(json!({ "typed": text }));
}
Some("insertText") => {
@@ -4346,10 +4891,19 @@ async fn handle_keyboard(cmd: &Value, state: &DaemonState) -> Result<Value, Stri
// Phase 5 handlers
// ---------------------------------------------------------------------------
async fn handle_tab_list(state: &DaemonState) -> Result<Value, String> {
let mgr = state.browser.as_ref().ok_or("Browser not launched")?;
async fn handle_tab_list(cmd: &Value, state: &mut DaemonState) -> Result<Value, String> {
let mgr = state.browser.as_mut().ok_or("Browser not launched")?;
// Re-sync with the live browser so the list reflects tabs opened by other
// sessions or re-attached after a cross-process nav, and drops gone ones
// (issue #21). Best-effort: a stale list still beats erroring the command.
mgr.resync_targets().await.ok();
let tabs = mgr.tab_list();
Ok(json!({ "tabs": tabs }))
// Echo `full` so the formatter prints untruncated URLs (issue #19).
if cmd.get("full").and_then(|v| v.as_bool()).unwrap_or(false) {
Ok(json!({ "tabs": tabs, "full": true }))
} else {
Ok(json!({ "tabs": tabs }))
}
}
async fn handle_tab_new(cmd: &Value, state: &mut DaemonState) -> Result<Value, String> {
@@ -4380,13 +4934,50 @@ async fn handle_tab_switch(cmd: &Value, state: &mut DaemonState) -> Result<Value
let tab_ref_str = cmd
.get("tabId")
.and_then(|v| v.as_str())
.ok_or("Missing 'tabId' parameter (expected `t<N>` or a label)")?;
let tab_ref = super::browser::TabRef::parse(tab_ref_str)?;
let tab_id = mgr.resolve_tab_ref(&tab_ref)?;
.ok_or("Missing 'tabId' parameter (expected `t<N>`, a label, or a targetId)")?;
// Re-sync first so a tab opened by another session, or one that re-attached
// after a cross-process nav, is adoptable from here (issue #21).
mgr.resync_targets().await.ok();
// A CDP `targetId` (shown in `tab list`) is stable across sessions, so accept
// it directly for adopting a specific pre-existing tab — falling back to the
// per-session `t<N>` / label form.
let tab_id = match mgr.tab_id_for_target(tab_ref_str) {
Some(id) => id,
None => {
let tab_ref = super::browser::TabRef::parse(tab_ref_str)?;
mgr.resolve_tab_ref(&tab_ref)?
}
};
state.ref_map.clear();
state.iframe_sessions.clear();
state.active_frame_id = None;
let result = mgr.tab_switch_by_id(tab_id).await?;
let mut result = mgr.tab_switch_by_id(tab_id).await?;
// Liveness probe: confirm the new session actually answers before we report
// success, so `tab <id>` doesn't print a misleading ✓ for a session that's
// stale and will fail on the very next command (issue #29.3). On the churned
// -tabId case the ext-0.4.9 targetId recovery (#24) self-heals within ~6s, so
// we surface a warning rather than a hard error to avoid a false failure
// during that window.
if mgr.evaluate("1", None).await.is_err() {
if let Some(obj) = result.as_object_mut() {
obj.insert(
"warning".to_string(),
json!("switched tab is not responding yet (session re-attaching); retry the next command"),
);
}
}
// `--activate`: raise this tab to the foreground (the switch made it active;
// bring_to_front acts on the active tab) — for handing a specific tab to the
// human (issue #24-C). Best-effort; don't fail the switch if it can't.
if cmd
.get("activate")
.and_then(|v| v.as_bool())
.unwrap_or(false)
{
let _ = mgr.bring_to_front().await;
}
if let Some(ref server) = state.stream_server {
if let Ok(dims) = mgr
@@ -5216,6 +5807,18 @@ async fn handle_bringtofront(state: &DaemonState) -> Result<Value, String> {
Ok(json!({ "broughtToFront": true }))
}
async fn handle_current(state: &mut DaemonState) -> Result<Value, String> {
let mgr = state.browser.as_mut().ok_or("Browser not launched")?;
// Refresh so `current` reflects the live URL/title even after a cross-process
// nav (the relay's cached target_info can lag) (#26).
mgr.resync_targets().await.ok();
let mut info = mgr.active_page_info().ok_or("No active tab")?;
if let Some(obj) = info.as_object_mut() {
obj.insert("current".to_string(), json!(true));
}
Ok(info)
}
async fn handle_timezone(cmd: &Value, state: &DaemonState) -> Result<Value, String> {
let mgr = state.browser.as_ref().ok_or("Browser not launched")?;
let timezone = cmd
@@ -6767,6 +7370,28 @@ async fn handle_drag(cmd: &Value, state: &mut DaemonState) -> Result<Value, Stri
.and_then(|v| v.as_str())
.ok_or("Missing 'target' parameter")?;
// Over the relay (or into an iframe) a coordinate drag drifts to the
// foreground tab and can't reach an OOPIF — DOM-dispatch an HTML5 drag in the
// element's own session instead (issues #31/#36). `coord` mode forces the
// coordinate path for pointer-driven drags (canvas/sliders) on a launched
// browser.
if std::env::var("AGENT_BROWSER_CLICK_MODE").as_deref() != Ok("coord")
&& (crate::connect::relay_url().is_some()
|| state.ref_map.ref_is_in_iframe(source)
|| state.ref_map.ref_is_in_iframe(target))
{
super::interaction::dom_drag(
&mgr.client,
&session_id,
&state.ref_map,
source,
target,
&state.iframe_sessions,
)
.await?;
return Ok(json!({ "dragged": { "source": source, "target": target }, "via": "dom" }));
}
let (sx, sy, _, _, source_session_id) = super::element::resolve_element_center(
&mgr.client,
&session_id,
@@ -7121,6 +7746,7 @@ async fn handle_diff_screenshot(cmd: &Value, state: &DaemonState) -> Result<Valu
quality: None,
annotate: false,
output_dir: None,
clip: None,
};
let result = screenshot::take_screenshot(
@@ -8717,13 +9343,7 @@ async fn handle_keydown(cmd: &Value, state: &DaemonState) -> Result<Value, Strin
.and_then(|v| v.as_str())
.ok_or("Missing 'key' parameter")?;
mgr.client
.send_command(
"Input.dispatchKeyEvent",
Some(json!({ "type": "keyDown", "key": key })),
Some(&session_id),
)
.await?;
interaction::dispatch_single_key(&mgr.client, &session_id, key, "keyDown").await?;
Ok(json!({ "keydown": key }))
}
@@ -8735,13 +9355,7 @@ async fn handle_keyup(cmd: &Value, state: &DaemonState) -> Result<Value, String>
.and_then(|v| v.as_str())
.ok_or("Missing 'key' parameter")?;
mgr.client
.send_command(
"Input.dispatchKeyEvent",
Some(json!({ "type": "keyUp", "key": key })),
Some(&session_id),
)
.await?;
interaction::dispatch_single_key(&mgr.client, &session_id, key, "keyUp").await?;
Ok(json!({ "keyup": key }))
}
@@ -8859,6 +9473,26 @@ mod tests {
use crate::test_utils::EnvGuard;
use std::fs;
#[test]
fn test_cf_recommendation() {
assert_eq!(cf_recommendation(false, false), "proceed");
assert_eq!(cf_recommendation(false, true), "proceed");
assert_eq!(cf_recommendation(true, false), "solve");
assert_eq!(cf_recommendation(true, true), "reissue");
}
#[test]
fn test_clearance_state() {
// no cookie
assert_eq!(clearance_state(None, 1000.0), (false, false));
// session cookie (expires <= 0) → present, never expired
assert_eq!(clearance_state(Some(-1.0), 1000.0), (true, false));
// valid: expiry in the future
assert_eq!(clearance_state(Some(2000.0), 1000.0), (true, false));
// expired: expiry in the past
assert_eq!(clearance_state(Some(500.0), 1000.0), (true, true));
}
#[test]
fn test_url_glob_to_regex() {
assert_eq!(url_glob_to_regex("**/dashboard"), "^.*/dashboard$");
+1007 -32
View File
File diff suppressed because it is too large Load Diff
+59
View File
@@ -341,6 +341,46 @@ fn build_chrome_args(options: &LaunchOptions) -> Result<ChromeArgs, String> {
})
}
/// Cross-process advisory lock that serializes concurrent launches of the SAME
/// Chrome profile (issue #11). Held via `flock` on a per-profile lock file; the
/// kernel releases it automatically when the holding process exits, so a crash
/// can't wedge the queue. Best-effort: if the lock can't be acquired the launch
/// proceeds unlocked rather than failing.
struct ProfileLaunchLock {
#[cfg(unix)]
_file: std::fs::File,
}
impl ProfileLaunchLock {
fn acquire(profile: &str) -> Option<Self> {
let safe: String = profile
.chars()
.map(|c| if c.is_alphanumeric() { c } else { '_' })
.collect();
let path = std::env::temp_dir().join(format!("chrome-use-launch-{safe}.lock"));
let file = std::fs::OpenOptions::new()
.create(true)
.write(true)
.truncate(false)
.open(&path)
.ok()?;
#[cfg(unix)]
{
use std::os::unix::io::AsRawFd;
// Blocking exclusive lock: concurrent same-profile launches queue.
if unsafe { libc::flock(file.as_raw_fd(), libc::LOCK_EX) } != 0 {
return None;
}
Some(ProfileLaunchLock { _file: file })
}
#[cfg(not(unix))]
{
let _ = file;
Some(ProfileLaunchLock {})
}
}
}
pub fn launch_chrome(options: &LaunchOptions) -> Result<ChromeProcess, String> {
let chrome_path = match &options.executable_path {
Some(p) => PathBuf::from(p),
@@ -363,6 +403,13 @@ pub fn launch_chrome(options: &LaunchOptions) -> Result<ChromeProcess, String> {
// rewrite options so the retry loop uses the copied profile.
let mut resolved_options: Option<LaunchOptions> = None;
let mut profile_temp_dir: Option<PathBuf> = None;
// Serialize concurrent launches of the SAME named profile across processes
// (issue #11). Without this, N parallel `open --profile <same>` collide on
// the profile-copy disk I/O / Chrome's profile lock, every candidate burns
// its full launch timeout, and all fail. The flock queues them instead and
// auto-releases on process exit, so a crash can't wedge the queue. Held
// until Chrome is up (function return).
let mut _launch_lock: Option<ProfileLaunchLock> = None;
if let Some(ref profile) = options.profile {
if is_chrome_profile_name(profile) {
@@ -372,6 +419,7 @@ pub fn launch_chrome(options: &LaunchOptions) -> Result<ChromeProcess, String> {
.to_string()
})?;
let resolved = resolve_chrome_profile(&user_data_dir, profile)?;
_launch_lock = ProfileLaunchLock::acquire(&resolved);
let temp_path = copy_chrome_profile(&user_data_dir, &resolved)?;
let mut opts = options.clone();
@@ -1886,6 +1934,17 @@ mod tests {
assert!(is_chrome_profile_name(""));
}
#[test]
fn test_profile_launch_lock_acquires_and_sanitizes() {
// Uncontended acquire succeeds and writes a sanitized per-profile lock
// file (issue #11: serialize concurrent same-profile launches).
let lock = ProfileLaunchLock::acquire("Profile 5/weird:name");
assert!(lock.is_some(), "uncontended lock should acquire");
let expected = std::env::temp_dir().join("chrome-use-launch-Profile_5_weird_name.lock");
assert!(expected.exists(), "lock file should exist at {expected:?}");
drop(lock);
}
#[test]
fn test_is_chrome_profile_name_paths() {
assert!(!is_chrome_profile_name("/tmp/dir"));
+10
View File
@@ -21,6 +21,16 @@ pub async fn run_daemon(session: &str) {
// (via the ab-connect extension) land in a per-session Chrome tab group.
let _ = super::browser::DAEMON_SESSION.set(session.to_string());
// Bootstrap / refresh the site-adapter pack in the background (first-run +
// periodic TTL). This populates ~/.chrome-use/sites/.index.json so navigation
// can auto-suggest `site` commands for the page you land on, with zero added
// latency to any command. Best-effort; offline is a no-op.
if crate::site::needs_refresh() {
tokio::spawn(async {
let _ = crate::site::update().await;
});
}
let socket_dir = get_daemon_socket_dir();
if !socket_dir.exists() {
let _ = fs::create_dir_all(&socket_dir);
+71
View File
@@ -35,6 +35,7 @@ fn native_test_fixture_html(name: &str) -> &'static str {
"html5_drag_probe" => include_str!("test_fixtures/html5_drag_probe.html"),
"pointer_capture_probe" => include_str!("test_fixtures/pointer_capture_probe.html"),
"upload_probe" => include_str!("test_fixtures/upload_probe.html"),
"iframe_button_probe" => include_str!("test_fixtures/iframe_button_probe.html"),
_ => panic!("Unknown native test fixture: {}", name),
}
}
@@ -573,6 +574,76 @@ async fn e2e_snapshot_and_click_ref() {
assert_success(&resp);
}
/// Clicking a button INSIDE an iframe by `@ref` must deliver a TRUSTED activation
/// (`event.isTrusted === true`), not a synthetic DOM `.click()`. Security-sensitive
/// embedded forms (Google Payments' `保存`) reject `isTrusted:false` clicks, so an
/// enabled submit button silently no-op'd (issue #39). The fix routes iframe-ref
/// clicks to a real `Input.dispatchMouseEvent` on the element's own frame session.
/// The fixture's iframe button writes `clicked:<isTrusted>` into its own text on
/// click, which the cross-frame snapshot reads back.
#[tokio::test]
#[ignore]
async fn e2e_iframe_button_click_is_trusted() {
let mut state = DaemonState::new();
let resp = execute_command(
&json!({ "id": "1", "action": "launch", "headless": true }),
&mut state,
)
.await;
assert_success(&resp);
let resp = execute_command(
&json!({ "id": "2", "action": "navigate", "url": native_test_fixture_url("iframe_button_probe") }),
&mut state,
)
.await;
assert_success(&resp);
// Snapshot (interactive) — the button lives in the iframe and must appear with
// a ref; that ref carries the frame_id so the click resolves into the frame.
let resp = execute_command(
&json!({ "id": "3", "action": "snapshot", "interactive": true }),
&mut state,
)
.await;
assert_success(&resp);
let snapshot = get_data(&resp)["snapshot"].as_str().unwrap_or("");
let ref_id = snapshot
.lines()
.find(|l| l.contains("button \"save\""))
.and_then(|l| l.split("ref=").nth(1))
.map(|r| r.trim_end_matches(']').trim())
.unwrap_or_else(|| panic!("iframe button not found in snapshot:\n{snapshot}"));
// Click it by ref.
let resp = execute_command(
&json!({ "id": "4", "action": "click", "selector": ref_id }),
&mut state,
)
.await;
assert_success(&resp);
tokio::time::sleep(tokio::time::Duration::from_millis(300)).await;
// The button rewrote its own text with the click's isTrusted flag; read it
// back across frames.
let resp = execute_command(
&json!({ "id": "5", "action": "snapshot", "interactive": true }),
&mut state,
)
.await;
assert_success(&resp);
let after = get_data(&resp)["snapshot"].as_str().unwrap_or("");
assert!(
after.contains("clicked:true"),
"iframe button click must be trusted (isTrusted:true); snapshot:\n{after}"
);
let resp = execute_command(&json!({ "id": "99", "action": "close" }), &mut state).await;
assert_success(&resp);
}
// ---------------------------------------------------------------------------
// Screenshot
// ---------------------------------------------------------------------------
+351 -8
View File
@@ -100,6 +100,15 @@ impl RefMap {
self.map.get(ref_id)
}
/// Whether `selector_or_ref` is a `@ref` whose snapshot entry lives inside an
/// iframe (has a `frame_id`). Pointer interactions use this to choose
/// DOM-dispatch over coordinates for OOPIF elements (issue #36).
pub fn ref_is_in_iframe(&self, selector_or_ref: &str) -> bool {
parse_ref(selector_or_ref)
.and_then(|r| self.map.get(&r).map(|e| e.frame_id.is_some()))
.unwrap_or(false)
}
pub fn entries_sorted(&self) -> Vec<(String, RefEntry)> {
let mut entries = self
.map
@@ -796,16 +805,43 @@ pub(super) fn extract_ax_string(value: &Option<AXValue>) -> String {
/// Build a JS expression that finds a DOM element by CSS selector or XPath.
fn build_find_element_js(selector: &str) -> String {
if let Some(xpath) = selector.strip_prefix("xpath=") {
format!(
return format!(
"document.evaluate({}, document, null, XPathResult.FIRST_ORDERED_NODE_TYPE, null).singleNodeValue",
serde_json::to_string(xpath).unwrap_or_default()
)
} else {
format!(
"document.querySelector({})",
serde_json::to_string(selector).unwrap_or_default()
)
);
}
// Bare string (or explicit `text=`): try CSS first, then fall back to
// matching an interactive element by its VISIBLE TEXT. snapshot exposes
// buttons/links by their name, so `click "購入手続きへ"` should resolve by
// that label — previously it was fed straight to `querySelector` as CSS and
// failed as an invalid selector even though the button was right there
// (issue #24-B). CSS still wins when it matches, so existing selectors are
// unaffected; nested/non-ASCII labels now resolve too.
let text_only = selector.strip_prefix("text=");
let force_text = text_only.is_some();
let sel_json = serde_json::to_string(selector).unwrap_or_default();
let want_json = serde_json::to_string(text_only.unwrap_or(selector)).unwrap_or_default();
format!(
r#"(() => {{
const sel = {sel};
const css = {force_text} ? null : (() => {{ try {{ return document.querySelector(sel); }} catch (_e) {{ return null; }} }})();
if (css) return css;
const norm = s => (s == null ? '' : String(s)).replace(/\s+/g, ' ').trim();
const w = norm({want}); if (!w) return null;
const wl = w.toLowerCase();
const interactive = Array.from(document.querySelectorAll(
'button,a,[role=button],[role=link],[role=menuitem],[role=tab],[role=option],input[type=submit],input[type=button],input[type=reset],summary,label,[onclick]'));
const textOf = e => norm(e.innerText || e.textContent) || norm(e.value) ||
norm(e.getAttribute && e.getAttribute('aria-label')) || norm(e.getAttribute && e.getAttribute('title'));
let hit = interactive.find(e => textOf(e) === w) || interactive.find(e => textOf(e).toLowerCase().includes(wl));
if (hit) return hit;
const leaves = Array.from(document.querySelectorAll('*')).filter(e => !e.children.length);
return leaves.find(e => norm(e.textContent) === w) || leaves.find(e => norm(e.textContent).toLowerCase().includes(wl)) || null;
}})()"#,
sel = sel_json,
want = want_json,
force_text = force_text
)
}
/// Build a JS expression that counts matching DOM elements by CSS selector or XPath.
@@ -948,6 +984,268 @@ pub async fn get_element_text(
.unwrap_or_default())
}
/// Text content collected from a single frame of the page.
#[derive(Debug, Clone)]
pub struct FrameText {
pub frame_id: String,
pub url: String,
/// "top" | "inline" (same-process child frame) | "oopif" (out-of-process).
pub kind: &'static str,
pub text: String,
}
// The expression we run in every frame to read its visible text. innerText
// honors CSS visibility (skips display:none), textContent is the fallback.
const FRAME_INNERTEXT_JS: &str = "(function(){try{var b=document.body||document.documentElement;return b?(b.innerText||b.textContent||''):'';}catch(e){return '';}})()";
async fn eval_text_default(client: &CdpClient, session_id: &str) -> String {
let res = client
.send_command(
"Runtime.evaluate",
Some(serde_json::json!({
"expression": FRAME_INNERTEXT_JS,
"returnByValue": true,
})),
Some(session_id),
)
.await;
res.ok()
.and_then(|v| v.get("result").and_then(|r| r.get("value")).cloned())
.and_then(|v| v.as_str().map(|s| s.to_string()))
.unwrap_or_default()
}
// Same-process child frames share the top renderer but live in their own
// execution context. Page.createIsolatedWorld hands us a context id bound to
// that frame so Runtime.evaluate reads the child document, not the parent.
async fn eval_text_in_frame(client: &CdpClient, session_id: &str, frame_id: &str) -> String {
let ctx = client
.send_command(
"Page.createIsolatedWorld",
Some(serde_json::json!({ "frameId": frame_id, "worldName": "chrome_use_text" })),
Some(session_id),
)
.await
.ok()
.and_then(|v| v.get("executionContextId").and_then(|c| c.as_i64()));
let Some(ctx_id) = ctx else {
return String::new();
};
let res = client
.send_command(
"Runtime.evaluate",
Some(serde_json::json!({
"expression": FRAME_INNERTEXT_JS,
"returnByValue": true,
"contextId": ctx_id,
})),
Some(session_id),
)
.await;
res.ok()
.and_then(|v| v.get("result").and_then(|r| r.get("value")).cloned())
.and_then(|v| v.as_str().map(|s| s.to_string()))
.unwrap_or_default()
}
fn flatten_frame_tree(node: &Value, is_top: bool, out: &mut Vec<(String, String, bool)>) {
if let Some(frame) = node.get("frame") {
if let Some(id) = frame.get("id").and_then(|v| v.as_str()) {
let url = frame
.get("url")
.and_then(|v| v.as_str())
.unwrap_or("")
.to_string();
out.push((id.to_string(), url, is_top));
}
}
if let Some(children) = node.get("childFrames").and_then(|v| v.as_array()) {
for child in children {
flatten_frame_tree(child, false, out);
}
}
}
/// Collect visible text from every frame reachable in the active session,
/// including out-of-process iframes (which never appear in the top frame's
/// `Page.getFrameTree` and so are invisible to `document.body.innerText`).
///
/// Same-process child frames are read through `Page.createIsolatedWorld`;
/// OOPIFs are read through their own auto-attached debugger session
/// (`iframe_sessions`, keyed by frameId == targetId). This is the engine
/// behind `get text --all-frames` and `chrome-use frames` — the fix for
/// listing/marketplace pages whose description lives in a child frame (#27).
pub async fn collect_all_frames_text(
client: &CdpClient,
top_session: &str,
iframe_sessions: &HashMap<String, String>,
) -> Result<Vec<FrameText>, String> {
let mut out: Vec<FrameText> = Vec::new();
let mut seen: std::collections::HashSet<String> = std::collections::HashSet::new();
// 1. Top session: the top frame plus its same-process descendants. OOPIF
// frames that happen to surface here are skipped — they're read via
// their dedicated session in step 2 (cross-process isolated worlds fail).
let tree = client
.send_command_no_params("Page.getFrameTree", Some(top_session))
.await?;
let mut frames: Vec<(String, String, bool)> = Vec::new();
flatten_frame_tree(&tree["frameTree"], true, &mut frames);
for (fid, url, is_top) in frames {
if iframe_sessions.contains_key(&fid) {
continue;
}
if !seen.insert(fid.clone()) {
continue;
}
let (kind, text) = if is_top {
("top", eval_text_default(client, top_session).await)
} else {
(
"inline",
eval_text_in_frame(client, top_session, &fid).await,
)
};
out.push(FrameText {
frame_id: fid,
url,
kind,
text,
});
}
// 2. Each out-of-process iframe, read through its own session.
for (fid, sid) in iframe_sessions {
if !seen.insert(fid.clone()) {
continue;
}
let url = client
.send_command_no_params("Page.getFrameTree", Some(sid))
.await
.ok()
.and_then(|t| {
t.get("frameTree")
.and_then(|ft| ft.get("frame"))
.and_then(|f| f.get("url"))
.and_then(|v| v.as_str())
.map(|s| s.to_string())
})
.unwrap_or_default();
let text = eval_text_default(client, sid).await;
out.push(FrameText {
frame_id: fid.clone(),
url,
kind: "oopif",
text,
});
}
Ok(out)
}
// Readability-lite: prefer the page's semantic main-content region over the
// whole body so global header/nav/footer chrome (and, on many listing pages,
// the "related items" sidebar) doesn't drown out the actual content. Runs on
// the live, rendered tree (innerText needs layout — a detached clone returns
// empty), so we pick the densest <main>/<article> region rather than cloning
// and stripping. Falls back to <body> when no substantial main region exists.
const MAIN_CONTENT_JS: &str = r#"(function(){
function txt(el){try{return (el.innerText||'').trim();}catch(e){return '';}}
var sels=['main','[role=main]','article','#main','#contents','#l-content'];
var best=null,bestLen=0;
for(var i=0;i<sels.length;i++){
var els=document.querySelectorAll(sels[i]);
for(var j=0;j<els.length;j++){var l=txt(els[j]).length;if(l>bestLen){bestLen=l;best=els[j];}}
}
if(best&&bestLen>200)return txt(best);
return txt(document.body);
})()"#;
/// Extract the page's main-content text (readability-lite), preferring a
/// semantic `<main>`/`<article>` region over the full body. Used by
/// `get text --main` to avoid header/nav/sidebar boilerplate (#27).
pub async fn get_main_content_text(client: &CdpClient, session_id: &str) -> Result<String, String> {
let res = client
.send_command(
"Runtime.evaluate",
Some(serde_json::json!({
"expression": MAIN_CONTENT_JS,
"returnByValue": true,
})),
Some(session_id),
)
.await?;
Ok(res
.get("result")
.and_then(|r| r.get("value"))
.and_then(|v| v.as_str())
.unwrap_or_default()
.to_string())
}
// Text nodes whose parent is one of these carry no visible content.
fn is_noise_tag(name: &str) -> bool {
matches!(name, "SCRIPT" | "STYLE" | "NOSCRIPT" | "TEMPLATE" | "HEAD")
}
// Walk a CDP DOM.Node tree, collecting text-node values. Unlike `innerText`
// (JS, blocked by CLOSED shadow roots), the CDP DOM tree from
// `DOM.getDocument(pierce:true)` includes closed shadow roots and child
// documents — so this reaches text JS can't. `parent_noise` carries whether an
// ancestor was <script>/<style>/etc so their text is skipped.
fn collect_dom_text(node: &Value, parent_noise: bool, out: &mut String) {
let node_type = node.get("nodeType").and_then(|v| v.as_i64()).unwrap_or(0);
let node_name = node.get("nodeName").and_then(|v| v.as_str()).unwrap_or("");
if node_type == 3 {
if !parent_noise {
if let Some(t) = node.get("nodeValue").and_then(|v| v.as_str()) {
let t = t.trim();
if !t.is_empty() {
if !out.is_empty() {
out.push(' ');
}
out.push_str(t);
}
}
}
return;
}
let noise = parent_noise || is_noise_tag(node_name);
if let Some(children) = node.get("children").and_then(|v| v.as_array()) {
for child in children {
collect_dom_text(child, noise, out);
}
}
if let Some(shadow) = node.get("shadowRoots").and_then(|v| v.as_array()) {
for sr in shadow {
collect_dom_text(sr, noise, out);
}
}
if let Some(doc) = node.get("contentDocument") {
collect_dom_text(doc, noise, out);
}
}
/// Extract text from the page via the CDP DOM tree with `pierce:true`, which
/// reaches into CLOSED shadow roots and child documents that `innerText`/`eval`
/// cannot. Lets an agent read content rendered into a closed shadow DOM (e.g. an
/// extension's injected debug panel) without any extra Chrome permission — it
/// rides the per-tab debugger session that's already attached (#30).
pub async fn get_pierced_text(client: &CdpClient, session_id: &str) -> Result<String, String> {
let doc = client
.send_command(
"DOM.getDocument",
Some(serde_json::json!({ "depth": -1, "pierce": true })),
Some(session_id),
)
.await?;
let mut out = String::new();
if let Some(root) = doc.get("root") {
collect_dom_text(root, false, &mut out);
}
Ok(out)
}
pub async fn get_element_attribute(
client: &CdpClient,
session_id: &str,
@@ -1421,6 +1719,31 @@ mod tests {
assert_eq!(parse_ref("@e123"), Some("e123".to_string()));
}
#[test]
fn test_collect_dom_text_pierces_closed_shadow_and_skips_noise() {
// A CDP DOM.Node tree: a host element whose CLOSED shadow root holds the
// text, plus a <script> whose text must be skipped.
let tree = serde_json::json!({
"nodeType": 1, "nodeName": "BODY",
"children": [
{ "nodeType": 1, "nodeName": "SCRIPT",
"children": [ { "nodeType": 3, "nodeName": "#text", "nodeValue": "var secret=1;" } ] },
{ "nodeType": 1, "nodeName": "DIV",
"shadowRoots": [
{ "nodeType": 11, "nodeName": "#document-fragment",
"children": [
{ "nodeType": 1, "nodeName": "SPAN",
"children": [ { "nodeType": 3, "nodeName": "#text", "nodeValue": "DECRYPTED 42" } ] }
] }
] }
]
});
let mut out = String::new();
collect_dom_text(&tree, false, &mut out);
assert_eq!(out, "DECRYPTED 42");
assert!(!out.contains("secret"), "script text must be skipped");
}
#[test]
fn test_parse_ref_equals_prefix() {
assert_eq!(parse_ref("ref=e1"), Some("e1".to_string()));
@@ -1452,10 +1775,30 @@ mod tests {
#[test]
fn test_build_selector_js_css() {
let js = build_selector_js("#submit-btn");
assert!(js.contains("document.querySelector(\"#submit-btn\")"));
// CSS is now tried via a `sel` variable, with a visible-text fallback
// appended (issue #24-B). It must still use querySelector (not xpath).
assert!(js.contains("const sel = \"#submit-btn\""));
assert!(js.contains("document.querySelector(sel)"));
assert!(!js.contains("document.evaluate"));
}
#[test]
fn test_build_find_element_js_text_fallback() {
// A bare label gets a text-matching fallback so `click "購入手続きへ"`
// resolves by visible text, not just CSS (issue #24-B).
let js = build_find_element_js("購入手続きへ");
assert!(js.contains("購入手続きへ"));
assert!(js.contains("interactive")); // the text-match branch
assert!(js.contains("textOf"));
// `text=` forces the text path (skips CSS).
let forced = build_find_element_js("text=Buy now");
assert!(forced.contains("true ? null")); // force_text => css skipped
// xpath is unchanged.
let xp = build_find_element_js("xpath=//button");
assert!(xp.contains("document.evaluate"));
assert!(!xp.contains("interactive"));
}
#[test]
fn test_build_selector_js_xpath() {
let js = build_selector_js("xpath=//button[@id='ok']");
+407 -33
View File
@@ -7,6 +7,17 @@ use super::cdp::types::*;
use super::element::{parse_ref, resolve_element_center, resolve_element_object_id, RefMap};
use super::humanize;
/// Whether a pointer interaction should be DOM-dispatched (invoke the event on
/// the element in its own session) rather than dispatched at a viewport
/// coordinate via `Input.dispatchMouseEvent`. True when the target is inside an
/// iframe (an OOPIF element's box can't be mapped to a top-viewport point) or we
/// drive over the extension relay (a coordinate Input event isn't confined to the
/// target tab on a busy real Chrome — it drifts onto the foreground tab; issues
/// #31/#36). DOM-dispatch always hits the right element in the right tab.
fn prefer_dom_dispatch(ref_map: &RefMap, selector_or_ref: &str) -> bool {
ref_map.ref_is_in_iframe(selector_or_ref) || crate::connect::relay_url().is_some()
}
pub async fn click(
client: &CdpClient,
session_id: &str,
@@ -45,6 +56,45 @@ pub async fn click(
.await;
}
// An element INSIDE an iframe needs a TRUSTED activation: a DOM `.click()` is
// `isTrusted:false`, which security-sensitive embedded forms reject — Google
// Payments' enabled `保存` button silently no-ops on a synthetic click (issue
// #39). A coordinate `Input.dispatchMouseEvent` can't help either: `getBoxModel`
// for a sub-frame node returns frame-local coordinates that don't compose the
// iframe's offset, so the click lands in the wrong place. The frame-agnostic
// trusted path is keyboard activation — focus the element in its own frame, then
// dispatch a real Enter on the page session; Chrome routes the key to the
// focused element regardless of frame (same as `type --focused`), and Enter on a
// focused button/link fires a trusted `click`. `coord` mode opts out.
let in_iframe = ref_map.ref_is_in_iframe(selector_or_ref);
if mode != "coord" && button == "left" && click_count == 1 && in_iframe {
return dom_activate(
client,
session_id,
ref_map,
selector_or_ref,
iframe_sessions,
)
.await;
}
// On the relay (the user's real Chrome) a TOP-document coordinate click used to
// drift onto the foreground tab; that root cause is fixed (#5: the agent drives
// its own pinned tab), but DOM-dispatch stays the conservative default here.
if mode != "coord"
&& button == "left"
&& click_count == 1
&& crate::connect::relay_url().is_some()
{
return dom_click(
client,
session_id,
ref_map,
selector_or_ref,
iframe_sessions,
)
.await;
}
let resolved = resolve_element_center(
client,
session_id,
@@ -235,6 +285,112 @@ async fn dom_click(
Ok(())
}
/// Trusted activation of an element inside an iframe (issue #39). Focuses the
/// element in its own frame session, then dispatches a real Enter/Space on the
/// page session — Chrome routes the key to the focused element across frames, and
/// Enter/Space on a focused button/link/checkbox fires a `click` with
/// `isTrusted: true`, which security-sensitive embedded forms (Google Payments
/// `保存`) require. Non-activatable roles (a `div[onclick]`) can't be keyboard-
/// activated, so they fall back to a DOM `.click()`.
async fn dom_activate(
client: &CdpClient,
session_id: &str,
ref_map: &RefMap,
selector_or_ref: &str,
iframe_sessions: &HashMap<String, String>,
) -> Result<(), String> {
let role = parse_ref(selector_or_ref)
.and_then(|r| ref_map.get(&r).map(|e| e.role.clone()))
.unwrap_or_default();
// Space toggles checkbox-like controls; Enter activates buttons/links/menus.
let key = match role.as_str() {
"checkbox" | "radio" | "switch" | "option" | "menuitemcheckbox" | "menuitemradio" => {
Some("space")
}
"button" | "link" | "menuitem" | "tab" | "treeitem" => Some("enter"),
_ => None,
};
let Some(key) = key else {
// Not keyboard-activatable — best effort via DOM .click() (untrusted).
return dom_click(
client,
session_id,
ref_map,
selector_or_ref,
iframe_sessions,
)
.await;
};
let (object_id, effective_session_id) = resolve_element_object_id(
client,
session_id,
ref_map,
selector_or_ref,
iframe_sessions,
)
.await?;
// Focus the element in its OWN frame session so the keystroke lands on it.
client
.send_command_typed::<_, Value>(
"Runtime.callFunctionOn",
&CallFunctionOnParams {
function_declaration: "function() { this.focus(); }".to_string(),
object_id: Some(object_id),
arguments: None,
return_by_value: Some(true),
await_promise: Some(false),
},
Some(&effective_session_id),
)
.await?;
// Trusted key on the page session — routed to the focused (in-frame) element.
press_key(client, session_id, key).await?;
wait_for_paint_settled(client, &effective_session_id).await;
Ok(())
}
/// DOM-dispatch a double-click on the element in its own session (no coordinates)
/// — the relay/iframe-safe counterpart to a coordinate dblclick. Fires the full
/// click,click,dblclick sequence so handlers bound to any of them respond.
async fn dom_dblclick(
client: &CdpClient,
session_id: &str,
ref_map: &RefMap,
selector_or_ref: &str,
iframe_sessions: &HashMap<String, String>,
) -> Result<(), String> {
let (object_id, effective_session_id) = resolve_element_object_id(
client,
session_id,
ref_map,
selector_or_ref,
iframe_sessions,
)
.await?;
client
.send_command_typed::<_, Value>(
"Runtime.callFunctionOn",
&CallFunctionOnParams {
function_declaration: r#"function() {
const opts = { bubbles: true, cancelable: true, view: window };
this.dispatchEvent(new MouseEvent('click', opts));
this.dispatchEvent(new MouseEvent('click', { ...opts, detail: 2 }));
this.dispatchEvent(new MouseEvent('dblclick', opts));
}"#
.to_string(),
object_id: Some(object_id),
arguments: None,
return_by_value: Some(true),
await_promise: Some(false),
},
Some(&effective_session_id),
)
.await?;
wait_for_paint_settled(client, &effective_session_id).await;
Ok(())
}
pub async fn dblclick(
client: &CdpClient,
session_id: &str,
@@ -242,6 +398,20 @@ pub async fn dblclick(
selector_or_ref: &str,
iframe_sessions: &HashMap<String, String>,
) -> Result<(), String> {
// Same relay/iframe drift hazard as a single click — DOM-dispatch the
// double-click there instead of a coordinate one (issues #31/#36).
if std::env::var("AGENT_BROWSER_CLICK_MODE").as_deref() != Ok("coord")
&& prefer_dom_dispatch(ref_map, selector_or_ref)
{
return dom_dblclick(
client,
session_id,
ref_map,
selector_or_ref,
iframe_sessions,
)
.await;
}
click(
client,
session_id,
@@ -254,6 +424,50 @@ pub async fn dblclick(
.await
}
/// DOM-dispatch a hover (pointer/mouse enter+move) on the element in its own
/// session — reaches OOPIF elements and never drifts to the foreground tab over
/// the relay, unlike a coordinate `mouseMoved` (issues #31/#36).
async fn dom_hover(
client: &CdpClient,
session_id: &str,
ref_map: &RefMap,
selector_or_ref: &str,
iframe_sessions: &HashMap<String, String>,
) -> Result<(), String> {
let (object_id, effective_session_id) = resolve_element_object_id(
client,
session_id,
ref_map,
selector_or_ref,
iframe_sessions,
)
.await?;
client
.send_command_typed::<_, Value>(
"Runtime.callFunctionOn",
&CallFunctionOnParams {
function_declaration: r#"function() {
const r = this.getBoundingClientRect();
const cx = r.left + r.width / 2, cy = r.top + r.height / 2;
const base = { bubbles: true, cancelable: true, view: window, clientX: cx, clientY: cy };
this.dispatchEvent(new PointerEvent('pointerover', base));
this.dispatchEvent(new PointerEvent('pointerenter', { ...base, bubbles: false }));
this.dispatchEvent(new MouseEvent('mouseover', base));
this.dispatchEvent(new MouseEvent('mouseenter', { ...base, bubbles: false }));
this.dispatchEvent(new MouseEvent('mousemove', base));
}"#
.to_string(),
object_id: Some(object_id),
arguments: None,
return_by_value: Some(true),
await_promise: Some(false),
},
Some(&effective_session_id),
)
.await?;
Ok(())
}
pub async fn hover(
client: &CdpClient,
session_id: &str,
@@ -261,6 +475,18 @@ pub async fn hover(
selector_or_ref: &str,
iframe_sessions: &HashMap<String, String>,
) -> Result<(), String> {
// Coordinate `mouseMoved` drifts to the foreground tab over the relay and
// can't reach an OOPIF — DOM-dispatch the hover there (issues #31/#36).
if prefer_dom_dispatch(ref_map, selector_or_ref) {
return dom_hover(
client,
session_id,
ref_map,
selector_or_ref,
iframe_sessions,
)
.await;
}
let (x, y, _w, _h, effective_session_id) = resolve_element_center(
client,
session_id,
@@ -289,6 +515,63 @@ pub async fn hover(
Ok(())
}
/// DOM-dispatch an HTML5 drag-and-drop from `source` to `target` in their shared
/// session — the relay/iframe-safe counterpart to the coordinate drag, which
/// drifts to the foreground tab over the relay and can't reach an OOPIF (issues
/// #31/#36). Covers HTML5 DnD (sortable lists, file/card boards); pointer-driven
/// drag (canvas, sliders) still needs the coordinate path. Errors if source and
/// target live in different frames — a synthetic cross-frame DnD isn't reliable.
pub async fn dom_drag(
client: &CdpClient,
session_id: &str,
ref_map: &RefMap,
source: &str,
target: &str,
iframe_sessions: &HashMap<String, String>,
) -> Result<(), String> {
let (src_obj, src_session) =
resolve_element_object_id(client, session_id, ref_map, source, iframe_sessions).await?;
let (tgt_obj, tgt_session) =
resolve_element_object_id(client, session_id, ref_map, target, iframe_sessions).await?;
if src_session != tgt_session {
return Err(
"drag source and target are in different frames; cross-frame drag-and-drop over the \
relay isn't supported drag within a single frame, or use a launched browser with \
AGENT_BROWSER_CLICK_MODE=coord"
.to_string(),
);
}
client
.send_command_typed::<_, Value>(
"Runtime.callFunctionOn",
&CallFunctionOnParams {
function_declaration: r#"function(target) {
const dt = new DataTransfer();
const ev = (type, el) => el.dispatchEvent(
new DragEvent(type, { bubbles: true, cancelable: true, dataTransfer: dt }));
ev('dragstart', this);
ev('drag', this);
ev('dragenter', target);
ev('dragover', target);
ev('drop', target);
ev('dragend', this);
}"#
.to_string(),
object_id: Some(src_obj),
arguments: Some(vec![CallArgument {
value: None,
object_id: Some(tgt_obj),
}]),
return_by_value: Some(true),
await_promise: Some(false),
},
Some(&src_session),
)
.await?;
wait_for_paint_settled(client, &src_session).await;
Ok(())
}
pub async fn fill(
client: &CdpClient,
session_id: &str,
@@ -306,32 +589,50 @@ pub async fn fill(
)
.await?;
// Focus the element
client
.send_command_typed::<_, Value>(
"Runtime.callFunctionOn",
&CallFunctionOnParams {
function_declaration: "function() { this.focus(); }".to_string(),
object_id: Some(object_id.clone()),
arguments: None,
return_by_value: Some(true),
await_promise: Some(false),
},
Some(&effective_session_id),
)
.await?;
// Emulate a real edit so framework-controlled inputs (React/Vue) and
// site-side listeners actually see the change (issue #25): the old path set
// `this.value` directly and used Input.insertText, which left React's
// internal value-tracker out of sync and never fired change/blur — so
// dependent logic (e.g. Mercari's postal-code → 都道府県 autocomplete) never
// ran even though the value was visible. Set the value through the element's
// PROTOTYPE setter (which React's _valueTracker hooks), then dispatch
// input → change → blur/focusout. `type <sel> <text>` remains for sites that
// need per-keystroke events.
let fill_js = format!(
r#"function() {{
const el = this;
const v = {val};
try {{ el.focus(); }} catch (e) {{}}
const tag = el.tagName;
const fire = (type, ctor) => el.dispatchEvent(new (ctor || Event)(type, {{ bubbles: true }}));
if (tag === 'SELECT') {{
el.value = v; fire('input'); fire('change'); return true;
}}
if (el.isContentEditable) {{
el.textContent = v; fire('input', window.InputEvent || Event); fire('change');
try {{ el.blur(); }} catch (e) {{}} fire('focusout'); return true;
}}
const proto = tag === 'TEXTAREA' ? window.HTMLTextAreaElement.prototype
: window.HTMLInputElement.prototype;
const desc = Object.getOwnPropertyDescriptor(proto, 'value');
const set = desc && desc.set ? (x) => desc.set.call(el, x) : (x) => {{ el.value = x; }};
set(''); // reset the framework tracker
fire('input', window.InputEvent || Event);
set(v); // native setter → React/Vue registers
fire('input', window.InputEvent || Event);
fire('change');
try {{ el.blur(); }} catch (e) {{}}
fire('focusout'); // blur-triggered lookups/validation
return true;
}}"#,
val = serde_json::to_string(value).unwrap_or_default()
);
// Select all + delete to clear
client
.send_command_typed::<_, Value>(
"Runtime.callFunctionOn",
&CallFunctionOnParams {
function_declaration: r#"function() {
this.select && this.select();
this.value = '';
this.dispatchEvent(new Event('input', { bubbles: true }));
}"#
.to_string(),
function_declaration: fill_js,
object_id: Some(object_id),
arguments: None,
return_by_value: Some(true),
@@ -341,17 +642,6 @@ pub async fn fill(
)
.await?;
// Insert text (keyboard input dispatched at page level, use parent session_id)
client
.send_command_typed::<_, Value>(
"Input.insertText",
&InsertTextParams {
text: value.to_string(),
},
Some(session_id),
)
.await?;
Ok(())
}
@@ -365,6 +655,7 @@ pub async fn type_text(
clear: bool,
delay_ms: Option<u64>,
iframe_sessions: &HashMap<String, String>,
key_events: bool,
) -> Result<(), String> {
let (object_id, effective_session_id) = resolve_element_object_id(
client,
@@ -411,7 +702,7 @@ pub async fn type_text(
.await?;
}
type_text_into_active_context(client, session_id, text, delay_ms).await
type_text_into_active_context(client, session_id, text, delay_ms, key_events).await
}
pub async fn type_text_into_active_context(
@@ -419,6 +710,7 @@ pub async fn type_text_into_active_context(
session_id: &str,
text: &str,
delay_ms: Option<u64>,
key_events: bool,
) -> Result<(), String> {
// Per-character timing: an explicit `delay_ms` wins (caller asked for a
// fixed cadence); otherwise fall back to humanize — variable, human-like
@@ -468,6 +760,46 @@ pub async fn type_text_into_active_context(
Some(session_id),
)
.await?;
} else if key_events {
// Real keystrokes (keyDown+keyUp carrying `text`) for autocomplete /
// combobox widgets that only react to key events and ignore the
// `input` that `Input.insertText` fires — e.g. Google's address
// postal-code → city/prefecture lookup (issue #36 / #4). The keyDown's
// `text` still inserts the character, so the field also fills.
let (key, code, key_code) = char_to_key_info(ch);
let s = ch.to_string();
client
.send_command_typed::<_, Value>(
"Input.dispatchKeyEvent",
&DispatchKeyEventParams {
event_type: "keyDown".to_string(),
key: Some(key.clone()),
code: Some(code.clone()),
text: Some(s.clone()),
unmodified_text: Some(s),
windows_virtual_key_code: Some(key_code),
native_virtual_key_code: Some(key_code),
modifiers: None,
},
Some(session_id),
)
.await?;
client
.send_command_typed::<_, Value>(
"Input.dispatchKeyEvent",
&DispatchKeyEventParams {
event_type: "keyUp".to_string(),
key: Some(key),
code: Some(code),
text: None,
unmodified_text: None,
windows_virtual_key_code: Some(key_code),
native_virtual_key_code: Some(key_code),
modifiers: None,
},
Some(session_id),
)
.await?;
} else {
// VS Code/Electron webviews reject repeated dispatchKeyEvent calls
// carrying printable `text`. Insert printable characters directly
@@ -557,6 +889,48 @@ pub async fn press_key_with_modifiers(
Ok(())
}
/// Dispatch a SINGLE key event (`keyDown` or `keyUp`) carrying the full key
/// descriptor — `key`, `code`, `windowsVirtualKeyCode`/`nativeVirtualKeyCode`,
/// and (on key-down) printable `text`. Powers the `keydown`/`keyup` commands.
///
/// The previous implementation sent only `{key}`, so games and shortcut handlers
/// that read `event.code` (e.g. `"KeyD"`, `"ArrowRight"`) or `event.keyCode` saw
/// nothing — a held key set no movement flag and did nothing (dogfood: holding a
/// direction in a canvas platformer barely nudged the player). Sending the same
/// descriptor `press` uses makes hold-to-move work regardless of which field the
/// page keys off.
pub async fn dispatch_single_key(
client: &CdpClient,
session_id: &str,
key: &str,
event_type: &str,
) -> Result<(), String> {
let (key_name, code, key_code) = named_key_info(key);
// Printable text is only meaningful on key-down; key-up never inserts.
let text = if event_type == "keyDown" {
key_text(&key_name)
} else {
None
};
client
.send_command_typed::<_, Value>(
"Input.dispatchKeyEvent",
&DispatchKeyEventParams {
event_type: event_type.to_string(),
key: Some(key_name),
code: Some(code),
text: text.clone(),
unmodified_text: text,
windows_virtual_key_code: Some(key_code),
native_virtual_key_code: Some(key_code),
modifiers: None,
},
Some(session_id),
)
.await?;
Ok(())
}
pub async fn scroll(
client: &CdpClient,
session_id: &str,
+39
View File
@@ -330,6 +330,45 @@ mod tests {
}
}
#[test]
fn reattach_with_same_session_restores_target() {
// Issue #17 recovery contract. A tab's chrome.debugger session is torn
// down (cross-process nav, SW restart, …) then re-attached. The fix has
// the extension reuse the SAME `cb-tab-<tabId>` id across that churn, so
// after detach+reattach the relay must expose the NEW target under the
// SAME session — which is exactly the session the daemon is still bound
// to, so its eval/snapshot auto-follow the new page instead of going stale.
let mut s = RelayState::new();
s.handle_ext_message(&attached_event("T_old", "cb-tab-42"), "tok");
s.handle_ext_message(
&json!({
"method": "forwardCDPEvent",
"params": { "method": "Target.detachedFromTarget", "params": { "sessionId": "cb-tab-42" } }
}),
"tok",
);
s.handle_ext_message(&attached_event("T_new", "cb-tab-42"), "tok");
let route = s.route_client_command(1, &json!({ "id": 1, "method": "Target.getTargets" }));
match route {
ClientRoute::Local(v) => {
let infos = v["result"]["targetInfos"].as_array().unwrap();
assert_eq!(infos.len(), 1, "only the new target should remain");
assert_eq!(infos[0]["targetId"], "T_new");
}
_ => panic!("getTargets must be local"),
}
// The daemon's existing session id still resolves — to the new target.
let route = s.route_client_command(
1,
&json!({ "id": 2, "method": "Target.attachToTarget", "params": { "targetId": "T_new" } }),
);
assert_eq!(
route,
ClientRoute::Local(json!({ "id": 2, "result": { "sessionId": "cb-tab-42" } }))
);
}
#[test]
fn browser_get_version_is_answered_locally() {
// Liveness probe must NOT be forwarded (the extension can't do
+14 -1
View File
@@ -60,6 +60,9 @@ pub struct ScreenshotOptions {
pub quality: Option<i32>,
pub annotate: bool,
pub output_dir: Option<String>,
/// Explicit pixel region (x, y, width, height) — `--clip` (issue #34). Takes
/// precedence over selector/full_page.
pub clip: Option<(f64, f64, f64, f64)>,
}
impl Default for ScreenshotOptions {
@@ -72,6 +75,7 @@ impl Default for ScreenshotOptions {
quality: None,
annotate: false,
output_dir: None,
clip: None,
}
}
}
@@ -187,7 +191,16 @@ async fn capture_screenshot_base64(
capture_beyond_viewport: if options.full_page { Some(true) } else { None },
};
if options.full_page {
if let Some((x, y, width, height)) = options.clip {
// Explicit pixel region wins over selector/full_page (issue #34).
params.clip = Some(Viewport {
x,
y,
width,
height,
scale: 1.0,
});
} else if options.full_page {
let metrics: Value = client
.send_command_no_params("Page.getLayoutMetrics", Some(session_id))
.await?;
+36 -5
View File
@@ -330,6 +330,13 @@ impl RoleNameTracker {
}
}
/// Max iframe nesting depth `take_snapshot` expands. Embedded payment/checkout
/// widgets nest a few frames deep (e.g. AdSense → payments.google.com → an inner
/// form frame); expanding past the first level is what gives those inner refs a
/// `frame_id` so clicks resolve into the right frame (issue #36). Capped to keep
/// a pathological frame tree from blowing up the snapshot.
const MAX_IFRAME_DEPTH: usize = 3;
pub async fn take_snapshot(
client: &CdpClient,
session_id: &str,
@@ -337,6 +344,28 @@ pub async fn take_snapshot(
ref_map: &mut RefMap,
frame_id: Option<&str>,
iframe_sessions: &HashMap<String, String>,
) -> Result<String, String> {
take_snapshot_at_depth(
client,
session_id,
options,
ref_map,
frame_id,
iframe_sessions,
0,
)
.await
}
#[allow(clippy::too_many_arguments)]
async fn take_snapshot_at_depth(
client: &CdpClient,
session_id: &str,
options: &SnapshotOptions,
ref_map: &mut RefMap,
frame_id: Option<&str>,
iframe_sessions: &HashMap<String, String>,
depth: usize,
) -> Result<String, String> {
client
.send_command_no_params("DOM.enable", Some(session_id))
@@ -606,10 +635,11 @@ pub async fn take_snapshot(
}
// Recurse into child iframes: for each Iframe node with a backend_node_id,
// resolve the child frame ID and take a snapshot of its content.
// We only recurse from the main frame (frame_id == None) to avoid
// unbounded depth; nested iframes within iframes are not expanded.
if frame_id.is_none() {
// resolve the child frame ID and snapshot its content. Recurse to
// MAX_IFRAME_DEPTH (not just the main frame) so refs inside nested
// payment/checkout widgets get a `frame_id` and clicks resolve into the right
// frame (issue #36); the cap bounds a pathological frame tree.
if depth < MAX_IFRAME_DEPTH {
let mut iframe_snapshots: Vec<(String, String)> = Vec::new(); // (ref_id, child_snapshot)
for node in tree_nodes.iter() {
if node.role != "Iframe" || !node.has_ref {
@@ -622,13 +652,14 @@ pub async fn take_snapshot(
if let Ok(child_fid) = resolve_iframe_frame_id(client, session_id, bid).await {
// Snapshot the child frame; errors are silently ignored
// (e.g. cross-origin iframes)
if let Ok(child_text) = Box::pin(take_snapshot(
if let Ok(child_text) = Box::pin(take_snapshot_at_depth(
client,
session_id,
options,
ref_map,
Some(&child_fid),
iframe_sessions,
depth + 1,
))
.await
{
@@ -0,0 +1,28 @@
<!doctype html>
<html>
<head>
<meta charset="utf-8" />
<title>iframe button probe</title>
</head>
<body>
<h1>iframe button probe</h1>
<iframe
id="frame"
width="320"
height="140"
srcdoc="
<!doctype html>
<html>
<body style='margin:24px'>
<button id='b' style='padding:24px;font-size:22px'>save</button>
<script>
document.getElementById('b').addEventListener('click', function (e) {
this.textContent = 'clicked:' + e.isTrusted;
});
</script>
</body>
</html>
"
></iframe>
</body>
</html>
+265 -19
View File
@@ -186,6 +186,123 @@ pub fn print_response_with_opts(resp: &Response, action: Option<&str>, opts: &Ou
}
if let Some(data) = &resp.data {
// Auto-trigger: when you land on / read a page whose domain has site
// adapters, surface them so the agent pulls structured data via
// `chrome-use site <name>/<cmd>` instead of scraping the DOM. (In --json
// mode this same info rides along in the `siteAdapters` field above.)
if let Some(hint) = data.get("siteAdapters") {
let domain = hint.get("domain").and_then(|v| v.as_str()).unwrap_or("");
let cmds: Vec<&str> = hint
.get("commands")
.and_then(|v| v.as_array())
.map(|a| a.iter().filter_map(|v| v.as_str()).collect())
.unwrap_or_default();
if !cmds.is_empty() {
eprintln!("💡 site adapters for {domain} — prefer these for structured data:");
eprintln!(" {}", color::dim(&cmds.join(", ")));
eprintln!(
" {}",
color::dim(&format!("e.g. chrome-use site {} --json", cmds[0]))
);
}
}
// A click that opened a new tab: surface it so the agent doesn't read the
// unchanged old page as a failed click (issue #24-A).
if let Some(opened) = data.get("openedTab") {
let tid = opened.get("tabId").and_then(|v| v.as_str()).unwrap_or("?");
let url = opened.get("url").and_then(|v| v.as_str()).unwrap_or("");
let followed = data
.get("followed")
.and_then(|v| v.as_bool())
.unwrap_or(false);
let verb = if followed {
"switched to new tab"
} else {
"opened new tab"
};
eprintln!(
"{} {} [{}] {}",
color::cyan(""),
verb,
tid,
color::dim(url)
);
}
// `current`: the active tab's stable handle (#26).
if data
.get("current")
.and_then(|v| v.as_bool())
.unwrap_or(false)
{
let tid = data.get("tabId").and_then(|v| v.as_str()).unwrap_or("?");
let title = data.get("title").and_then(|v| v.as_str()).unwrap_or("");
let url = data.get("url").and_then(|v| v.as_str()).unwrap_or("");
let target = data.get("targetId").and_then(|v| v.as_str()).unwrap_or("");
println!("{} [{}] {} - {}", color::cyan(""), tid, title, url);
println!(" {}", color::dim(&format!("target: {}", target)));
return;
}
// Cloudflare challenge/clearance preflight (`cf-status`). Checked early
// because its response carries `url`/`title`, which later generic
// renderers would otherwise swallow.
if action == Some("cf_status") {
let challenged = data
.get("challenged")
.and_then(|v| v.as_bool())
.unwrap_or(false);
let rec = data
.get("recommendation")
.and_then(|v| v.as_str())
.unwrap_or("?");
let cl = data.get("clearance");
let present = cl
.and_then(|c| c.get("present"))
.and_then(|v| v.as_bool())
.unwrap_or(false);
let expired = cl
.and_then(|c| c.get("expired"))
.and_then(|v| v.as_bool())
.unwrap_or(false);
let expires_in = cl.and_then(|c| c.get("expiresIn")).and_then(|v| v.as_i64());
let device = data
.get("deviceVerified")
.and_then(|v| v.as_bool())
.unwrap_or(false);
let (icon, headline) = match rec {
"proceed" => (color::success_indicator().to_string(), "cleared — no challenge, proceed"),
"solve" => (color::warning_indicator().to_string(), "Cloudflare challenge active, no valid clearance — solve it"),
"reissue" => (color::warning_indicator().to_string(), "challenge active but a clearance cookie exists — stale (IP/UA changed?), re-solve"),
_ => (color::cyan("").to_string(), "unknown"),
};
println!("{} {}", icon, headline);
println!(
" challenged: {}",
if challenged { "yes" } else { "no" }
);
let cl_desc = if !present {
"absent".to_string()
} else if expired {
"present but EXPIRED".to_string()
} else if let Some(s) = expires_in {
format!("valid, expires in {}m {}s", s / 60, s % 60)
} else {
"present (session)".to_string()
};
println!(" cf_clearance: {}", cl_desc);
println!(
" device trusted: {}",
if device {
"yes (CF_VERIFIED_DEVICE)"
} else {
"no"
}
);
return;
}
// Dialog status response
if action == Some("dialog") {
if let Some(has_dialog) = data.get("hasDialog").and_then(|v| v.as_bool()) {
@@ -297,6 +414,43 @@ pub fn print_response_with_opts(resp: &Response, action: Option<&str>, opts: &Ou
// Snapshot
if let Some(snapshot) = data.get("snapshot").and_then(|v| v.as_str()) {
print_with_boundaries(snapshot, origin, opts);
// Canvas-app hint: the tree was near-empty but the page paints to a
// <canvas>, so refs are a dead end — point at the screenshot path.
if let Some(note) = data.get("note").and_then(|v| v.as_str()) {
eprintln!("{}", color::dim(note));
}
return;
}
// Frame list (`chrome-use frames`)
if action == Some("frames") {
if let Some(list) = data.get("frames").and_then(|v| v.as_array()) {
let count = list.len();
println!(
"{}",
color::bold(&format!(
"{} frame{}",
count,
if count == 1 { "" } else { "s" }
))
);
for f in list {
let idx = f.get("index").and_then(|v| v.as_i64()).unwrap_or(0);
let kind = f.get("kind").and_then(|v| v.as_str()).unwrap_or("?");
let url = f.get("url").and_then(|v| v.as_str()).unwrap_or("");
let len = f.get("textLen").and_then(|v| v.as_i64()).unwrap_or(0);
println!(
" [{}] {:<6} {} chars {}",
idx,
kind,
len,
color::dim(if url.is_empty() { "(about:blank)" } else { url })
);
}
eprintln!(
"{}",
color::dim("read everything with: chrome-use get text --all-frames")
);
}
return;
}
// Title
@@ -477,6 +631,9 @@ pub fn print_response_with_opts(resp: &Response, action: Option<&str>, opts: &Ou
}
// Tabs
if let Some(tabs) = data.get("tabs").and_then(|v| v.as_array()) {
// `tab list --full` prints untruncated URLs so a long SSO/redirect
// URL can actually be re-opened after a stale session (issue #19).
let full = data.get("full").and_then(|v| v.as_bool()).unwrap_or(false);
for tab in tabs {
let tab_id = tab.get("tabId").and_then(|v| v.as_str()).unwrap_or("?");
let tab_label = tab.get("label").and_then(|v| v.as_str());
@@ -491,8 +648,13 @@ pub fn print_response_with_opts(resp: &Response, action: Option<&str>, opts: &Ou
let title = title.as_str();
let url = tab.get("url").and_then(|v| v.as_str()).unwrap_or("");
// Truncate very long URLs (e.g. multi-KB JWT/OTP login links) so
// the list stays readable instead of flooding the terminal.
let url = truncate_middle(url, 120);
// the list stays readable instead of flooding the terminal
// unless `--full` was asked for (to re-open the exact URL).
let url = if full {
url.to_string()
} else {
truncate_middle(url, 120)
};
let active = tab.get("active").and_then(|v| v.as_bool()).unwrap_or(false);
let marker = if active {
color::cyan("")
@@ -504,25 +666,35 @@ pub fn print_response_with_opts(resp: &Response, action: Option<&str>, opts: &Ou
} else {
println!("{} [{}] {} - {}", marker, tab_id, title, url);
}
// `--full` also surfaces the stable cross-session CDP targetId so
// a stranded tab can be adopted from another session via
// `tab <targetId>` (issue #21).
if full {
if let Some(target_id) = tab.get("targetId").and_then(|v| v.as_str()) {
println!(" {}", color::dim(&format!("target: {}", target_id)));
}
}
}
return;
}
// Tab switch
if action == Some("tab_switch") {
if let Some(tab_id) = data.get("tabId").and_then(|v| v.as_str()) {
if let Some(url) = data.get("url").and_then(|v| v.as_str()) {
println!(
"{} Switched to tab [{}] ({})",
color::success_indicator(),
tab_id,
url
);
let warning = data.get("warning").and_then(|v| v.as_str());
// A non-responding session isn't a real success — show a warning
// indicator instead of the green ✓ (issue #29.3).
let indicator = if warning.is_some() {
color::warning_indicator()
} else {
println!(
"{} Switched to tab [{}]",
color::success_indicator(),
tab_id
);
color::success_indicator()
};
if let Some(url) = data.get("url").and_then(|v| v.as_str()) {
println!("{} Switched to tab [{}] ({})", indicator, tab_id, url);
} else {
println!("{} Switched to tab [{}]", indicator, tab_id);
}
if let Some(w) = warning {
eprintln!("{}", color::dim(w));
}
return;
}
@@ -1341,6 +1513,12 @@ Usage: chrome-use type <selector> <text>
Types text into the specified element character by character.
Unlike fill, this does not clear existing content first.
Options:
--key-events Send real per-character keyDown/keyUp instead of
(alias --keys) Input.insertText. Use for autocomplete / combobox fields
that only react to key events e.g. a postal-code box
that auto-fills city/prefecture, or Google Places.
Global Options:
--json Output as JSON
--session <name> Use specific session
@@ -1348,6 +1526,7 @@ Global Options:
Examples:
chrome-use type "#search" "hello"
chrome-use type @e2 "additional text"
chrome-use type @e5 "201-0001" --key-events # trigger the address autocomplete
See Also:
For typing into contenteditable editors (Lexical, ProseMirror, etc.)
@@ -1612,12 +1791,23 @@ Usage: chrome-use scroll [direction] [amount] [options]
Scrolls the page or a specific element in the specified direction.
Without --selector, scroll dispatches a real (isTrusted) mouse wheel at a
viewport coordinate, so it scrolls whatever container is under the pointer
including cross-origin iframes (Google Payments, Stripe, embedded checkout/KYC)
that plain page scroll can't reach.
Arguments:
direction up, down, left, right (default: down)
amount Pixels to scroll (default: 300)
Options:
-s, --selector <sel> CSS selector for a scrollable container
-s, --selector <sel> CSS selector for a scrollable container (same-origin)
--at <x,y> Dispatch the wheel at this viewport pixel (read it from a
screenshot) precise way into a cross-origin iframe
--frame <n> Scroll the n-th frame from `chrome-use frames` (wheel at
that frame's center)
Without --selector/--at/--frame the wheel lands at the viewport center.
Global Options:
--json Output as JSON
@@ -1629,6 +1819,8 @@ Examples:
chrome-use scroll up 200
chrome-use scroll left 100
chrome-use scroll down 500 --selector "div.scroll-container"
chrome-use scroll down 700 --at 640,400 # wheel at a pixel over an iframe
chrome-use scroll down 700 --frame 2 # scroll frame 2 from `frames`
"##
}
"scrollintoview" | "scrollinto" => {
@@ -1709,6 +1901,8 @@ Pass --hide-scrollbars false when launching to keep native scrollbars visible.
Options:
--full, -f Capture full page (not just viewport)
[selector] Capture just an element (CSS or @ref), e.g. `screenshot ".header" h.png`
--clip <x,y,w,h> Capture a pixel region, e.g. `screenshot --clip 0,0,200,40 corner.png`
--annotate Overlay numbered labels on interactive elements.
Each label [N] corresponds to ref @eN from snapshot.
Prints a legend mapping labels to element roles/names.
@@ -1729,6 +1923,8 @@ Examples:
chrome-use screenshot
chrome-use screenshot ./screenshot.png
chrome-use screenshot --full ./full-page.png
chrome-use screenshot ".header .indicator" corner.png # just one element
chrome-use screenshot --clip 1600,0,200,40 corner.png # a pixel region
chrome-use screenshot --annotate # Labeled screenshot + legend
chrome-use screenshot --annotate ./page.png # Save annotated screenshot
chrome-use screenshot --annotate --json # JSON output with annotations
@@ -1870,7 +2066,9 @@ Usage: chrome-use get <subcommand> [args]
Retrieves various types of information from elements or the page.
Subcommands:
text <selector> Get text content of element
text [selector] Element text; no selector = WHOLE PAGE, all frames
text --main Main-content text only (skip nav/header/sidebar)
text --pierce Read through CLOSED shadow DOM (injected panels)
html <selector> Get inner HTML of element
value <selector> Get value of input element
attr <selector> <name> Get attribute value
@@ -1886,7 +2084,10 @@ Global Options:
--session <name> Use specific session
Examples:
chrome-use get text @e1
chrome-use get text # whole page across ALL frames (default)
chrome-use get text @e1 # one element
chrome-use get text --main # main content, no nav/sidebar boilerplate
chrome-use frames # list frames + where the text lives
chrome-use get html "#content"
chrome-use get value "#email-input"
chrome-use get attr "#link" href
@@ -2757,6 +2958,20 @@ Notes:
- Streaming is always enabled. Set AGENT_BROWSER_STREAM_PORT to bind to a
specific port instead of the default OS-assigned port.
The WS is BIDIRECTIONAL the high-throughput way to drive a live/real-time page
(games, canvas apps) instead of one screenshot + one CLI call per action:
- Server -> client (JSON text frames):
{"type":"frame","data":"<base64 jpeg>"} live screencast (~60fps)
plus status / tabs messages.
- Client -> server (send JSON text):
{"type":"input_keyboard","eventType":"keyDown|keyUp","key":" ","code":"Space",
"windowsVirtualKeyCode":32}
{"type":"input_mouse","eventType":"mousePressed|mouseReleased|mouseMoved",
"x":640,"y":360,"button":"left","clickCount":1}
{"type":"input_touch","eventType":"touchStart|touchEnd","touchPoints":[...]}
Connect once and run a tight local loop: read frames, send timed input no
per-action process spawn, no round-trip. Works over the extension relay too.
Global Options:
--json Output as JSON
--session <name> Use specific session
@@ -3049,7 +3264,12 @@ Core Commands:
dblclick <sel> Double-click element
type <sel> <text> Type into element
fill <sel> <text> Clear and fill
press <key> Press key (Enter, Tab, Control+a)
press <key> [--hold <ms>] Press key (Enter, Tab, Control+a). --hold keeps it
down <ms> then releases precise (in-daemon), for
games/charge: `press d --hold 800`
keydown <key> Hold a key down (no auto-release) for games/shortcuts
keyup <key> Release a held key. Pair with keydown to hold-to-move:
`keydown d` `keyup d`
keyboard type <text> Type text with real keystrokes (no selector)
keyboard inserttext <text> Insert text without key events
hover <sel> Hover element
@@ -3077,10 +3297,15 @@ Navigation:
Get Info: chrome-use get <what> [selector]
text, html, value, attr <name>, title, url, count, box, styles, cdp-url
text (no selector = whole page, all frames), text --main, frames (list)
Check State: chrome-use is <what> <selector>
visible, enabled, checked
Anti-bot: chrome-use stealth | cf-status
stealth stealth self-check (webdriver/UA/plugins + overrides)
cf-status Cloudflare challenge + cf_clearance preflight (skip re-solving)
Find Elements: chrome-use find <locator> <value> <action> [text]
role, text, label, placeholder, alt, title, testid, first, last, nth
@@ -3108,7 +3333,12 @@ Storage:
storage <local|session> Manage web storage
Tabs:
tab [new|list|close|<n>] Manage tabs
tab [new|list|close|<ref>] Manage tabs (<ref> = t<N>, a label, or a CDP targetId)
tab list --full Full URLs + stable cross-session targetId per tab
tab <targetId> Adopt a specific tab (incl. another session's) by its
stable targetId, no reload preserves in-page state
open <url> --reuse-tab Reuse an existing tab on that URL instead of spawning
a duplicate (matches origin+path; preserves state)
Diff:
diff snapshot Compare current vs last snapshot
@@ -3156,6 +3386,14 @@ Batch:
batch [--bail] ["cmd" ...] Execute multiple commands sequentially (args or stdin)
--bail stops on first error (default: continue all)
Site adapters: turn a website into a structured-data CLI (runs as you, in your tab)
site update Fetch the community adapter pack into ~/.chrome-use/sites
site list List installed adapters (name/cmd)
site info <name>/<cmd> Show an adapter's @meta (args, domain, capabilities)
site <name>/<cmd> [args] Run an adapter: navigate to its site + return JSON
e.g. site github/issues epiral/repo, site reddit/search rust
Positional args fill declared args in order; --key value overrides
Auth Vault:
auth save <name> [opts] Save auth profile (--url, --username, --password/--password-stdin)
auth login <name> Login using saved credentials (waits for form fields)
@@ -3170,6 +3408,10 @@ Confirmation:
Sessions:
session Show current session name
session list List active sessions
sessions List running session daemons (alias of daemon status)
daemon status List running session daemons (+ relay state)
daemon restart Kill all session daemons; keeps the extension relay
up. Clears stale/cross-leaked state after an upgrade.
Chat (AI):
chat <message> Send a natural language instruction (single-shot)
@@ -3363,6 +3605,9 @@ iOS Simulator (requires Xcode and Appium):
chrome-use -p ios device list # List simulators
chrome-use -p ios swipe up # Swipe gesture
chrome-use -p ios tap @e1 # Touch element
Hit a bug or rough edge? A 30-second issue genuinely sharpens this tool:
https://github.com/leeguooooo/chrome-use/issues
"#
);
}
@@ -3445,6 +3690,7 @@ fn print_screenshot_diff(data: &serde_json::Map<String, serde_json::Value>) {
pub fn print_version() {
println!("chrome-use {}", env!("CARGO_PKG_VERSION"));
println!("report bugs / rough edges: https://github.com/leeguooooo/chrome-use/issues");
}
#[cfg(test)]
+385
View File
@@ -0,0 +1,385 @@
//! Site adapters: turn any website into a structured-data CLI by running a small
//! per-command JS adapter inside your real, logged-in browser tab (it reuses the
//! site's cookies / same-origin fetch / its own webpack modules — the site thinks
//! it's you, because it is).
//!
//! The adapter format is the community **bb-sites** convention
//! (<https://github.com/epiral/bb-sites>): one `.js` file per command, a
//! `/* @meta {...} */` JSON header (name, description, domain, args), then an
//! `async function(args){ ... return {...} }`. chrome-use ships none of those
//! adapters — `chrome-use site update` fetches the upstream repo at runtime into
//! `~/.chrome-use/sites` (like a package manager pulling a dependency), so the
//! adapters stay the property of their authors. Running an adapter navigates to
//! its `@meta.domain` and `eval`s the function in the site's own logged-in page.
use std::path::PathBuf;
use serde_json::Value;
const SITES_ZIP_URL: &str = "https://github.com/epiral/bb-sites/archive/refs/heads/main.zip";
/// `~/.chrome-use/sites` — where synced adapters live.
pub fn sites_dir() -> Option<PathBuf> {
dirs_home().map(|h| h.join(".chrome-use").join("sites"))
}
fn dirs_home() -> Option<PathBuf> {
std::env::var_os("HOME").map(PathBuf::from)
}
/// Parsed adapter: its `@meta` JSON and the raw `async function(args){...}` source.
pub struct Adapter {
pub meta: Value,
pub func_src: String,
}
impl Adapter {
pub fn domain(&self) -> Option<&str> {
self.meta.get("domain").and_then(|v| v.as_str())
}
}
/// Load `<sites>/<name>/<cmd>.js`, splitting the `/* @meta {...} */` header from
/// the function body. `spec` is `name/cmd`.
pub fn load_adapter(spec: &str) -> Result<Adapter, String> {
let (name, cmd) = spec
.split_once('/')
.ok_or_else(|| format!("site: expected <name>/<command>, got `{spec}`"))?;
if name.is_empty()
|| cmd.is_empty()
|| name.contains("..")
|| cmd.contains("..")
|| name.contains('/')
|| cmd.contains('/')
{
return Err(format!("site: invalid adapter spec `{spec}`"));
}
let dir = sites_dir().ok_or("site: cannot resolve home dir")?;
let path = dir.join(name).join(format!("{cmd}.js"));
if !path.exists() {
return Err(format!(
"site: adapter `{spec}` not found. Run `chrome-use site update` to sync adapters, \
or `chrome-use site list` to see what's installed."
));
}
let raw = std::fs::read_to_string(&path).map_err(|e| format!("site: read {spec}: {e}"))?;
parse_adapter(&raw, spec)
}
/// Split the `@meta` JSON block and the function source from an adapter file.
pub fn parse_adapter(raw: &str, spec: &str) -> Result<Adapter, String> {
let start = raw
.find("@meta")
.and_then(|i| raw[i..].find('{').map(|j| i + j))
.ok_or_else(|| format!("site: {spec} missing /* @meta {{...}} */ header"))?;
// Find the matching close brace for the @meta object (brace-count, string-aware).
let bytes = raw.as_bytes();
let mut depth = 0i32;
let mut in_str = false;
let mut esc = false;
let mut end = None;
for (k, &b) in bytes.iter().enumerate().skip(start) {
if in_str {
if esc {
esc = false;
} else if b == b'\\' {
esc = true;
} else if b == b'"' {
in_str = false;
}
continue;
}
match b {
b'"' => in_str = true,
b'{' => depth += 1,
b'}' => {
depth -= 1;
if depth == 0 {
end = Some(k + 1);
break;
}
}
_ => {}
}
}
let end = end.ok_or_else(|| format!("site: {spec} @meta header has no closing brace"))?;
let meta: Value = serde_json::from_str(&raw[start..end])
.map_err(|e| format!("site: {spec} @meta is not valid JSON: {e}"))?;
// The function is everything after the meta comment's closing `*/`.
let after = raw[end..].find("*/").map(|i| end + i + 2).unwrap_or(end);
let func_src = raw[after..].trim().to_string();
if func_src.is_empty() {
return Err(format!("site: {spec} has no function body after @meta"));
}
Ok(Adapter { meta, func_src })
}
/// Build the JS to eval: `(<adapter function>)(<args JSON>)`. The adapter's
/// `async function(args)` returns a promise; chrome-use's eval awaits it.
pub fn build_eval(adapter: &Adapter, args: &Value) -> String {
let args_json = serde_json::to_string(args).unwrap_or_else(|_| "{}".to_string());
format!("({})({})", adapter.func_src, args_json)
}
/// List installed adapters as `name/cmd` strings (sorted).
pub fn list_adapters() -> Result<Vec<String>, String> {
let dir = sites_dir().ok_or("site: cannot resolve home dir")?;
if !dir.exists() {
return Ok(Vec::new());
}
let mut out = Vec::new();
for site in std::fs::read_dir(&dir)
.map_err(|e| e.to_string())?
.flatten()
{
if !site.path().is_dir() {
continue;
}
let name = site.file_name().to_string_lossy().to_string();
for cmd in std::fs::read_dir(site.path())
.map_err(|e| e.to_string())?
.flatten()
{
let p = cmd.path();
if p.extension().and_then(|e| e.to_str()) == Some("js") {
if let Some(stem) = p.file_stem().and_then(|s| s.to_str()) {
out.push(format!("{name}/{stem}"));
}
}
}
}
out.sort();
Ok(out)
}
/// Download the bb-sites repo zip and extract its adapters into `~/.chrome-use/sites`.
pub async fn update() -> Result<usize, String> {
let dir = sites_dir().ok_or("site: cannot resolve home dir")?;
let client = reqwest::Client::builder()
.user_agent("chrome-use")
.build()
.map_err(|e| e.to_string())?;
let bytes = client
.get(SITES_ZIP_URL)
.send()
.await
.map_err(|e| format!("site update: download failed: {e}"))?
.error_for_status()
.map_err(|e| format!("site update: {e}"))?
.bytes()
.await
.map_err(|e| format!("site update: read body: {e}"))?;
let cursor = std::io::Cursor::new(bytes);
let mut zip = zip::ZipArchive::new(cursor).map_err(|e| format!("site update: bad zip: {e}"))?;
std::fs::create_dir_all(&dir).map_err(|e| e.to_string())?;
let mut count = 0usize;
for i in 0..zip.len() {
let mut f = zip.by_index(i).map_err(|e| e.to_string())?;
let Some(enclosed) = f.enclosed_name() else {
continue;
};
// Strip the top-level `bb-sites-main/` component from the archive path.
let rel: PathBuf = enclosed.components().skip(1).collect();
if rel.as_os_str().is_empty() {
continue;
}
let out = dir.join(&rel);
if f.is_dir() {
let _ = std::fs::create_dir_all(&out);
continue;
}
if let Some(parent) = out.parent() {
let _ = std::fs::create_dir_all(parent);
}
let mut buf = Vec::new();
std::io::copy(&mut f, &mut buf).map_err(|e| e.to_string())?;
std::fs::write(&out, &buf).map_err(|e| e.to_string())?;
if out.extension().and_then(|e| e.to_str()) == Some("js") {
count += 1;
}
}
// Build the domain→adapters index and stamp the sync time so navigation can
// suggest adapters (auto-trigger) and `needs_refresh` can pace re-syncs.
write_domain_index(&dir);
if let Some(p) = last_update_path() {
let _ = std::fs::write(p, now_secs().to_string());
}
Ok(count)
}
/// `~/.chrome-use/sites/.last_update` — unix-seconds marker of the last sync.
fn last_update_path() -> Option<PathBuf> {
sites_dir().map(|d| d.join(".last_update"))
}
/// `~/.chrome-use/sites/.index.json` — `{ "github.com": ["github/issues", …], … }`,
/// built on `update` so navigation can look up adapters by domain without parsing
/// all ~145 adapter files on every command.
fn index_path() -> Option<PathBuf> {
sites_dir().map(|d| d.join(".index.json"))
}
fn now_secs() -> u64 {
std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.map(|d| d.as_secs())
.unwrap_or(0)
}
/// Parse every installed adapter and write the domain→adapters index. Within a
/// domain, read-only adapters are listed first (then alphabetical) so the
/// auto-suggested example leads with a safe read, not a write action.
fn write_domain_index(dir: &std::path::Path) {
let mut by_domain: std::collections::BTreeMap<String, Vec<(bool, String)>> = Default::default();
for spec in list_adapters().unwrap_or_default() {
if let Ok(a) = load_adapter(&spec) {
if let Some(d) = a.domain() {
let read_only = a
.meta
.get("readOnly")
.and_then(|v| v.as_bool())
.unwrap_or(false);
by_domain
.entry(d.to_string())
.or_default()
.push((read_only, spec));
}
}
}
let ordered: std::collections::BTreeMap<String, Vec<String>> = by_domain
.into_iter()
.map(|(domain, mut v)| {
// read-only (true) first, then by spec name
v.sort_by(|a, b| b.0.cmp(&a.0).then_with(|| a.1.cmp(&b.1)));
(domain, v.into_iter().map(|(_, s)| s).collect())
})
.collect();
if let Ok(json) = serde_json::to_string(&ordered) {
let _ = std::fs::write(dir.join(".index.json"), json);
}
}
const DEFAULT_TTL_DAYS: u64 = 7;
/// Whether the adapter pack should be (re)synced: true on first use (nothing
/// installed) or when the last sync is older than the TTL. Disabled by
/// `AGENT_BROWSER_SITES_NO_AUTO_UPDATE=1`; TTL overridable via
/// `AGENT_BROWSER_SITES_TTL_DAYS` (0 = always).
pub fn needs_refresh() -> bool {
if std::env::var_os("AGENT_BROWSER_SITES_NO_AUTO_UPDATE").is_some() {
return false;
}
let Some(dir) = sites_dir() else {
return false;
};
// First use: no adapters installed yet.
if list_adapters().map(|l| l.is_empty()).unwrap_or(true) {
let _ = &dir;
return true;
}
let ttl_days = std::env::var("AGENT_BROWSER_SITES_TTL_DAYS")
.ok()
.and_then(|s| s.parse::<u64>().ok())
.unwrap_or(DEFAULT_TTL_DAYS);
let ttl = ttl_days.saturating_mul(86_400);
match last_update_path().and_then(|p| std::fs::read_to_string(p).ok()) {
Some(s) => match s.trim().parse::<u64>() {
Ok(ts) => now_secs().saturating_sub(ts) >= ttl,
Err(_) => true,
},
None => true, // no marker → treat as stale
}
}
/// Adapters whose `@meta.domain` matches `host` (exact, or `host` is a subdomain
/// of it) — for auto-suggesting `site` commands when you land on a known site.
/// Reads the prebuilt `.index.json`; empty if the pack isn't synced yet.
pub fn adapters_for_domain(host: &str) -> Vec<String> {
let host = host.trim_start_matches("www.");
let Some(raw) = index_path().and_then(|p| std::fs::read_to_string(p).ok()) else {
return Vec::new();
};
let Ok(idx) = serde_json::from_str::<std::collections::BTreeMap<String, Vec<String>>>(&raw)
else {
return Vec::new();
};
// Preserve the index's per-domain ordering (read-only adapters first); just
// dedup if a host somehow matches multiple domain keys.
let mut out: Vec<String> = Vec::new();
for (domain, specs) in idx {
let d = domain.trim_start_matches("www.");
if host == d || host.ends_with(&format!(".{d}")) {
for s in specs {
if !out.contains(&s) {
out.push(s);
}
}
}
}
out
}
/// Map CLI args to the adapter's `args` object. Positional args fill the adapter's
/// declared `args` keys in order; `--key value` overrides by name. The adapter
/// validates required args itself.
pub fn map_args(adapter: &Adapter, positional: &[String], named: &[(String, String)]) -> Value {
let mut obj = serde_json::Map::new();
let keys: Vec<String> = adapter
.meta
.get("args")
.and_then(|a| a.as_object())
.map(|m| m.keys().cloned().collect())
.unwrap_or_default();
for (i, val) in positional.iter().enumerate() {
if let Some(k) = keys.get(i) {
obj.insert(k.clone(), Value::String(val.clone()));
}
}
for (k, v) in named {
obj.insert(k.clone(), Value::String(v.clone()));
}
Value::Object(obj)
}
#[cfg(test)]
mod tests {
use super::*;
const SAMPLE: &str = r#"/* @meta
{
"name": "github/issues",
"domain": "github.com",
"args": { "repo": {"required": true}, "state": {"required": false} }
}
*/
async function(args) { return { repo: args.repo }; }"#;
#[test]
fn parses_meta_and_function() {
let a = parse_adapter(SAMPLE, "github/issues").unwrap();
assert_eq!(a.domain(), Some("github.com"));
assert!(a.func_src.starts_with("async function(args)"));
}
#[test]
fn build_eval_wraps_and_passes_args() {
let a = parse_adapter(SAMPLE, "github/issues").unwrap();
let args = map_args(
&a,
&["owner/repo".into()],
&[("state".into(), "closed".into())],
);
let js = build_eval(&a, &args);
assert!(js.contains("async function(args)"));
assert!(js.contains("\"repo\":\"owner/repo\""));
assert!(js.contains("\"state\":\"closed\""));
}
#[test]
fn rejects_bad_spec() {
assert!(load_adapter("noslash").is_err());
assert!(load_adapter("../etc/passwd").is_err());
}
}
+522
View File
@@ -0,0 +1,522 @@
//! `chrome-use test <suite.yaml>` — a tiny, re-runnable browser test runner.
//!
//! Turns repetitive browser checks into unit-test-style suites for the frontend.
//! A suite is a YAML file of cases; each case is a list of `steps` (which reuse
//! chrome-use's own commands) followed by `assert`s (which compile to a single
//! `eval` expression read back as a boolean). The runner drives the session by
//! re-invoking the chrome-use binary per step, so it inherits every flag /
//! launch / daemon / `@ref` semantic for free; the daemon stays up for the
//! session, so each step is just a fast socket round-trip.
//!
//! ```yaml
//! suite: chatgpt smoke
//! setup:
//! - account: chatgpt/huayue # cookie-use injects this login (optional)
//! cases:
//! - name: home loads logged in
//! steps:
//! - open: https://chatgpt.com/
//! - wait: { load: networkidle }
//! assert:
//! - url: { contains: chatgpt.com }
//! - visible: "#prompt-textarea"
//! ```
use crate::flags::Flags;
use serde_json::Value;
use std::process::Command;
use std::time::Instant;
pub fn run_test(suite_path: &str, flags: &Flags) -> i32 {
let text = match std::fs::read_to_string(suite_path) {
Ok(t) => t,
Err(e) => {
eprintln!("{} cannot read suite '{}': {}", err(), suite_path, e);
return 2;
}
};
// YAML deserializes straight into serde_json::Value (maps→objects, etc.).
let suite: Value = match serde_yaml::from_str(&text) {
Ok(v) => v,
Err(e) => {
eprintln!("{} invalid YAML in '{}': {}", err(), suite_path, e);
return 2;
}
};
let cases = match suite.get("cases").and_then(|c| c.as_array()) {
Some(c) if !c.is_empty() => c.clone(),
_ => {
eprintln!("{} suite has no `cases`", err());
return 2;
}
};
let suite_name = suite
.get("suite")
.and_then(|s| s.as_str())
.unwrap_or("suite");
let exe = match std::env::current_exe() {
Ok(p) => p.to_string_lossy().into_owned(),
Err(e) => {
eprintln!("{} cannot find own binary: {}", err(), e);
return 2;
}
};
// A dedicated launched browser by default (deterministic, re-runnable). If
// the user named a --session, target that existing one instead.
let (session, do_launch) = if flags.session == "default" {
("cu-test".to_string(), true)
} else {
(flags.session.clone(), flags.force_launch)
};
let owns_session = session == "cu-test";
let mut base: Vec<String> = vec!["--session".into(), session.clone()];
if do_launch {
base.push("--launch".into());
}
if let Some(p) = &flags.profile {
base.push("--profile".into());
base.push(p.clone());
}
let artifacts_dir = flags
.download_path
.clone()
.unwrap_or_else(|| "cu-test-artifacts".to_string());
let runner = Runner {
exe,
base,
artifacts_dir,
};
// --- setup (runs once) ---
if let Some(setup) = suite.get("setup").and_then(|s| s.as_array()) {
for item in setup {
if let Err(e) = runner.run_setup_item(item, &session) {
eprintln!("{} setup failed: {}", err(), e);
if owns_session {
runner.close();
}
return 2;
}
}
}
// --- cases ---
println!("suite: {} (session {})", suite_name, session);
let mut passed = 0usize;
let mut failed = 0usize;
for case in &cases {
let name = case
.get("name")
.and_then(|n| n.as_str())
.unwrap_or("(unnamed)");
let start = Instant::now();
let outcome = runner.run_case(case);
let secs = start.elapsed().as_secs_f64();
match outcome {
Ok(()) => {
passed += 1;
println!(" {} {} {:.1}s", ok(), name, secs);
}
Err(failure) => {
failed += 1;
println!(" {} {} {:.1}s", cross(), name, secs);
println!(" {}", failure.reason);
if let Some(shot) = runner.capture_artifact(name) {
println!("{}", shot);
}
}
}
}
if owns_session {
runner.close();
}
println!(
"{} cases · {} passed · {} failed",
cases.len(),
passed,
failed
);
i32::from(failed > 0)
}
struct Failure {
reason: String,
}
struct Runner {
exe: String,
base: Vec<String>,
artifacts_dir: String,
}
impl Runner {
/// Run one chrome-use sub-command. Returns the `data` object on success.
fn cli(&self, args: &[String]) -> Result<Option<Value>, String> {
let out = Command::new(&self.exe)
.args(&self.base)
.args(args)
.arg("--json")
.output()
.map_err(|e| format!("spawning chrome-use: {}", e))?;
let stdout = String::from_utf8_lossy(&out.stdout);
if let Ok(v) = serde_json::from_str::<Value>(stdout.trim()) {
let success = v
.get("success")
.and_then(|b| b.as_bool())
.unwrap_or(out.status.success());
if !success {
return Err(v
.get("error")
.and_then(|e| e.as_str())
.unwrap_or("command failed")
.to_string());
}
return Ok(v.get("data").cloned());
}
if out.status.success() {
Ok(None)
} else {
Err(String::from_utf8_lossy(&out.stderr).trim().to_string())
}
}
fn close(&self) {
let _ = self.cli(&["close".to_string()]);
}
fn run_setup_item(&self, item: &Value, session: &str) -> Result<(), String> {
// `account: <id>` injects a stored cookie-use login into this session.
if let Some(acct) = item.get("account").and_then(|a| a.as_str()) {
let target = format!("session:{}", session);
let out = Command::new("cookie-use")
.args(["use", acct, "--target", &target, "--no-open"])
.output();
return match out {
Ok(o) if o.status.success() => Ok(()),
Ok(o) => Err(format!(
"cookie-use use {} failed: {}",
acct,
String::from_utf8_lossy(&o.stderr).trim()
)),
Err(e) => Err(format!(
"cookie-use not available ({}); skip `account:` or install it",
e
)),
};
}
// Otherwise it's a normal step.
let args = step_to_args(item)?;
self.cli(&args).map(|_| ())
}
fn run_case(&self, case: &Value) -> Result<(), Failure> {
if let Some(steps) = case.get("steps").and_then(|s| s.as_array()) {
for step in steps {
let args = step_to_args(step).map_err(|e| Failure {
reason: format!("bad step: {}", e),
})?;
self.cli(&args).map_err(|e| Failure {
reason: format!(
"step `{}` failed: {}",
args.first().cloned().unwrap_or_default(),
e
),
})?;
}
}
if let Some(asserts) = case.get("assert").and_then(|a| a.as_array()) {
for a in asserts {
let (expr, describe) = assert_to_eval(a).map_err(|e| Failure {
reason: format!("bad assert: {}", e),
})?;
let data = self.cli(&["eval".to_string(), expr]).map_err(|e| Failure {
reason: format!("assert `{}` could not run: {}", describe, e),
})?;
let result = data.as_ref().and_then(|d| d.get("result"));
if !is_truthy(result) {
let got = result
.map(value_short)
.unwrap_or_else(|| "undefined".into());
return Err(Failure {
reason: format!("assert {} → got {}", describe, got),
});
}
}
}
Ok(())
}
/// Best-effort screenshot of the failing state. Returns the saved path.
fn capture_artifact(&self, case_name: &str) -> Option<String> {
let _ = std::fs::create_dir_all(&self.artifacts_dir);
let path = format!("{}/{}.png", self.artifacts_dir, slug(case_name));
match self.cli(&["screenshot".to_string(), path.clone()]) {
Ok(Some(d)) => d
.get("path")
.and_then(|p| p.as_str())
.map(String::from)
.or(Some(path)),
Ok(None) => Some(path),
Err(_) => None,
}
}
}
/// Map a YAML step (a one-key object) to chrome-use CLI args.
fn step_to_args(step: &Value) -> Result<Vec<String>, String> {
let obj = step
.as_object()
.ok_or_else(|| "step must be a key: value mapping".to_string())?;
let (key, val) = obj.iter().next().ok_or_else(|| "empty step".to_string())?;
let s = |v: &Value| v.as_str().map(String::from);
match key.as_str() {
"open" | "goto" | "navigate" => {
let url = s(val).ok_or("open: expected a URL string")?;
Ok(vec!["open".into(), url])
}
"click" => Ok(vec![
"click".into(),
s(val).ok_or("click: expected a selector")?,
]),
"press" => Ok(vec!["press".into(), s(val).ok_or("press: expected a key")?]),
"eval" => Ok(vec![
"eval".into(),
s(val).ok_or("eval: expected JS string")?,
]),
"fill" | "type" => {
let sel = field(val, &["sel", "selector"]).ok_or("fill/type: need sel")?;
let text = field(val, &["text", "value"]).ok_or("fill/type: need text")?;
Ok(vec![key.clone(), sel, text])
}
"scroll" => {
if let Some(dir) = s(val) {
Ok(vec!["scroll".into(), dir])
} else {
let dir = field(val, &["dir", "direction"]).ok_or("scroll: need dir")?;
let mut a = vec!["scroll".into(), dir];
if let Some(px) = field(val, &["px", "pixels"]) {
a.push(px);
}
Ok(a)
}
}
"wait" => {
if let Some(n) = val.as_i64() {
Ok(vec!["wait".into(), n.to_string()])
} else if let Some(load) = field(val, &["load"]) {
Ok(vec!["wait".into(), "--load".into(), load])
} else if let Some(sel) = s(val) {
Ok(vec!["wait".into(), sel])
} else {
Err("wait: expected ms, a selector, or { load: <state> }".into())
}
}
other => Err(format!("unknown step `{}`", other)),
}
}
/// Compile a YAML assert (one-key object) into (js-bool-expr, human-describe).
fn assert_to_eval(a: &Value) -> Result<(String, String), String> {
let obj = a
.as_object()
.ok_or_else(|| "assert must be a key: value mapping".to_string())?;
let (key, val) = obj
.iter()
.next()
.ok_or_else(|| "empty assert".to_string())?;
match key.as_str() {
"url" => {
let (op, want) = str_op(val).ok_or("url: need contains/equals/matches")?;
Ok((
cmp_expr("location.href", &op, &want),
format!("url {} {:?}", op, want),
))
}
"visible" => {
let sel = val.as_str().ok_or("visible: expected a selector")?;
Ok((visible_expr(sel), format!("visible {:?}", sel)))
}
"hidden" => {
let sel = val.as_str().ok_or("hidden: expected a selector")?;
Ok((
format!("!({})", visible_expr(sel)),
format!("hidden {:?}", sel),
))
}
"text" => {
let sel = field(val, &["sel", "selector"]).ok_or("text: need sel")?;
let (op, want) = str_op(val).ok_or("text: need contains/equals/matches")?;
let base = format!(
"((document.querySelector({})||{{}}).textContent||\"\")",
js(&sel)
);
Ok((
cmp_expr(&base, &op, &want),
format!("text {:?} {} {:?}", sel, op, want),
))
}
"count" => {
let sel = field(val, &["sel", "selector"]).ok_or("count: need sel")?;
let n = val
.get("eq")
.or_else(|| val.get("equals"))
.and_then(|v| v.as_i64())
.ok_or("count: need eq: <n>")?;
Ok((
format!("document.querySelectorAll({}).length==={}", js(&sel), n),
format!("count {:?} == {}", sel, n),
))
}
"eval" => {
let expr = val.as_str().ok_or("eval: expected JS string")?;
Ok((format!("!!({})", expr), format!("eval {:?}", expr)))
}
other => Err(format!("unknown assert `{}`", other)),
}
}
fn visible_expr(sel: &str) -> String {
format!(
"(function(){{var e=document.querySelector({});return !!(e&&(e.offsetWidth||e.offsetHeight||e.getClientRects().length));}})()",
js(sel)
)
}
/// Extract (op, want) from `{contains|equals|matches: <str>}`.
fn str_op(val: &Value) -> Option<(String, String)> {
for op in ["contains", "equals", "matches"] {
if let Some(s) = val.get(op).and_then(|v| v.as_str()) {
return Some((op.to_string(), s.to_string()));
}
}
None
}
fn cmp_expr(base: &str, op: &str, want: &str) -> String {
match op {
"equals" => format!("({})==={}", base, js(want)),
"matches" => format!("new RegExp({}).test({})", js(want), base),
_ => format!("({}).includes({})", base, js(want)), // contains
}
}
/// First present field among `keys`, as a string.
fn field(val: &Value, keys: &[&str]) -> Option<String> {
for k in keys {
if let Some(v) = val.get(*k) {
return match v {
Value::String(s) => Some(s.clone()),
Value::Number(n) => Some(n.to_string()),
Value::Bool(b) => Some(b.to_string()),
_ => None,
};
}
}
None
}
/// JSON-encode a string so it embeds safely as a JS literal.
fn js(s: &str) -> String {
serde_json::to_string(s).unwrap_or_else(|_| "\"\"".into())
}
fn is_truthy(v: Option<&Value>) -> bool {
match v {
Some(Value::Bool(b)) => *b,
Some(Value::Null) | None => false,
Some(Value::Number(n)) => n.as_f64().map(|f| f != 0.0).unwrap_or(false),
Some(Value::String(s)) => !s.is_empty(),
Some(_) => true,
}
}
fn value_short(v: &Value) -> String {
let s = v.to_string();
if s.len() > 60 {
format!("{}", &s[..60])
} else {
s
}
}
fn slug(name: &str) -> String {
let s: String = name
.chars()
.map(|c| if c.is_alphanumeric() { c } else { '-' })
.collect();
s.trim_matches('-').to_lowercase()
}
#[cfg(test)]
mod tests {
use super::*;
use serde_json::json;
#[test]
fn step_mapping() {
assert_eq!(
step_to_args(&json!({"open": "https://x.com"})).unwrap(),
vec!["open", "https://x.com"]
);
assert_eq!(
step_to_args(&json!({"fill": {"sel": "#a", "text": "hi"}})).unwrap(),
vec!["fill", "#a", "hi"]
);
assert_eq!(
step_to_args(&json!({"wait": {"load": "networkidle"}})).unwrap(),
vec!["wait", "--load", "networkidle"]
);
assert_eq!(
step_to_args(&json!({"wait": 500})).unwrap(),
vec!["wait", "500"]
);
assert!(step_to_args(&json!({"bogus": 1})).is_err());
}
#[test]
fn assert_compilation() {
let (e, _) = assert_to_eval(&json!({"url": {"contains": "x.com"}})).unwrap();
assert!(e.contains("location.href") && e.contains(".includes("));
let (e, _) = assert_to_eval(&json!({"count": {"sel": ".a", "eq": 3}})).unwrap();
assert!(e.contains("querySelectorAll") && e.ends_with("===3"));
let (e, _) = assert_to_eval(&json!({"hidden": "#x"})).unwrap();
assert!(e.starts_with("!("));
let (e, _) = assert_to_eval(&json!({"eval": "window.ok"})).unwrap();
assert_eq!(e, "!!(window.ok)");
assert!(assert_to_eval(&json!({"bogus": 1})).is_err());
}
#[test]
fn truthiness() {
assert!(is_truthy(Some(&json!(true))));
assert!(!is_truthy(Some(&json!(false))));
assert!(!is_truthy(None));
assert!(!is_truthy(Some(&json!(""))));
assert!(is_truthy(Some(&json!("x"))));
assert!(!is_truthy(Some(&json!(0))));
}
#[test]
fn js_escaping() {
// Selectors with quotes must embed safely.
assert_eq!(js(r#"a"b"#), r#""a\"b""#);
}
}
fn ok() -> &'static str {
"\x1b[32m✓\x1b[0m"
}
fn cross() -> &'static str {
"\x1b[31m✗\x1b[0m"
}
fn err() -> &'static str {
"\x1b[31merror:\x1b[0m"
}
+161 -1
View File
@@ -1,5 +1,7 @@
use crate::color;
use std::process::{exit, Command};
use std::path::PathBuf;
use std::process::{exit, Command, Stdio};
use std::time::{SystemTime, UNIX_EPOCH};
const CURRENT_VERSION: &str = env!("CARGO_PKG_VERSION");
@@ -7,6 +9,164 @@ const CURRENT_VERSION: &str = env!("CARGO_PKG_VERSION");
/// upgrade path and the install path are identical (GitHub Release, no npm).
const INSTALL_URL: &str = "https://raw.githubusercontent.com/leeguooooo/chrome-use/main/install.sh";
/// GitHub API for the latest published release (used by the update check).
const LATEST_RELEASE_API: &str =
"https://api.github.com/repos/leeguooooo/chrome-use/releases/latest";
/// Re-check the latest version at most this often (seconds).
const UPDATE_CHECK_INTERVAL_SECS: u64 = 86_400; // once a day
fn now_secs() -> u64 {
SystemTime::now()
.duration_since(UNIX_EPOCH)
.map(|d| d.as_secs())
.unwrap_or(0)
}
fn update_cache_path() -> PathBuf {
crate::connection::config_home().join("update-check.json")
}
fn write_update_cache(checked_at: u64, latest: &str) {
let path = update_cache_path();
if let Some(parent) = path.parent() {
let _ = std::fs::create_dir_all(parent);
}
let body = serde_json::json!({ "checked_at": checked_at, "latest": latest }).to_string();
let _ = std::fs::write(&path, body);
}
/// Parse a dotted version (`1.2.1`, `v1.2.1`, `1.2.1-fork.3`) into a comparable
/// `(major, minor, patch)`, ignoring any pre-release/build suffix.
fn parse_version(v: &str) -> Option<(u64, u64, u64)> {
let core = v.trim().trim_start_matches('v');
let core = core.split(['-', '+']).next().unwrap_or(core);
let mut parts = core.split('.');
let major = parts.next()?.parse().ok()?;
let minor = parts.next().unwrap_or("0").parse().ok()?;
let patch = parts.next().unwrap_or("0").parse().ok()?;
Some((major, minor, patch))
}
fn is_newer(latest: &str, current: &str) -> bool {
matches!((parse_version(latest), parse_version(current)), (Some(l), Some(c)) if l > c)
}
/// Public semver-ish comparison (`latest` strictly newer than `current`), so
/// `doctor` can flag a stale extension/CLI without re-implementing parsing.
pub fn version_is_newer(latest: &str, current: &str) -> bool {
is_newer(latest, current)
}
/// The latest CLI version recorded by the background update check, if any.
/// `doctor` uses it to show "a newer chrome-use is available" without a network
/// call (the `__update-check` worker refreshes the cache out of band).
pub fn cached_latest_version() -> Option<String> {
std::fs::read_to_string(update_cache_path())
.ok()
.and_then(|s| serde_json::from_str::<serde_json::Value>(&s).ok())
.and_then(|j| {
j.get("latest")
.and_then(|v| v.as_str())
.map(|s| s.to_string())
})
.filter(|s| !s.is_empty())
}
/// Hidden `__update-check` subcommand: fetch the latest release tag and cache it.
/// Spawned detached by [`maybe_notify_update`] so the network call never blocks a
/// real command. Uses `curl` (no extra deps, matches `upgrade`).
pub fn run_update_check() {
let latest = Command::new("curl")
.args([
"-fsSL",
"--max-time",
"8",
"-H",
"User-Agent: chrome-use-update-check",
LATEST_RELEASE_API,
])
.output()
.ok()
.filter(|o| o.status.success())
.and_then(|o| serde_json::from_slice::<serde_json::Value>(&o.stdout).ok())
.and_then(|j| {
j.get("tag_name")
.and_then(|v| v.as_str())
.map(|s| s.trim_start_matches('v').to_string())
});
if let Some(latest) = latest {
write_update_cache(now_secs(), &latest);
}
}
/// Non-blocking "update available" notice. Called once per command run:
/// - prints a one-line hint to **stderr** (never stdout, so `--json` is clean)
/// when a cached release is newer than the running binary;
/// - refreshes the cached latest version at most once a day via a **detached**
/// background process, so the current command never waits on the network.
///
/// Skipped for meta commands (upgrade/install/doctor/`__*`/--version/--help),
/// in CI, in daemon mode, and when CHROME_USE_NO_UPDATE_CHECK /
/// AGENT_BROWSER_NO_UPDATE_CHECK is set.
pub fn maybe_notify_update() {
if std::env::var_os("CHROME_USE_NO_UPDATE_CHECK").is_some()
|| std::env::var_os("AGENT_BROWSER_NO_UPDATE_CHECK").is_some()
|| std::env::var_os("CI").is_some()
|| std::env::var_os("AGENT_BROWSER_DAEMON").is_some()
{
return;
}
let first = std::env::args().nth(1).unwrap_or_default();
if first.starts_with("__")
|| matches!(
first.as_str(),
"upgrade" | "install" | "doctor" | "dashboard" | "daemon"
)
{
return;
}
if std::env::args().any(|a| matches!(a.as_str(), "--version" | "-V" | "--help" | "-h")) {
return;
}
let (checked_at, latest) = std::fs::read_to_string(update_cache_path())
.ok()
.and_then(|s| serde_json::from_str::<serde_json::Value>(&s).ok())
.map(|j| {
(
j.get("checked_at").and_then(|v| v.as_u64()).unwrap_or(0),
j.get("latest")
.and_then(|v| v.as_str())
.unwrap_or_default()
.to_string(),
)
})
.unwrap_or((0, String::new()));
if is_newer(&latest, CURRENT_VERSION) {
eprintln!(
"{} chrome-use {latest} is available (you have {CURRENT_VERSION}) — run `chrome-use upgrade`",
color::warning_indicator()
);
}
// Refresh in the background at most once a day. Bump the timestamp first
// (keeping the last-known latest) so concurrent runs don't all spawn a
// checker, then fire a detached child that does the network fetch.
if now_secs().saturating_sub(checked_at) >= UPDATE_CHECK_INTERVAL_SECS {
write_update_cache(now_secs(), &latest);
if let Ok(exe) = std::env::current_exe() {
let _ = Command::new(exe)
.arg("__update-check")
.stdin(Stdio::null())
.stdout(Stdio::null())
.stderr(Stdio::null())
.spawn();
}
}
}
/// Upgrade to the latest GitHub Release.
///
/// The stealth fork ships as a prebuilt binary attached to a GitHub Release —
Binary file not shown.
Binary file not shown.
+1 -1
View File
@@ -6,6 +6,6 @@ from **openclaw-browser-relay** by chengyixu
Changes for chrome-use: rebranded to "chrome-use connect"; the
transport is rewritten from a localhost WebSocket + shared token to Chrome
**native messaging** (host `com.leeguoo.chrome_use`) — no port, no token,
**native messaging** (host `com.agent_browser.connect`) — no port, no token,
Chrome authenticates the extension to the host by id. WebSocket/token/options
code removed.
+163 -18
View File
@@ -16,7 +16,7 @@
// attach + Target handling; the transport is rewritten from WebSocket+token to
// native messaging.
const HOST_NAME = 'com.leeguoo.chrome_use'
const HOST_NAME = 'com.agent_browser.connect'
const SKIP_URL = /^(chrome|chrome-extension|devtools|chrome-untrusted|edge|about):/i
/** @type {chrome.runtime.Port|null} */
@@ -24,13 +24,22 @@ let port = null
/** Whether the native-messaging host (the local chrome-use CLI) is linked.
* Read by the popup status page. */
let hostConnected = false
let nextSession = 1
/** tabId -> { sessionId, targetId } */
const tabs = new Map()
/** sessionId -> tabId (main session per tab) */
const sessionToTab = new Map()
/** child (OOPIF/worker) sessionId -> tabId */
const childSessionToTab = new Map()
/** sessionId -> CDP targetId, kept ACROSS detach so a dead `cb-tab-<oldTabId>`
* session can be recovered by its stable targetId when the cross-process nav
* gave the tab a new Chrome tabId (issue #24). Capped to bound memory. */
const sessionTargets = new Map()
function rememberSessionTarget(sessionId, targetId) {
if (!sessionId || !targetId) return
sessionTargets.delete(sessionId)
sessionTargets.set(sessionId, targetId)
if (sessionTargets.size > 256) sessionTargets.delete(sessionTargets.keys().next().value)
}
/** tab-group name -> chrome tabGroups id (best-effort cache) */
const groupIdByName = new Map()
@@ -109,6 +118,12 @@ function connectHost() {
// reconnect. Keep chrome.debugger attached so reconnect is cheap.
for (const tabId of tabs.keys()) setBadge(tabId, 'connecting')
})
// Report our version so the host can tell the CLI/`doctor` which extension
// build is live (otherwise the extension version is a black box — the user
// can't tell they're on an old one). Best-effort; ignored by older hosts.
try {
postToHost({ method: 'hello', version: chrome.runtime.getManifest().version })
} catch {}
// Tell the daemon about everything we already have attached, then attach
// anything new.
reannounceAttachedTabs()
@@ -150,6 +165,90 @@ function tabForTarget(targetId) {
return null
}
// The STABLE Chrome tabId encoded in a `cb-tab-<tabId>` session id (#17), or
// null for any other session shape (child/iframe sessions). The tabId is the
// real source of truth: it survives the renderer-process swaps (cross-origin
// OAuth/SSO navs) that tear down the page's CDP target — which is why binding to
// it (like claude-in-chrome) rides through the hop that killed the old
// target/sessionId binding (issue #23).
function tabIdFromSession(sessionId) {
const m = /^cb-tab-(\d+)$/.exec(sessionId || '')
return m ? Number(m[1]) : null
}
// Ensure the debugger is attached to a `cb-tab-<tabId>` session's tab, re-attaching
// across the transient window of a process swap (with a couple of short retries).
// Returns the tabId on success, or null when the tab is genuinely gone
// (closed / restricted). (issues #20.1, #23)
async function recoverSessionTab(sessionId) {
const tabId = tabIdFromSession(sessionId)
// 1) Fast path: the encoded Chrome tabId still exists — re-attach it (covers
// the common renderer-process swap where the tabId is preserved, #23).
if (tabId != null) {
for (let i = 0; i < 3; i++) {
const tab = await chrome.tabs.get(tabId).catch(() => null)
if (!eligible(tab)) break // tabId is gone — fall through to targetId recovery
try {
await attachTab(tabId)
if (tabs.has(tabId)) return tabId
} catch {
// mid-swap: tab exists but isn't attachable yet — back off and retry.
}
await new Promise((r) => setTimeout(r, 120 + i * 150))
}
}
// 2) The Chrome tabId is gone, but the CDP targetId is STABLE across the nav.
// Some cross-process hops (Mercari's signin token exchange) give the tab a
// NEW tabId while keeping the same target, so `cb-tab-<oldTabId>` can't be
// recovered by tabId. Find the tab now hosting our remembered targetId via
// chrome.debugger.getTargets(), attach it, and ALIAS the dead session to it
// so the daemon's session id keeps resolving. Longer window: this hop can
// take several seconds to settle (issue #24).
const targetId = sessionTargets.get(sessionId)
if (targetId) {
for (let i = 0; i < 6; i++) {
const targets = await chrome.debugger.getTargets().catch(() => null)
const t = targets && targets.find((x) => x.id === targetId && x.tabId != null)
if (t && t.tabId != null) {
const tab = await chrome.tabs.get(t.tabId).catch(() => null)
if (eligible(tab)) {
try {
await attachTab(t.tabId)
if (tabs.has(t.tabId)) {
sessionToTab.set(sessionId, t.tabId) // alias dead session -> live tab
return t.tabId
}
} catch {
// not attachable yet — keep waiting for the swap to settle.
}
}
}
await new Promise((r) => setTimeout(r, 300 + i * 300))
}
}
return null
}
// Send a CDP command to a tab, riding a debugger detach that can happen between
// our attach check and the command itself (a renderer-process swap mid-flight).
// On a detached-style failure, drop the stale handle, re-attach the stable tab,
// and retry once — so a cross-process nav never surfaces as a hard error (#23).
async function sendCdpToTab(tabId, method, params) {
const dbg = { tabId }
try {
return await chrome.debugger.sendCommand(dbg, method, params)
} catch (e) {
const msg = String((e && e.message) || e)
if (!/detached|not attached|target.*(closed|gone)|no target|cannot access|frame.*detached/i.test(msg)) {
throw e
}
detachTab(tabId, false)
const ok = await recoverSessionTab(`cb-tab-${tabId}`)
if (!ok) throw e
return await chrome.debugger.sendCommand(dbg, method, params)
}
}
function anyConnectedTab() {
const it = tabs.keys().next()
return it.done ? null : it.value
@@ -208,13 +307,26 @@ async function handleForwardCdpCommand(msg) {
// Fail loudly instead so the agent sees an actionable error, not bad data.
let tabId
if (sessionId) {
tabId = tabForSession(sessionId)
if (!tabId) {
throw new Error(
`stale sessionId ${sessionId} for ${method}: its tab is gone (closed, ` +
`navigated across processes, or lost after an extension restart). ` +
`Re-attach by re-opening your target URL before retrying.`,
)
// The stable Chrome tabId encoded in `cb-tab-<tabId>` is the source of truth
// (it survives renderer-process swaps; the CDP target/sessionId does not).
// Resolve via it primarily — don't depend on a session→tab map entry that the
// detach handler may have cleared — and ensure the debugger is attached,
// re-attaching across a cross-process nav before failing (issues #20.1, #23).
// `tabForSession` still covers child/iframe sessions that aren't `cb-tab-*`.
tabId = tabIdFromSession(sessionId) ?? tabForSession(sessionId)
if (tabId == null) {
throw new Error(`unknown sessionId ${sessionId} for ${method}`)
}
if (!tabs.has(tabId)) {
const recovered = await recoverSessionTab(sessionId)
if (!recovered) {
throw new Error(
`stale sessionId ${sessionId} for ${method}: its tab is gone (closed, ` +
`navigated across processes, or lost after an extension restart). ` +
`Re-attach by re-opening your target URL before retrying.`,
)
}
tabId = recovered
}
} else if (typeof params?.targetId === 'string') {
tabId = tabForTarget(params.targetId)
@@ -224,18 +336,17 @@ async function handleForwardCdpCommand(msg) {
// applies to any attached tab.
tabId = anyConnectedTab()
}
if (!tabId) throw new Error(`no attached tab for ${method}`)
const dbg = { tabId }
if (tabId == null) throw new Error(`no attached tab for ${method}`)
// Re-enabling Runtime can leave a stale state; bounce it (matches upstream).
if (method === 'Runtime.enable') {
try {
await chrome.debugger.sendCommand(dbg, 'Runtime.disable')
await sendCdpToTab(tabId, 'Runtime.disable', undefined)
await new Promise((r) => setTimeout(r, 30))
} catch {}
return await chrome.debugger.sendCommand(dbg, 'Runtime.enable', params)
return await sendCdpToTab(tabId, 'Runtime.enable', params)
}
return await chrome.debugger.sendCommand(dbg, method, params)
return await sendCdpToTab(tabId, method, params)
}
// ---- attach / detach ------------------------------------------------------
@@ -261,10 +372,20 @@ async function attachTab(tabId) {
const targetInfo = info?.targetInfo
const targetId = String(targetInfo?.targetId || '')
if (!targetId) throw new Error('attachTab: no targetId')
const sessionId = `cb-tab-${nextSession++}`
// Derive the session id from the STABLE Chrome tabId, not a monotonic counter
// (issue #17). A tab's chrome.debugger session can be torn down and
// re-established — cross-process navigation, a service-worker restart wiping
// these in-memory maps, DevTools stealing the debugger — and each time the tab
// re-attaches. With a counter, re-attach minted a BRAND-NEW `cb-tab-N`, which
// orphaned the daemon's binding (it's still pinned to the old id and the relay
// never tells it to rebind) → permanent "stale sessionId / tab is gone". The
// tabId is stable across all of that, so `cb-tab-<tabId>` restores the SAME
// session the daemon already holds → eval/snapshot auto-follow the new page.
const sessionId = `cb-tab-${tabId}`
const entry = { sessionId, targetId }
tabs.set(tabId, entry)
sessionToTab.set(sessionId, tabId)
rememberSessionTarget(sessionId, targetId)
setBadge(tabId, port ? 'on' : 'connecting')
postToHost({
method: 'forwardCDPEvent',
@@ -347,9 +468,33 @@ chrome.debugger.onEvent.addListener((source, method, params) =>
}),
)
chrome.debugger.onDetach.addListener((source) =>
void whenReady(() => {
if (source.tabId) detachTab(source.tabId, true)
chrome.debugger.onDetach.addListener((source, reason) =>
void whenReady(async () => {
const tabId = source.tabId
if (!tabId) return
detachTab(tabId, true)
// A cross-process navigation (e.g. an SSO redirect like
// login.account.rakuten.com that swaps the render process / spawns OOPIFs)
// detaches the debugger, but the TAB survives. Without re-attaching, the
// session goes permanently stale and even open/navigate fails — exactly the
// #19 follow-up. So proactively re-attach (the stable `cb-tab-<tabId>`
// session id then restores the daemon's binding). Don't fight a detach the
// user or DevTools initiated.
if (reason === 'canceled_by_user' || reason === 'replaced_with_devtools') return
if (!port) return
// The swapped-in process needs a moment to settle; retry with backoff.
for (let i = 0; i < 6; i++) {
await new Promise((r) => setTimeout(r, 250 + i * 200))
if (tabs.has(tabId)) return // already re-attached (e.g. via onUpdated)
const tab = await chrome.tabs.get(tabId).catch(() => null)
if (!tab || !eligible(tab)) return // tab gone or now a restricted page
try {
await attachTab(tabId)
return
} catch (e) {
console.warn(`ab-connect: reattach attempt ${i + 1} for tab ${tabId} failed:`, e)
}
}
}),
)
+1 -1
View File
@@ -1,7 +1,7 @@
{
"manifest_version": 3,
"name": "chrome-use",
"version": "0.5.0",
"version": "0.4.9",
"description": "Let chrome-use drive your logged-in Chrome \u2014 install once, no token, no per-use confirmation.",
"key": "MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA6vQIyscGIPYPZdSpPwPL0+0gxUROyRgCpmvCSDoc8XUm4qm97VbKnD9Ijc1lV22lNWZtE78gaRjt6BeSfuMgnBymnhLKjN1gU6AI5QUU0mrJyeHdWKvrKQR5FmsM2A7Xr1ykE2SiiS8zNUS3Y/6O5l+Nva7wrVy6E4a2dkBVQkOsu+DV+nEZvhIyuDY5D5SPXqNwUTWTaglwj5mjvHz36xSwCWlPmrtJ+ED0AUyrb2z4GIOmvk4kqtBVrh/UD058klLo4CkYOnIybB5aV6WYuwarfPY4bF/dLggPem+ewLNTUNBuwrxj/A4nUv0LJTuRO8rR7f8WR9qnRCY0Ic5saQIDAQAB",
"icons": {
+15 -7
View File
@@ -28,7 +28,7 @@
<body>
<header>
<h1>Chrome Web Store 提交指南</h1>
<div class="sub">chrome-use · 上传包 <code>extensions/ab-connect.zip</code> · id 锁定为 <code>ciiljdlhdpfckdcfkphgmfalanpdejep</code></div>
<div class="sub">chrome-use · <strong>更新现有商店条目</strong> <code>knfcmbamhjmaonkfnjhldjedeobeafmk</code> · 上传 <strong>key 已删</strong> 的包(纯改名,保住老用户/评分)</div>
</header>
<p>为什么必须走商店:实测 Chrome 149 在<strong>非企业托管</strong>的 Mac 上,会把"非 Web Store"的 force-install 扩展直接标成 <code>[BLOCKED]</code>。商店扩展不受此限。这也是 codex / claude 扩展都发商店的原因。</p>
@@ -44,11 +44,15 @@
<li>(隐私政策需要一个公开 URL,见第四节 —— 我可以帮你开 GitHub Pages 托管 <code>privacy.html</code>)</li>
</ol>
<h2>二、上传</h2>
<h2>二、上传(更新现有条目,纯改名)</h2>
<p>你已经有一个上架条目(原名 <em>agent-browser-stealth</em>,Item ID <code>knfcmbamhjmaonkfnjhldjedeobeafmk</code>)。这次只是把它<strong>改名成 chrome-use</strong>,所以走 <span class="field">更新版本</span>,<u>不要</u> New item —— 这样老用户自动更新、评分/安装量都保留。</p>
<ol>
<li>devconsole → <span class="field">New item</span> → 上传 <code>extensions/ab-connect.zip</code></li>
<li>上传后确认分配到的 Item ID = <code>ciiljdlhdpfckdcfkphgmfalanpdejep</code>(因为 manifest 里保留了 <code>key</code>,id 会被锁成这个,native messaging 的 allowed_origins 才对得上)。<strong>若 id 不是这个,告诉我,我重签。</strong></li>
<li>devconsole → 打开 <strong>现有的 agent-browser-stealth 条目</strong>(id <code>knfcmbamhjmaonkfnjhldjedeobeafmk</code>)→ <span class="field">Package → Upload new package</span></li>
<li>上传 <strong>key 已删</strong> 的包 <code>chrome-use-store-vX.Y.Z.zip</code>(<em>必须删掉 manifest <code>key</code> 字段</em>,否则商店报"key 字段不符";仓库里 <code>ab-connect/manifest.json</code> 带 key 是给本地 Load-unpacked 用的,别直接传那个)。上传后 Item ID <strong>保持 <code>knfcmbam…</code> 不变</strong>;用户看到的扩展名变成 <strong>chrome-use</strong></li>
<li>native messaging 的 <code>allowed_origins</code> 同时放行 <code>knfcmbam…</code><code>ciiljdl…</code> 两个 id,所以改名后 relay 照常连得上,<strong>不会断现有用户</strong></li>
<li><strong>不要</strong>在这次发布里改 <code>background.js</code> 的 native host 名(保持 <code>com.agent_browser.connect</code>);<code>com.leeguoo.chrome_use</code> 是给将来真迁移用的。</li>
</ol>
<div class="warn"><strong>若你确实想另开一个全新的 "chrome-use" 条目(新 id、评分清零、用户需重装)</strong>:那才用保留 key 的包,id 会锁成 <code>ciiljdlhdpfckdcfkphgmfalanpdejep</code>。仅在你想彻底脱离旧 <em>stealth</em> 品牌时才这么做 —— 默认按上面"更新现有条目"走。</div>
<h2>三、商店信息(直接复制以下文案)</h2>
@@ -114,8 +118,12 @@ automate pages the user is working with, entirely on the user's machine and at t
<pre>https://leeguooooo.github.io/chrome-use/extensions/store/privacy.html</pre>
<p>(部署需 1–2 分钟生效。raw 备用直链:<code>https://raw.githubusercontent.com/leeguooooo/chrome-use/main/extensions/store/privacy.html</code>。)</p>
<h2>六、截图 / Screenshots(至少 1 张,1280×800 或 640×400</h2>
<p>可以截一张 CLI + Chrome 并排的演示图。<em>需要的话我用 cua-driver 截一张合规尺寸的图给你</em></p>
<h2>六、图标 + 截图 / Icon &amp; Screenshots</h2>
<p><strong>已生成,涂鸦风(和 cookie-use README 同一套)</strong>上传到对应字段即可:</p>
<ul>
<li><span class="field">Store icon(128×128)</span>:<code>chrome-use-store-icon-128.png</code></li>
<li><span class="field">Screenshots(每张正好 1280×800)</span>:<code>chrome-use-store-shot1-1280x800.png</code>(CMD 牵线操控已登录浏览器)、<code>shot2</code>(机械臂抓浏览器方向盘)、<code>shot3</code>(浏览器插线连终端 CONNECTED)。</li>
</ul>
<h2>七、提交后</h2>
<ol>
@@ -127,6 +135,6 @@ automate pages the user is working with, entirely on the user's machine and at t
<strong>今天的临时可用方案:</strong> 在你这台 Mac 上 <code>chrome://extensions</code> → 打开开发者模式 → Load unpacked → 选 <code>extensions/ab-connect</code>,30 秒手动装一次,native messaging + <code>extension connect</code> 立即可用。等商店过审再切静默路径。
</div>
<footer>chrome-use · 提交包与文案随扩展版本更新;改扩展后重跑 <code>scripts/pack-extension.sh</code> 并重打 <code>ab-connect.zip</code></footer>
<footer>chrome-use · 更新现有条目 <code>knfcmbam…</code>(纯改名);上传包必须删 key。改扩展后重打 key-stripped 的 <code>chrome-use-store-vX.Y.Z.zip</code> 再传</footer>
</body>
</html>
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "chrome-use",
"version": "1.1.0",
"version": "1.5.20",
"description": "chrome-use — drive your real, logged-in Chrome from any AI agent, stealth by default",
"type": "module",
"packageManager": "pnpm@11.1.3",
+225 -4
View File
@@ -36,6 +36,29 @@ Refs (`@e1`, `@e2`, ...) are assigned fresh on every snapshot. They become
submits, dynamic re-renders, dialog opens. Always re-snapshot before your
next ref interaction.
> **Hard rule: snapshot-first, never screenshot-to-locate.** For form fields and
> buttons, ALWAYS `snapshot -i` and act on refs/selectors. Do **not** reach for
> `screenshot` + coordinate clicks to find or hit an element — `snapshot -i` now
> pierces **cross-origin iframes** (embedded Google Payments / Stripe / checkout /
> KYC forms) and lists their elements by `@ref`, including input values. Use
> coordinates only for canvas/WebGL, or when `snapshot` genuinely returns nothing
> for your target. Screenshots are for *visual verification you report*, never the
> agent's own input — and a full-page `screenshot` of a real retina browser is
> often too large for an image reader anyway. (If you ever feel you *need* a
> screenshot to read state or locate something, that's a bug — please file it.)
> **Snapshot-first, always. Never default to `screenshot` + coordinate clicking
> for form fields or buttons.** Run `snapshot -i` and act on `@refs`. Use
> coordinates only for canvas/WebGL, or when `snapshot` genuinely returns nothing
> for your target. This holds **even inside cross-origin embedded iframes**
> since v1.5.12 `snapshot -i` pierces out-of-process iframes (Google Payments,
> Stripe, embedded checkout/KYC) and lists their elements with refs, so
> `click @e` / `type @e` / `fill @e` work directly. A screenshot is for a genuine
> *visual* check you report to the user — not your own input. (Full-page
> screenshots of a real retina Chrome are often too large for the image reader
> anyway.) Driving off pixels on the relay also risks a coordinate event drifting
> onto the user's foreground tab — refs never do. See issue #37.
## Before you automate: pick the cheapest tool
Driving a browser is the heavy option. chrome-use earns its keep when you
@@ -45,6 +68,7 @@ need a **real, logged-in browser** — not for reading text off a public page.
|---|---|
| Discover what exists / find sources | `WebSearch` |
| Specific facts from a static or public page | `WebFetch` or `curl` (no browser) |
| **Structured data from a known site** (GitHub issues, Reddit/HN search, Bilibili/Twitter feed, …) — esp. behind login | `chrome-use site <name>/<cmd>` (see below) — skip snapshot+click entirely |
| Login state, interaction, JS-rendered or anti-bot pages | **chrome-use** (this skill) |
| A page the user saved before / an internal system | `chrome-use find-url <keywords>` (their bookmarks), then open it |
| The user's **own already-open, logged-in** Chrome window | the **extension connect** flow (below) |
@@ -108,7 +132,17 @@ Each `--session` that connects gets its **own colored Chrome tab group** (named
after the session) and drives only its own tabs — multiple agents share the one
real browser without cross-talk, and the user's own tabs are never grouped. CDP
drives the page without moving the user's mouse/keyboard, so it doesn't fight
them for control. **Anti-detection ranking: this real logged-in Chrome (extension
them for control.
**Strict multi-agent isolation.** A session over the relay tracks and drives
**only the tabs it created** (its own group). It does **not** adopt the user's
existing tabs, other agents' tabs, or pop-ups (e.g. an OAuth/login window — that's
the user's), so several agents (and other tools opening tabs) can work in the same
real Chrome concurrently without ever dropping or stealing each other's tabs —
another agent's tab churn can't make your bound tab vanish or drift your commands
onto the wrong page. Consequence: `tab list` shows only *your* session's tabs; to
drive a specific page, navigate to it in your own tab instead of expecting a
pre-existing or popped-up tab to appear in the list. **Anti-detection ranking: this real logged-in Chrome (extension
connect) > a headed launched browser > headless (forbidden).** A genuine human
browser has no headless/automation tells at all, so prefer it for anything
anti-bot-sensitive.
@@ -127,6 +161,19 @@ cadence, and scroll/drag ease. Default `off`; a per-navigation detector
auto-escalates pages guarded by Akamai/PerimeterX/DataDome to `human`. Leave it
on auto; force `human` only when you already know the target scores behaviour.
**Cloudflare clearance — solve once, reuse.** Passing a Cloudflare challenge
mints a `cf_clearance` cookie (HttpOnly — invisible to `eval`/`document.cookie`;
read it via `chrome-use cookies`). It's bound to your **IP + User-Agent**: reuse
the same exit IP and UA and you skip the challenge until it expires. Driving the
user's real Chrome (relay) persists it natively; for isolated sessions,
`--session-name <name>` save/restores it. Before spending effort solving, run
`chrome-use cf-status` (aliases `cf`, `clearance`): it reports whether the page
is *currently* a Cloudflare challenge and whether a still-valid `cf_clearance`
exists, with a recommendation — `proceed` (already cleared, don't re-solve),
`solve` (challenge up, no clearance), or `reissue` (clearance present but page
still blocks → IP/UA drifted, re-solve). Use it as a preflight to avoid
re-solving what you already cleared.
## Two ways to drive a page — and when to drop to `eval`
You have a **real Chrome with the user's DOM**. Two layers, mix them freely:
@@ -150,6 +197,34 @@ chrome-use eval "[...document.forms[0].elements].filter(e=>!e.validity.valid).ma
chrome-use eval "document.querySelector('#stubborn').click()" # direct DOM click, bypasses overlays
```
## Site adapters — the cheapest path for "read structured data from site X"
Before you `open` + `snapshot` + click your way through GitHub/Reddit/Bilibili/etc.,
check whether a **site adapter** already exists. An adapter is a community-written JS
function that hits the site's own JSON API *from inside your logged-in tab* and returns
clean structured data — no clicking, no scraping, no screenshots. It's the same idea as
`eval`, packaged per-site.
```bash
chrome-use site update # one-time: fetch the adapter pack (~145 cmds)
chrome-use site list # what's installed (github/issues, reddit/search, …)
chrome-use site info github/issues # an adapter's args + which domain it runs on
chrome-use site github/issues owner/repo --json # run it → JSON (navigates there for you)
```
- Positional args fill the adapter's declared args **in order**; `--key value` overrides by name.
- It navigates to the adapter's domain (reusing the current tab if you're already on it), so
login-gated feeds (`bilibili/feed`, `twitter/...`) work because they run as *you*.
- If no adapter fits, fall back to the normal `snapshot`/`eval` loop. Adapters come from the
[bb-sites](https://github.com/epiral/bb-sites) community pack; chrome-use fetches & runs them.
> **Auto-trigger — act on it.** chrome-use keeps the pack synced automatically (first use +
> weekly), and when you `open`/`navigate`/`snapshot` a page whose domain has adapters it tells
> you: a `💡 site adapters for <domain>` line on stderr, and a `siteAdapters: {domain, commands}`
> field in `--json`. **When you see that, prefer the listed `site <name>/<cmd>` over snapshot+click
> for reading data** — it's the cheaper, more reliable path and it's already installed. You don't
> need to run `site update` yourself; just use the command it names.
## Quickstart
```bash
@@ -207,7 +282,11 @@ assigned fresh on every snapshot.
For unstructured reading (no refs needed):
```bash
chrome-use get text @e1 # visible text of an element
chrome-use get text # WHOLE PAGE — all frames by default (see below)
chrome-use get text @e1 # visible text of one element (or a CSS selector)
chrome-use get text --main # main content only — skip nav/header/sidebar
chrome-use get text --pierce # read through CLOSED shadow DOM (injected panels)
chrome-use frames # list every frame + where the text lives
chrome-use get html @e1 # innerHTML
chrome-use get attr @e1 href # any attribute
chrome-use get value @e1 # input value
@@ -216,6 +295,27 @@ chrome-use get url # current URL
chrome-use get count ".item" # count matching elements
```
**Whole-page text is cross-frame by default.** `chrome-use get text` with no
selector aggregates visible text across **every** frame — top document plus
same-process child frames plus cross-origin iframes — so you never silently miss
content that lives in an iframe (Yahoo Auctions / Rakuten / Mercari shop
descriptions, embedded checkout/spec frames). Each child frame is delimited with
a `----- frame [kind] url -----` marker. You do **not** need to remember a flag —
the default already reads all frames. (`--all-frames` is still accepted as an
explicit alias.)
So: when text looks missing or wrong, you don't have to guess — just
`chrome-use get text` reads everything. To **see** the structure (which frame
holds what), run `chrome-use frames`. To **cut boilerplate** (global nav/header/
footer, "related items" sidebars), use `chrome-use get text --main`. If content
is lazy-loaded, `scroll` it into view first, then read.
**Closed shadow DOM.** Some injected UI (browser-extension debug panels, web
components) renders into a *closed* shadow root that `eval`/`innerText` cannot
read. `chrome-use get text --pierce` reads through closed shadow roots and child
documents via the CDP DOM tree — use it when content is clearly on screen (you
see it in a screenshot) but `get text`/`eval` come back empty.
## Interacting
```bash
@@ -226,8 +326,15 @@ chrome-use hover @e1 # hover
chrome-use focus @e1 # focus (useful before keyboard input)
chrome-use fill @e2 "hello" # clear then type
chrome-use type @e2 " world" # type without clearing
chrome-use press Enter # press a key at current focus
chrome-use type @e5 "201-0001" --key-events # real keystrokes (not insertText) —
# use for autocomplete/combobox fields that
# only react to key events (e.g. a postal box
# that auto-fills city/prefecture, Google Places)
chrome-use press Enter # press a key at current focus (down+up)
chrome-use press Control+a # key combination
chrome-use keydown d # HOLD a key down (no auto-release)
chrome-use keyup d # release it — pair them to hold-to-move
# in a game: `keydown d; sleep; keyup d`
chrome-use check @e3 # check checkbox
chrome-use uncheck @e3 # uncheck
chrome-use select @e4 "option-value" # native <select> only
@@ -240,12 +347,32 @@ chrome-use pick @e4 --option "Europe" # ANY combobox (react-select / ARIA /
# (no silent no-op). Use this for custom
# dropdowns where `select` returns ✓ but
# changes nothing.
chrome-use upload @e5 file1.pdf # upload file(s)
chrome-use upload @e5 file1.pdf # upload file(s) — works over the extension relay too:
# chrome.debugger forbids setFileInputFiles, so the
# file's bytes are streamed into the page and rebuilt as
# a File there (chunked under native-messaging's 1 MiB cap).
# Works on file <input>s and drop/paste composers (e.g. X).
chrome-use scroll down 500 # scroll page (up/down/left/right)
chrome-use scroll down 700 --at 640,400 # wheel at a pixel — scrolls a cross-origin
# iframe (Payments/Stripe/checkout/KYC) that
# plain page scroll can't reach
chrome-use scroll down 700 --frame 2 # scroll frame 2 from `chrome-use frames`
chrome-use scrollintoview @e1 # scroll element into view
chrome-use drag @e1 @e2 # drag and drop
```
**Cross-origin iframes (embedded payment / checkout / KYC widgets — Google
Payments, Stripe, etc.) — drive them by ref, never by screenshot.** `snapshot -i`
pierces these out-of-process iframes and lists their elements by `@ref`
(including input values); `get text --all-frames` reads their text. Then just act
on the refs: `click @e`, `type @e`, `hover @e`, `dblclick @e`, `drag @a @b` all
work into the iframe. Over the extension relay these are dispatched through the
DOM (in the element's own frame), so they hit the right element in the right tab
— a coordinate click/scroll there can drift onto whatever tab is in the
foreground, so prefer refs. For below-the-fold content in such a frame, scroll it
with `scroll down N --at x,y` (a pixel over the frame) or `--frame n`. For a
postal/autocomplete box inside the frame, `type @e "…" --key-events`.
### When refs don't work or you don't want to snapshot
Use semantic locators:
@@ -292,6 +419,54 @@ chrome-use click --coords 449,320 # same, explicit flag
A bare-number argument is always a coordinate, never a selector.
### Canvas / WebGL apps (games, map & 3D viewers, drawing tools)
These paint everything to a `<canvas>` and expose **almost no accessibility
tree**, so `snapshot` comes back near-empty and refs are a dead end. `snapshot`
detects this and prints a one-line hint. Drive them the screenshot way:
```bash
chrome-use screenshot /tmp/s.png # SEE the state (your only read path —
# eval/get text return nothing useful)
chrome-use click 640 360 # interact by viewport coordinate
chrome-use press d --hold 800 # hold-to-move, precise (timed in-daemon —
# NOT keydown+shell-sleep+keyup, which
# adds ~250ms jitter per round-trip)
chrome-use press Space # discrete actions (jump/attack/confirm)
```
**Don't drive frame-by-frame with one CLI call per action** — that's the slowest,
lowest-fidelity way (each call is a process spawn + round-trip). Script a *timed
sequence in a single round-trip* with `batch` (it sends each step to the running
daemon; `press --hold` and `wait` block in-daemon, so timing is precise):
```bash
chrome-use batch "press d --hold 900" "press j" "press j" "wait 200" "press d --hold 500"
```
Also try reading real state instead of pixels: `eval` runs in the page's main
world, so for a framework/engine game you can often reach its globals (e.g. a
Phaser/PIXI/Three instance, a store, `window.__GAME__`) and read positions/score
directly — far better than guessing from a screenshot.
**For genuinely real-time driving, drop the CLI entirely and use the WebSocket.**
`chrome-use stream enable` opens a bidirectional WS (`stream status` prints the
`ws://127.0.0.1:<port>`). Connect once and you get a live ~60fps screencast AND
can send input on the same socket — no per-action process spawn, no round-trip,
works over the extension relay:
```js
// node (global WebSocket): live frames + locally-timed input
const ws = new WebSocket("ws://127.0.0.1:PORT")
ws.onmessage = e => { const m = JSON.parse(e.data); if (m.type==="frame") {/* base64 jpeg */} }
const k = (eventType,key,code,vk) => ws.send(JSON.stringify({type:"input_keyboard",eventType,key,code,windowsVirtualKeyCode:vk}))
k("keyDown"," ","Space",32); setTimeout(()=>k("keyUp"," ","Space",32), 80) // a jump
// also: {type:"input_mouse",eventType:"mousePressed",x,y,button:"left",clickCount:1}
```
This is the difference between watching a slideshow and playing the game. Reserve
screenshots for one-off checks; use the WS for any sustained real-time control.
## Waiting (read this)
Agents fail more often from bad waits than from bad selectors. Pick the
@@ -497,6 +672,43 @@ the same browser's existing targets, so a second session's first `open` can
navigate a sibling's tab. For concurrent agents on one real Chrome, use the
extension (each with a distinct `--session`), not raw `--cdp`.
Each session owns its own tab group and assigns its own `t<N>` indices (the same
physical tab is `t8` in one session, `t1` in another), so `t<N>` is **not** a
stable cross-session handle. To reach a *specific* tab from another session — e.g.
a tab that was filled in a session whose handle later died — use the **stable CDP
`targetId`**:
```bash
chrome-use tab list --full --session B # re-syncs live tabs; prints `target: <id>` per row
chrome-use tab <targetId> --session B # adopt that exact tab, NO reload (state preserved)
```
`tab list` re-discovers the live tab set on every call, so a fresh session sees
tabs other sessions opened (and re-attached ones), not just its own. Adopting by
`targetId` lands session B on the stranded tab without reloading it, so a
half-filled form survives. Still, the simplest recovery for a session whose own
tab died is to recover *that* session (reload / re-`open` / `daemon restart`).
To avoid piling up duplicate tabs when you re-`open` the same entry URL on
rebind, pass **`--reuse-tab`**: if a tab already shows that URL (matched by
origin+path), it switches to it instead of spawning a new one.
### Reset stuck daemon state
Each session runs a background daemon worker that holds the page handles. If a
session starts misbehaving — commands hit the wrong tab, refs/handles look stale,
or you upgraded `chrome-use` mid-session and old workers linger — restart the
daemons instead of hunting PIDs with `pgrep`/`kill`:
```bash
chrome-use daemon status # list running session daemons (+ relay state)
chrome-use daemon restart # kill every session daemon worker
```
`daemon restart` leaves the extension's native-messaging bridge (`__nm-host`)
alone, so the relay to your live Chrome stays up — the next command just spins up
a fresh, clean daemon against the same browser. It does **not** close any tabs.
### Mock network requests
```bash
@@ -606,6 +818,13 @@ forbids debugging). The session no longer has a live tab — re-run
replaces the old silent behaviour where the command ran on some *other*
tab and returned wrong data.
To recover, you need the tab's **exact** URL (query params and all — a long
SSO/redirect link breaks if truncated). `tab list` shortens long URLs with
`…`; use **`tab list --full`** to print them untruncated, then re-`open` the
right one. For multi-redirect SSO flows, re-open the **stable entry URL**
(not the mid-redirect one) and `wait` a few seconds for the SPA to settle
before snapshotting.
**Reads landing on the wrong page**
`eval`, `screenshot`, and `network requests` print the page they ran
against to stderr: `eval @ <url>`, `screenshot @ <url>`, `network @ <url>`.
@@ -668,6 +887,8 @@ and [references/authentication.md](references/authentication.md).
`chrome-use skills get electron`
- **Slack workspace automation**: `chrome-use skills get slack`
- **Exploratory testing / QA / bug hunts**: `chrome-use skills get dogfood`
- **Re-runnable test suites (frontend "unit tests")**: `chrome-use skills get test`
— turn repeated checks into a `chrome-use test <suite.yaml>` regression suite
- **Vercel Sandbox microVMs**: `chrome-use skills get vercel-sandbox`
- **AWS Bedrock AgentCore cloud browser**: `chrome-use skills get agentcore`
+90
View File
@@ -0,0 +1,90 @@
---
name: test
description: Write and run re-runnable, unit-test-style browser test suites with `chrome-use test <suite.yaml>`. Use when repetitive manual browser checks (does the page load logged in? is this element there? did the flow work?) should become a fixed, repeatable regression suite instead of being re-done by hand each time — frontend automated testing on top of chrome-use.
---
# chrome-use test — browser test suites
Turn the repetitive "open it, click around, check it's right" work into a
**re-runnable suite**, like unit tests for the frontend. Every time you find a
regression, add a case — the suite gets more valuable the more you use it.
```
chrome-use test <suite.yaml> [--launch | --session <name>] [--json]
```
- Exit code **0** if all cases pass, **1** if any fail → drop it straight into CI.
- Default: launches a fresh isolated browser (deterministic, repeatable) in a
`cu-test` session and closes it after. Pass `--session <name>` to run against an
already-connected session (e.g. the live Chrome via `chrome-use extension connect`).
- Failed cases auto-save a screenshot to `cu-test-artifacts/<case>.png`.
## Suite format (YAML)
```yaml
suite: chatgpt smoke # label (optional)
setup: # runs once before all cases (optional)
- account: chatgpt/huayue # inject a cookie-use stored login (optional)
- open: https://chatgpt.com/ # …or any normal step
cases:
- name: home loads logged in
steps: # steps reuse chrome-use's own commands
- open: https://chatgpt.com/
- wait: { load: networkidle }
assert: # all asserts must hold or the case fails
- url: { contains: chatgpt.com }
- visible: "#prompt-textarea"
- name: composer takes text
steps:
- fill: { sel: "#prompt-textarea", text: "hi" }
assert:
- text: { sel: "#prompt-textarea", contains: hi }
- eval: "!!window.__NEXT_DATA__"
```
## Steps (the verbs)
Each step is a one-key mapping; the key is a chrome-use command:
| Step | Meaning |
|---|---|
| `open: <url>` | navigate |
| `click: <selector\|@ref>` | click |
| `fill: { sel: <s>, text: <t> }` | clear + type |
| `type: { sel: <s>, text: <t> }` | type (no clear) |
| `press: <key>` | key press (e.g. `Enter`) |
| `wait: <ms>` / `wait: { load: networkidle }` / `wait: <selector>` | wait |
| `scroll: <up\|down\|...>` or `{ dir: down, px: 500 }` | scroll |
| `eval: "<js>"` | run JS |
## Assertions (the checks) — all compile to one truthy `eval`
| Assert | Passes when |
|---|---|
| `url: { contains\|equals\|matches: <v> }` | the page URL matches |
| `visible: <selector>` | element exists and is laid out |
| `hidden: <selector>` | element is absent / not laid out |
| `text: { sel: <s>, contains\|equals\|matches: <v> }` | element text matches |
| `count: { sel: <s>, eq: <n> }` | exactly N elements match |
| `eval: "<js>"` | the JS expression is truthy |
## Auth
`setup: - account: <id>` injects a [cookie-use](https://github.com/leeguooooo/cookie-use)
stored login into the test session, so the suite runs authenticated. (Needs
`cookie-use` installed; skip the line if you don't use it.)
## Workflow
1. Do the check once by hand with `open`/`snapshot`/`eval` to learn the selectors.
2. Write it up as a case in a `*.yaml` suite.
3. `chrome-use test suite.yaml` — green means it works; red shows the failing
assert + a screenshot.
4. Found a regression later? Add a case. Run the whole suite in CI.
## Limits (v1)
Assertions are evaluated independently after the steps run. No per-case retries,
no parallel cases, no snapshot/screenshot baseline diffing yet (use an `eval`
assert against known content for now). Steps run sequentially; a failing step
fails the case immediately.