Compare commits

...
57 Commits
Author SHA1 Message Date
leeguooooo d8f484eded fix(tabs): gate the about:blank cleanup to the relay only
CI / Version Sync Check (push) Has been cancelled
CI / Rust (push) Has been cancelled
CI / Rust (macos-latest - aarch64-apple-darwin) (push) Has been cancelled
CI / Rust (macos-latest - x86_64-apple-darwin) (push) Has been cancelled
CI / Rust (windows-latest - x86_64-pc-windows-msvc) (push) Has been cancelled
Release binaries / Build macOS ARM64 (push) Has been cancelled
Release binaries / Build macOS x64 (push) Has been cancelled
Release binaries / Build Linux ARM64 (push) Has been cancelled
Release binaries / Build Linux musl ARM64 (push) Has been cancelled
Release binaries / Build Linux musl x64 (push) Has been cancelled
Release binaries / Build Linux x64 (push) Has been cancelled
Release binaries / Build Windows x64 (push) Has been cancelled
CI / Native E2E Tests (push) Has been cancelled
CI / Windows Integration Test (push) Has been cancelled
CI / Global Install (macos-latest) (push) Has been cancelled
CI / Global Install (ubuntu-latest) (push) Has been cancelled
CI / Global Install (windows-latest) (push) Has been cancelled
Release binaries / Attach binaries to GitHub Release (push) Has been cancelled
The previous commit closed the leftover about:blank on ANY connection — but on a
launched browser the initial about:blank is the browser's own first tab, not
daemon scratch, so it must stay. Broke e2e_tab_ids_not_reused (launched). Gate
the cleanup on agent_group().is_some() (relay only), where the about:blank is a
tab WE created. e2e_tab_ids_not_reused passes; relay scratch-blank close intact.
2026-06-19 14:32:49 +09:00
leeguooooo 1eb40eabd5 fix(tabs): close the leftover initial about:blank when a real tab opens
A fresh session's daemon creates an about:blank scratch tab on connect; a
subsequent `tab new <url>` then opened the work tab beside it, so every session's
tab group showed a stray 'about:blank' next to the real page (e.g. about:blank +
ChatGPT). tab_new now closes any OWNED, still-blank tab once a real (non-blank)
tab exists, and re-pins the new tab. Verified live: `tab new <url>` on a fresh
session leaves only the work tab. 870 tests pass.
2026-06-19 14:06:42 +09:00
leeguooooo 601404ba72 feat(session): session stop <name> + session prune + lifecycle docs (#48)
Release binaries / Build macOS ARM64 (push) Has been cancelled
Release binaries / Build macOS x64 (push) Has been cancelled
Release binaries / Build Linux ARM64 (push) Has been cancelled
Release binaries / Build Linux musl ARM64 (push) Has been cancelled
Release binaries / Build Linux musl x64 (push) Has been cancelled
Release binaries / Build Linux x64 (push) Has been cancelled
Release binaries / Build Windows x64 (push) Has been cancelled
Release binaries / Attach binaries to GitHub Release (push) Has been cancelled
Explicit daemon reclamation to go with the v1.5.25 idle auto-shutdown:
- `session stop [name]` — stop one session daemon (default: current), graceful
  (SIGTERM → the daemon's shutdown runs close(), tidying the tabs it created).
- `session prune` — stop ALL session daemons now (clears the pile of idle
  daemons left after an automation/debug round; they respawn clean on next use).
  The __nm-host relay isn't a tracked session daemon, so the live-Chrome
  connection survives.
- --help Sessions section now documents the daemon lifecycle: spawn → 10-min idle
  auto-shutdown (AGENT_BROWSER_IDLE_TIMEOUT_MS / 0 to disable) → keep / stop / prune.

Closes #48. Verified live: session stop reclaimed a test daemon. 870 tests pass.
2026-06-18 14:44:35 +09:00
leeguooooo fd10766762 chore(ext): pack ab-connect 0.4.12 zip + crx (ABExt.ungroupTab for keep) 2026-06-18 12:16:14 +09:00
leeguooooo ba9b167ede feat(cleanup): default idle-shutdown + keep — stop leaving scratch tabs/groups behind
Release binaries / Build macOS ARM64 (push) Has been cancelled
Release binaries / Build macOS x64 (push) Has been cancelled
Release binaries / Build Linux ARM64 (push) Has been cancelled
Release binaries / Build Linux musl ARM64 (push) Has been cancelled
Release binaries / Build Linux musl x64 (push) Has been cancelled
Release binaries / Build Linux x64 (push) Has been cancelled
Release binaries / Build Windows x64 (push) Has been cancelled
Release binaries / Attach binaries to GitHub Release (push) Has been cancelled
Agents finish a task and just stop (never calling `close`), so daemons used to
run forever, leaving their per-session scratch tabs + tab group in the user's
Chrome. Two cases now handled:

- Default idle timeout (10 min; AGENT_BROWSER_IDLE_TIMEOUT_MS overrides, 0
  disables). On idle the daemon close()s the tabs IT created → the empty tab
  group is auto-removed by Chrome. Timer resets on every command, so active
  sessions are untouched. Adopted user tabs are never owned, so never closed.
- `keep` — leave the ACTIVE tab for the user: unown it (exempt from
  close/idle) + ask the extension to ungroup it (ABExt.ungroupTab → 0.4.12) so
  it becomes a normal tab. Scratch gets cleaned, deliverable tabs stay.

Also fix two clippy violations in the concurrently-landed #47 viewport code
(manual char comparison + iter().any→contains) that were failing main's CI.

ext 0.4.12: handle ABExt.ungroupTab (chrome.tabs.ungroup). 870 tests pass.
2026-06-18 12:11:59 +09:00
leeguooooo c077593e99 docs(viewport): document viewport/resize command for responsive debugging (#47) 2026-06-18 11:52:09 +09:00
leeguooooo 02e23ebe11 fix(build): include browser.rs clear_viewport/via_relay (#47) + cargo fmt
Release binaries / Build macOS ARM64 (push) Has been cancelled
Release binaries / Build macOS x64 (push) Has been cancelled
Release binaries / Build Linux ARM64 (push) Has been cancelled
Release binaries / Build Linux musl ARM64 (push) Has been cancelled
Release binaries / Build Linux musl x64 (push) Has been cancelled
Release binaries / Build Linux x64 (push) Has been cancelled
Release binaries / Build Windows x64 (push) Has been cancelled
Release binaries / Attach binaries to GitHub Release (push) Has been cancelled
v1.5.24 (d5cd9cd) shipped a commands.rs caller of `clear_viewport` but not the
browser.rs method it lives in (a concurrent in-progress #47 viewport/resize edit
was only partly staged), so main didn't compile and the format check failed.
Commit the matching browser.rs method + via_relay() helper and run cargo fmt.
Full tree builds; 863 tests pass.
2026-06-18 11:39:11 +09:00
leeguooooo d5cd9cd621 feat(canvas): extract WebGL/canvas-app content + fix site arg-order + adopt skill doc
canvas — chrome-use can now read canvas/WebGL apps (Figma, games, maps, charts,
drawing tools) that expose no DOM/refs:
  - canvas list: enumerate <canvas> (backing+CSS size, visibility, toDataUrl/tainted)
  - canvas capture [selector] [path]: save rendered pixels to PNG — toDataURL
    (full backing-store resolution), with a CDP screenshot fallback for WebGL
    without preserveDrawingBuffer or cross-origin-tainted canvases. --screenshot
    forces the screenshot path. Gets the RENDER, not hidden source data.
  Verified live: captured Figma's canvas at full 2522x1904 via toDataURL.

site — fix map_args losing the adapter's declared arg order: serde sorts @meta
keys alphabetically, so a 2-arg adapter like {projectId, path} mapped positionals
to {path, projectId} (swapped). Now parses declaration order from the raw @meta
text (Adapter.arg_order) + regression test. Affects any multi-arg adapter.

skill — core skill now documents `adopt <url|targetId>` (read a pre-existing tab,
the explicit way through strict isolation) and `canvas list`/`canvas capture` in
the canvas/WebGL section.

863 tests pass.
2026-06-18 11:31:24 +09:00
leeguooooo 284a60a54c feat(adopt): read a pre-existing tab without opening a new one
Release binaries / Build macOS ARM64 (push) Has been cancelled
Release binaries / Build macOS x64 (push) Has been cancelled
Release binaries / Build Linux ARM64 (push) Has been cancelled
Release binaries / Build Linux musl ARM64 (push) Has been cancelled
Release binaries / Build Linux musl x64 (push) Has been cancelled
Release binaries / Build Linux x64 (push) Has been cancelled
Release binaries / Build Windows x64 (push) Has been cancelled
Release binaries / Attach binaries to GitHub Release (push) Has been cancelled
New `chrome-use adopt <url-substring|targetId>`: drive a tab the user (or
another session) already has open, with ZERO new tabs. After group-scoped
isolation (#40) a session can't see foreign tabs, so adopt adds an explicit,
opt-in path:

- Relay (relay.rs): `ABRelay.getAllTargets` returns every attached target
  UNSCOPED (ignores group scoping), so the agent can find a specific tab by URL
  or targetId. +1 unit test.
- Daemon (browser.rs): `collect_all_targets` (unscoped, falls back to scoped on
  older relays) + `adopt_existing_target` — matches by exact targetId or
  case-insensitive URL substring, attaches it (the relay re-tags it into the
  adopter's group, so isolation holds), pins it; never creates a tab. On no
  match it errors AND lists the open tabs it can see, rather than launching.
  discover_and_attach_targets honors AGENT_BROWSER_ADOPT at first connect, so no
  about:blank is ever created.
- CLI (main.rs): `adopt` sets the env, forces a fresh daemon, and rewrites into
  `connect <relay-url>` (like `extension connect`) so the daemon attaches to the
  user's real Chrome before parse_command.

Extension (ab-connect 0.4.11): `reannounceAttachedTabs` now re-sends each tab's
url/title (it previously sent neither) so the relay's target list stays matchable
by URL after the MV3 service worker reconnects — otherwise reannounced tabs show
a blank url and `adopt <url>` can't find them. Repacked upload zip + crx.

Mechanism verified live (enumerated all 11 of the user's open tabs incl. the
target). 862 tests pass.
2026-06-17 21:18:01 +09:00
leeguooooo 10d196b6eb chore(ext): pack ab-connect 0.4.10 upload zip + crx (#40 group-scoped relay)
Rebuilt extensions/ab-connect.zip (key stripped for the Web Store) and the
reference .crx from the 0.4.10 source (openerTargetId + abGroup in the
synthesized Target.attachedToTarget).
2026-06-17 18:17:20 +09:00
leeguooooo 5be01e292d feat(relay): group-scoped Target.getTargets — restore follow-popup + cross-session adopt under isolation (#40)
Release binaries / Build macOS ARM64 (push) Has been cancelled
Release binaries / Build macOS x64 (push) Has been cancelled
Release binaries / Build Linux ARM64 (push) Has been cancelled
Release binaries / Build Linux musl ARM64 (push) Has been cancelled
Release binaries / Build Linux musl x64 (push) Has been cancelled
Release binaries / Build Linux x64 (push) Has been cancelled
Release binaries / Build Windows x64 (push) Has been cancelled
Release binaries / Attach binaries to GitHub Release (push) Has been cancelled
Move multi-agent isolation from blunt daemon-side filtering to relay-side
group scoping, so a session can adopt new tabs again (follow-popup, OAuth
results, cross-session adopt-by-targetId) without ever seeing the user's or
another agent's tabs.

Relay (relay.rs): track client->group (announced via new local ABRelay.setGroup,
or the first createTarget's agentGroup) and target->group (created tabs tagged
from the createTarget reply; an explicit attachToTarget tags the target into the
adopter's group = #21; a pop-up inherits its opener's group via openerTargetId).
Target.getTargets returns ONLY the requesting client's group; a client that never
announced a group (older daemon) gets the full list — fully backward-compatible.
+5 unit tests.

Daemon (browser.rs): announce_group() on connect sets relay_scoped. Adoption in
discover/resync/adopt_newly_opened is re-enabled ONLY when relay_scoped; without
it (launch / real CDP / older relay that didn't answer the announce) the daemon
keeps strict daemon-side isolation. So this can't regress the 125/125 isolation.

Extension (ab-connect 0.4.10): synthesized Target.attachedToTarget targetInfo now
carries openerTargetId (pop-ups inherit opener's group) and abGroup (the tab-group
title, so the relay re-attributes existing tabs after ITS own restart, since
createTarget tagging won't re-run). tabScopeHints().

Back-compat verified live: new daemon + OLD relay -> announce fails ->
relay_scoped=false -> strict fallback, open/eval/url all work. The new extension
(publish to CWS, strip manifest key) activates follow-popup; relay+daemon ship now.
861 tests pass.
2026-06-17 18:14:13 +09:00
leeguooooo a83d1b1df9 feat: rich-editor fill, box centers, screenshot downscale, disabled+docs (#41-#45)
Release binaries / Build macOS ARM64 (push) Has been cancelled
Release binaries / Build macOS x64 (push) Has been cancelled
Release binaries / Build Linux ARM64 (push) Has been cancelled
Release binaries / Build Linux musl ARM64 (push) Has been cancelled
Release binaries / Build Linux musl x64 (push) Has been cancelled
Release binaries / Build Linux x64 (push) Has been cancelled
Release binaries / Build Windows x64 (push) Has been cancelled
Release binaries / Attach binaries to GitHub Release (push) Has been cancelled
Dogfooding backlog from this session's embedded-form/editor work.

#41 fill on rich editors: detect CodeMirror 5 / Monaco / ProseMirror /
contenteditable and set via their own API or execCommand('insertText') so
beforeinput/input fire (a raw .value/textContent write no-op'd juejin's
CodeMirror and skipped React composers). Response echoes the `engine` used.
`fill <sel> --file <path>` / `--stdin` set large multiline text without
shell-escaping. `get value` now reads CodeMirror/Monaco/contenteditable too.

#42 screenshot --max-width/--max-height/--scale, plus a default 2000px
longest-edge cap (AGENT_BROWSER_SCREENSHOT_MAX_EDGE; 0 disables) so retina
full-page shots fit an agent's image reader and --scale 0.5 makes screenshot
px line up with click px. Annotated shots are never downscaled.

#43 `box @ref` (already a top-level alias of `get box`) now also returns
centerX/centerY/inViewport in CSS px — feed straight into `click x y` when a
ref-click no-ops (e.g. a button in a cross-origin iframe).

#44 no code change needed — disabled elements already list as
`button "Save" [disabled, ref=eN]`; the reporter's missing button was
DOM-gated on validity. Added a skill note: `find text` can't reach into a
cross-origin iframe — target those by snapshot @ref.

#45 core skill now distinguishes screenshot-to-locate (discouraged) from
screenshot-to-capture a reusable image asset via `screenshot [--clip] <file>`
(encouraged), so agents stop over-reading the prohibition.

#40 (group-scoped relay) stays deferred — needs an ab-connect extension change.

Verified live: fill --file round-trips multiline+CJK+backticks; contenteditable
engine=contenteditable + get value reads it back; box gives centerX/centerY/
inViewport; screenshot of retina example.com → 2000px; disabled button shows
[disabled]. 856 tests pass.
2026-06-17 17:49:38 +09:00
leeguooooo 50b27ac0e0 feat(site): auto-sync + auto-suggest adapters (auto-trigger)
Release binaries / Build macOS ARM64 (push) Has been cancelled
Release binaries / Build macOS x64 (push) Has been cancelled
Release binaries / Build Linux ARM64 (push) Has been cancelled
Release binaries / Build Linux musl ARM64 (push) Has been cancelled
Release binaries / Build Linux musl x64 (push) Has been cancelled
Release binaries / Build Linux x64 (push) Has been cancelled
Release binaries / Build Windows x64 (push) Has been cancelled
Release binaries / Attach binaries to GitHub Release (push) Has been cancelled
Make `site` trigger itself so an agent doesn't have to know adapters exist.

Auto-sync: the pack refreshes on first use and on a TTL (default 7d), both in
the `site` command path (blocking, fast) and as a non-blocking background task
on daemon startup — so ~/.chrome-use/sites/.index.json is always populated with
zero added latency. Tune via AGENT_BROWSER_SITES_TTL_DAYS; disable with
AGENT_BROWSER_SITES_NO_AUTO_UPDATE=1. `update` now writes .last_update + a
domain→adapters .index.json (read-only adapters ordered first).

Auto-suggest: `open`/`navigate`/`snapshot` onto a domain with adapters attaches
`siteAdapters: {domain, commands}` to the response; the CLI prints a
`💡 site adapters for <domain>` hint (stderr) and the field rides along in --json.
SKILL.md tells the agent to prefer the listed `site <name>/<cmd>` over scraping.
This keeps the 'never auto-disrupt user tabs' guarantee — it suggests, the agent
decides; nothing auto-runs on navigation.

site.rs: needs_refresh/adapters_for_domain/write_domain_index + timestamp/index
in update(). daemon.rs: background bootstrap. actions.rs: with_site_hint on
navigate + snapshot. output.rs: hint render. Verified live: open github.com →
hint leads with read-only github/issues; --json carries siteAdapters.
2026-06-17 17:15:34 +09:00
leeguooooo d81bc01645 feat(site): bb-sites adapters — turn any site into a structured-data CLI
Release binaries / Build macOS ARM64 (push) Has been cancelled
Release binaries / Build macOS x64 (push) Has been cancelled
Release binaries / Build Linux ARM64 (push) Has been cancelled
Release binaries / Build Linux musl ARM64 (push) Has been cancelled
Release binaries / Build Linux musl x64 (push) Has been cancelled
Release binaries / Build Linux x64 (push) Has been cancelled
Release binaries / Build Windows x64 (push) Has been cancelled
Release binaries / Attach binaries to GitHub Release (push) Has been cancelled
Add `chrome-use site` — run community bb-sites adapters over chrome-use's
stealth transport. An adapter is a per-command JS function that calls a
site's own JSON API from inside your logged-in tab (your cookies, same-origin
fetch, the site's modules) and returns clean JSON — no clicking/scraping.

- site update   fetch the upstream bb-sites pack into ~/.chrome-use/sites
- site list     list installed adapters (name/cmd)
- site info     show an adapter's @meta (args, domain, capabilities)
- site <name>/<cmd> [args]  navigate to its domain (reuse tab if already there) + eval, return JSON

chrome-use ships zero adapter code; `site update` fetches epiral/bb-sites at
runtime (like a package manager). Adapters remain their authors' property.

cli/src/site.rs (load/parse/build_eval/list/update/map_args + tests), wired
via commands.rs (parse), actions.rs (handle_site), main.rs (CLI dispatch).
Docs in README, README.zh, skill-data/core. Verified live: github/issues
returned 30 real issues as JSON over the relay.
2026-06-17 16:02:40 +09:00
leeguooooo c667e0e704 docs(zh): bring Chinese README to parity with English
Translate the sections the zh README was missing: 为什么用扩展 (extension vs raw
debug port + comparison table), 命令名 (chrome-use/abs same binary), 自动化测试
(chrome-use test YAML suite), 自己验证 (verify-yourself detector list), 调参 (env
var table); fill out 反检测 (0% stealth explanation + --launch CreepJS caveat),
the install 'other ways' details, and the blog/X footer. All 12 top-level sections
now match EN.
2026-06-17 14:49:52 +09:00
leeguooooo 08cb8dbeb9 docs: link blog + X in README 2026-06-17 14:15:52 +09:00
leeguooooo fd2cdcde77 chore(release): 1.5.18 — issue-reporting guidance + clear data:-over-relay error
Release binaries / Build macOS ARM64 (push) Has been cancelled
Release binaries / Build macOS x64 (push) Has been cancelled
Release binaries / Build Linux ARM64 (push) Has been cancelled
Release binaries / Build Linux musl ARM64 (push) Has been cancelled
Release binaries / Build Linux musl x64 (push) Has been cancelled
Release binaries / Build Linux x64 (push) Has been cancelled
Release binaries / Build Windows x64 (push) Has been cancelled
Release binaries / Attach binaries to GitHub Release (push) Has been cancelled
2026-06-17 14:10:07 +09:00
leeguooooo 8e001e3d88 feat(dx): guide users to file issues; clear error for data: URLs over the relay
#5: the CLI never pointed users at the issue tracker. `--help` and `--version`
now print the issues URL (https://github.com/leeguooooo/chrome-use/issues), so
agents hitting a rough edge know where to report it (the skill already nudges).

#1: navigating to a `data:` URL over the extension relay fails with a cryptic
`net::ERR_ABORTED` on an about:blank tab (chrome.debugger/chrome.tabs can't drive
a top-frame data: navigation; it works fine under --launch). Detect that case and
explain it — use a real http(s)://file:// URL or --launch — instead of leaking
the raw code. (Surfaced while stress-testing 5 concurrent agents.)
2026-06-17 14:10:04 +09:00
leeguooooo eb2bc343a0 chore(release): 1.5.17 — complete multi-agent isolation (no foreign tab/pop-up adoption on click)
Release binaries / Build macOS ARM64 (push) Has been cancelled
Release binaries / Build macOS x64 (push) Has been cancelled
Release binaries / Build Linux ARM64 (push) Has been cancelled
Release binaries / Build Linux musl ARM64 (push) Has been cancelled
Release binaries / Build Linux musl x64 (push) Has been cancelled
Release binaries / Build Linux x64 (push) Has been cancelled
Release binaries / Build Windows x64 (push) Has been cancelled
Release binaries / Attach binaries to GitHub Release (push) Has been cancelled
2026-06-17 13:31:50 +09:00
leeguooooo d86c9c4be2 fix(relay): don't adopt foreign tabs/pop-ups on click — completes multi-agent isolation (#last hole)
After strict isolation (v1.5.16) a session's tracked set is only its OWN tabs, so
in `adopt_newly_opened` (run after every click to follow a pop-up) EVERY foreign
tab looks "new" relative to the session's `before` set and got adopted — a
click-heavy flow on a busy shared Chrome pulled other agents'/the user's tabs
(github, Lark, iphone-use) into the session mid-flow.

A pop-up the agent itself opened can't be told apart from a foreign tab over the
relay (the synthesized targetInfo carries no opener/window/group), so on the relay
`adopt_newly_opened` now adopts nothing: the agent drives only tabs it explicitly
created; pop-ups (OAuth/login windows) are the user's. Launched browsers (every
tab ours) still follow pop-ups. Verified live: a fresh click-heavy relay session
stays clean (only its own tabs), and 5 concurrent agents churning tabs show zero
cross-agent drift.
2026-06-17 13:31:49 +09:00
leeguooooo 32c25a6627 chore(release): 1.5.16 — strict multi-agent tab isolation on the relay
Release binaries / Build macOS ARM64 (push) Has been cancelled
Release binaries / Build macOS x64 (push) Has been cancelled
Release binaries / Build Linux ARM64 (push) Has been cancelled
Release binaries / Build Linux musl ARM64 (push) Has been cancelled
Release binaries / Build Linux musl x64 (push) Has been cancelled
Release binaries / Build Linux x64 (push) Has been cancelled
Release binaries / Build Windows x64 (push) Has been cancelled
Release binaries / Attach binaries to GitHub Release (push) Has been cancelled
2026-06-17 11:50:10 +09:00
leeguooooo a8ce3dd3f8 fix(relay): strict multi-agent isolation — a session owns only its own tabs
Several agents (and other tools opening tabs) share one real Chrome via the
relay. Previously every session adopted ALL tabs from Target.getTargets, so
another agent's tab churn polluted the list, dropped the tab being driven, and
drifted commands onto the wrong page (the W-8BEN tax tab vanished mid-flow when a
concurrent iphone-use agent opened tabs).

A tab group belongs to exactly one agent. On the relay a session now tracks and
drives ONLY the tabs it created (its own colored group) plus pop-ups its own
clicks open — it never adopts the user's or other agents' tabs:

- discover_and_attach (relay): create the session's own tab and pin it; do not
  adopt any existing foreign tab.
- resync_targets (relay): never adopt unknown targets; never prune the session's
  tabs on a single getTargets snapshot (multi-agent churn / cross-process-nav
  gaps routinely omit live tabs) — prune only after RELAY_PRUNE_MISSES
  consecutive absent snapshots (debounced), pinned active always protected.
- adopt_newly_opened: a tab that appears right after this session's action is a
  pop-up we opened — record it as owned.

Launched browsers (every tab ours) keep adopting all tabs. Adds debounced_prune_ids
+ unit tests for the churn tolerance.
2026-06-17 11:50:09 +09:00
leeguooooo 997373fd57 chore(release): 1.5.15 — trusted activation for in-iframe buttons (#39)
Release binaries / Build macOS ARM64 (push) Has been cancelled
Release binaries / Build macOS x64 (push) Has been cancelled
Release binaries / Build Linux ARM64 (push) Has been cancelled
Release binaries / Build Linux musl ARM64 (push) Has been cancelled
Release binaries / Build Linux musl x64 (push) Has been cancelled
Release binaries / Build Linux x64 (push) Has been cancelled
Release binaries / Build Windows x64 (push) Has been cancelled
Release binaries / Attach binaries to GitHub Release (push) Has been cancelled
2026-06-17 10:16:17 +09:00
leeguooooo 5a858af93f fix(iframe): trusted activation for in-iframe buttons — keyboard, not synthetic click (#39)
A DOM `.click()` is isTrusted:false, which security-sensitive embedded forms
reject — Google Payments' enabled `保存` button silently no-op'd, so a
cross-origin payment/checkout/KYC form could be read, scrolled, and typed into
but never submitted. A coordinate click can't help either: getBoxModel for a
sub-frame node returns frame-local coords that don't compose the iframe offset,
so it lands wrong (verified — the same-origin probe came back isTrusted:false
via the coordinate fallback).

Fix: click on an in-iframe ref now focuses the element in its own frame session
and dispatches a real Enter (Space for checkbox-like roles) on the page session.
Chrome routes the key to the focused element across frames (same mechanism as
`type --focused`), and Enter/Space on a focused button/link/checkbox fires a
trusted click. Non-activatable roles fall back to DOM .click().

Adds e2e_iframe_button_click_is_trusted (+ fixture): an in-iframe button records
event.isTrusted into its own text; the test asserts the ref-click delivers
isTrusted:true.
2026-06-17 10:16:15 +09:00
leeguooooo 58dc02bfdc chore(release): 1.5.14 — fix eval await regression (replMode) + default scroll; green CI (#36, #38)
Release binaries / Build macOS ARM64 (push) Has been cancelled
Release binaries / Build macOS x64 (push) Has been cancelled
Release binaries / Build Linux ARM64 (push) Has been cancelled
Release binaries / Build Linux musl ARM64 (push) Has been cancelled
Release binaries / Build Linux musl x64 (push) Has been cancelled
Release binaries / Build Linux x64 (push) Has been cancelled
Release binaries / Build Windows x64 (push) Has been cancelled
Release binaries / Attach binaries to GitHub Release (push) Has been cancelled
2026-06-17 02:34:55 +09:00
leeguooooo c47601bd7b fix(eval): replMode only for sync let/const decls, keep awaitPromise for async (#38)
replMode and awaitPromise are mutually exclusive in Chrome — under replMode a
returned promise serialises to {} instead of being awaited, which broke every
fetch/async eval (e2e_domain_filter, e2e_headers, e2e_react_tree all regressed).
Enable replMode only for synchronous scripts that declare a top-level let/const
(the #38 case); promise-returning scripts keep awaitPromise — restoring the
pre-#38 await behaviour while still fixing the let-redeclaration collision.
2026-06-17 02:08:11 +09:00
leeguooooo 0296bc7a88 fix(scroll): keep default scroll on window.scrollBy; wheel only for --at/--frame (#36)
The centered-wheel default no-op'd on some pages (headless e2e_hover_scroll_press
regressed). Restore window.scrollBy for plain page scroll; the coordinate wheel
stays opt-in via --at/--frame for cross-origin iframe content.
2026-06-17 02:01:23 +09:00
leeguooooo 32e203b908 style: cargo fmt (fixes the CI format-check failure) 2026-06-17 01:33:22 +09:00
leeguooooo fc51cd63ba chore(release): 1.5.13 — eval replMode (re-declarable let/const) + snapshot-first skill rule (#37, #38)
Release binaries / Build macOS ARM64 (push) Has been cancelled
Release binaries / Build macOS x64 (push) Has been cancelled
Release binaries / Build Linux ARM64 (push) Has been cancelled
Release binaries / Build Linux musl ARM64 (push) Has been cancelled
Release binaries / Build Linux musl x64 (push) Has been cancelled
Release binaries / Build Linux x64 (push) Has been cancelled
Release binaries / Build Windows x64 (push) Has been cancelled
Release binaries / Attach binaries to GitHub Release (push) Has been cancelled
2026-06-17 01:15:31 +09:00
leeguooooo f714c7920b fix(eval): replMode so successive evals can re-declare let/const; snapshot-first skill rule (#37, #38)
#38: `chrome-use eval` now runs with Runtime.evaluate replMode (like the DevTools
console) — top-level `let`/`const` no longer throw "already been declared" across
successive evals (independent `eval` steps in a `test` suite collided in the
page's shared lexical scope), and top-level await is allowed. Main-world and
completion-value semantics are unchanged.

#37: core skill gains a hard rule — snapshot-first, never screenshot+coordinates
to locate form fields/buttons; `snapshot -i` now pierces cross-origin iframes and
lists their elements by @ref; screenshots are for visual checks only, and a
full-page retina screenshot often exceeds an image reader's limits.
2026-06-17 01:15:31 +09:00
leeguooooo 1ac8ef7732 chore(release): 1.5.12 — relay-safe hover/dblclick/drag, deeper iframe snapshot, key-events typing (#37)
Release binaries / Build macOS ARM64 (push) Has been cancelled
Release binaries / Build macOS x64 (push) Has been cancelled
Release binaries / Build Linux ARM64 (push) Has been cancelled
Release binaries / Build Linux musl ARM64 (push) Has been cancelled
Release binaries / Build Linux musl x64 (push) Has been cancelled
Release binaries / Build Linux x64 (push) Has been cancelled
Release binaries / Build Windows x64 (push) Has been cancelled
Release binaries / Attach binaries to GitHub Release (push) Has been cancelled
2026-06-17 00:58:23 +09:00
leeguooooo 9f24e66033 fix(relay): DOM-dispatch hover/dblclick/drag; deeper iframe snapshot; key-events typing (#37)
Follow-up to #36 — make the whole interaction surface reach cross-origin OOPIFs
and stop coordinate events drifting onto the user's foreground tab over the relay.

- hover/dblclick/drag now DOM-dispatch over the relay or into an iframe (like
  click already did): a coordinate Input event isn't confined to the target tab
  on a busy real Chrome and can't map an OOPIF element's box to a top-viewport
  point. drag does an HTML5 DnD in the element's frame; cross-frame drag errors
  loudly instead of drifting.
- snapshot recurses iframes to MAX_IFRAME_DEPTH (3) instead of one level, so refs
  inside nested payment/checkout widgets get a frame_id and resolve into the
  right frame.
- relay tab adoption merges several Target.getTargets snapshots — a single flaky
  relay snapshot was dropping live tabs (a driven tab vanished after restart).
- `type --key-events` (alias --keys) sends real per-character keyDown/keyUp
  instead of Input.insertText, so autocomplete/combobox widgets that ignore the
  insertText input event fire (Google address postal lookup; commits Angular
  reactive forms so Save enables).
- SKILL: hard "snapshot-first, never default to screenshot+coordinates" rule;
  snapshot -i pierces cross-origin iframes since v1.5.12; cross-origin iframe
  driving guidance (#37).
2026-06-17 00:58:12 +09:00
leeguooooo 70ab38d35f chore(release): 1.5.11 — cross-origin iframe scroll/click + open auto-reattach (#35, #36)
Release binaries / Build macOS ARM64 (push) Has been cancelled
Release binaries / Build macOS x64 (push) Has been cancelled
Release binaries / Build Linux ARM64 (push) Has been cancelled
Release binaries / Build Linux musl ARM64 (push) Has been cancelled
Release binaries / Build Linux musl x64 (push) Has been cancelled
Release binaries / Build Linux x64 (push) Has been cancelled
Release binaries / Build Windows x64 (push) Has been cancelled
Release binaries / Attach binaries to GitHub Release (push) Has been cancelled
2026-06-16 18:05:15 +09:00
leeguooooo 6830df50ea fix(relay): reach cross-origin iframes; auto-reattach open (#35, #36)
#35: `open` auto-reattaches when the bound relay tab is gone — drops the dead
page, opens a fresh tab in the session's group, and navigates it, instead of
only `tab new` recovering.

#36: scroll and click now reach content inside cross-origin OOPIFs:
- scroll dispatches a real wheel at a viewport point (default center, --at x,y,
  or --frame n) so it scrolls the iframe under the pointer, which
  window.scrollBy on the top document silently no-ops on.
- over the extension relay, clicks always use DOM-dispatch instead of
  coordinate Input events — a coordinate event isn't confined to the target tab
  on a busy real Chrome (it drifted onto the foreground tab) and an OOPIF
  element's box can't be mapped to a top-viewport point.
2026-06-16 18:05:06 +09:00
leeguooooo cd47ec43d0 chore(release): 1.5.10 — warn on debug-port launch while relay is up (#32)
Release binaries / Build macOS ARM64 (push) Has been cancelled
Release binaries / Build macOS x64 (push) Has been cancelled
Release binaries / Build Linux ARM64 (push) Has been cancelled
Release binaries / Build Linux musl ARM64 (push) Has been cancelled
Release binaries / Build Linux musl x64 (push) Has been cancelled
Release binaries / Build Linux x64 (push) Has been cancelled
Release binaries / Build Windows x64 (push) Has been cancelled
Release binaries / Attach binaries to GitHub Release (push) Has been cancelled
2026-06-16 14:32:29 +09:00
leeguooooo 2cd361817d fix(launch): warn when launching a debug-port Chrome while the relay is up (#32)
The connect-mode diagnostic (1.5.5) proved the 'Allow remote debugging?' modal
is NOT Chrome 149 UX (my earlier hypothesis) — it's chrome-use launching a fresh
debug-port Chrome on session=default while the ab-connect relay is up (32 logged
CONSENT-MODAL-RISK launches), almost always from a stray --launch/--no-auto-connect.
A launch now warns loudly when the relay is available, naming the modal and how
to avoid it (drop --launch/--new, don't pass --no-auto-connect), so the modal is
self-explained and the offending caller is fixable.
2026-06-16 14:32:28 +09:00
leeguooooo 42f47c49aa chore(release): 1.5.9 — strip zero-width title unicode (#33) + screenshot --clip/element (#34)
Release binaries / Build macOS ARM64 (push) Has been cancelled
Release binaries / Build macOS x64 (push) Has been cancelled
Release binaries / Build Linux ARM64 (push) Has been cancelled
Release binaries / Build Linux musl ARM64 (push) Has been cancelled
Release binaries / Build Linux musl x64 (push) Has been cancelled
Release binaries / Build Linux x64 (push) Has been cancelled
Release binaries / Build Windows x64 (push) Has been cancelled
Release binaries / Attach binaries to GitHub Release (push) Has been cancelled
2026-06-16 14:26:41 +09:00
leeguooooo e29800df72 fix(tab-list): strip zero-width unicode from titles (#33); feat(screenshot): --clip pixel region + documented element capture (#34)
#33: some sites prepend runs of ZWJ/word-joiner/invisible-times/BOM to
document.title (badging/anti-scrape); left in, they polluted 'tab list', broke
text matching, and wrecked column alignment. sanitize_title() now strips
zero-width/bidi-format chars at every title ingestion point + get_title().

#34: 'screenshot <selector>' (element capture) already worked but was
undocumented; added 'screenshot --clip x,y,w,h' for an explicit pixel region
(CDP captureScreenshot clip), documented both in --help. Verified live.
2026-06-16 14:26:40 +09:00
leeguooooo e7e849ea39 chore(release): 1.5.8 — file upload over the extension relay (#13)
Release binaries / Build macOS ARM64 (push) Has been cancelled
Release binaries / Build macOS x64 (push) Has been cancelled
Release binaries / Build Linux ARM64 (push) Has been cancelled
Release binaries / Build Linux musl ARM64 (push) Has been cancelled
Release binaries / Build Linux musl x64 (push) Has been cancelled
Release binaries / Build Linux x64 (push) Has been cancelled
Release binaries / Build Windows x64 (push) Has been cancelled
Release binaries / Attach binaries to GitHub Release (push) Has been cancelled
2026-06-16 14:05:33 +09:00
leeguooooo ebb02c65c8 fix(relay): file upload now works over the extension relay (#13)
chrome.debugger forbids DOM.setFileInputFiles, so 'upload' used to hard-fail on
the relay and push users to a --launch/direct-CDP session. Now it falls back to
reconstructing the File entirely in the page (Playwright/Cypress-style: build a
File from the bytes, assign input.files = dataTransfer.files, fire input/change;
for drop/paste composers like X, dispatch synthetic paste+drop with the
DataTransfer). The bytes are streamed in <1 MiB base64 chunks because the relay
tunnels CDP through native messaging (1 MiB/message cap) — a whole image as one
arg closed the channel. Verified live over the relay: an 809 KB PNG lands intact
on a file input with change firing. No more direct-CDP needed for uploads.
2026-06-16 14:05:31 +09:00
leeguooooo 4317db636f chore(release): 1.5.7 — cf-status Cloudflare clearance preflight
Release binaries / Build macOS ARM64 (push) Has been cancelled
Release binaries / Build macOS x64 (push) Has been cancelled
Release binaries / Build Linux ARM64 (push) Has been cancelled
Release binaries / Build Linux musl ARM64 (push) Has been cancelled
Release binaries / Build Linux musl x64 (push) Has been cancelled
Release binaries / Build Linux x64 (push) Has been cancelled
Release binaries / Build Windows x64 (push) Has been cancelled
Release binaries / Attach binaries to GitHub Release (push) Has been cancelled
2026-06-16 12:19:26 +09:00
leeguooooo c99838a034 feat(cloudflare): cf-status preflight — skip re-solving when cf_clearance is still valid
Passing a Cloudflare challenge mints an HttpOnly cf_clearance cookie bound to
IP+UA. 'chrome-use cf-status' (aliases cf/cloudflare-status/clearance) reports
whether the active page is currently a CF challenge and whether a still-valid
cf_clearance exists (read via CDP — HttpOnly is invisible to document.cookie),
plus CF_VERIFIED_DEVICE trust, and a recommendation: proceed (already cleared,
don't re-solve) / solve (challenge up, no clearance) / reissue (clearance present
but page still blocks → IP/UA drifted). Lets an agent avoid re-solving what it
already cleared — the persistence optimization. Pure helpers unit-tested; live
-verified on a real cf_clearance.
2026-06-16 12:19:24 +09:00
leeguooooo dc2aa4cade chore(release): 1.5.6 — pin adopted tab against transient relay snapshots (#31)
Release binaries / Build macOS ARM64 (push) Has been cancelled
Release binaries / Build macOS x64 (push) Has been cancelled
Release binaries / Build Linux ARM64 (push) Has been cancelled
Release binaries / Build Linux musl ARM64 (push) Has been cancelled
Release binaries / Build Linux musl x64 (push) Has been cancelled
Release binaries / Build Linux x64 (push) Has been cancelled
Release binaries / Build Windows x64 (push) Has been cancelled
Release binaries / Attach binaries to GitHub Release (push) Has been cancelled
2026-06-15 17:00:42 +09:00
leeguooooo 7085f3bf36 fix(relay): don't prune the pinned target on a transient getTargets snapshot (#31)
Driving a busy real Chrome via the relay, a single Target.getTargets call
occasionally returns a different window's tabs ('tab list hops windows'). resync
pruned every tracked page absent from that snapshot — including the agent's
explicitly-adopted (pinned) tab — after which active-target resolution fell back
to active_page_index and eval/click/snapshot drifted onto a foreign tab
(about:blank / chrome-extension:// / the user's page), breaking any 3+ step flow.

prunable_target_ids() now protects the pinned active target from snapshot-based
pruning; a genuine close still arrives as Target.targetDestroyed (event drain) and
removes it properly. Unit-tested.
2026-06-15 17:00:38 +09:00
leeguooooo 29815ff5f3 chore(release): 1.5.5 — connect-mode diagnostic log for the remote-debugging consent modal (#31)
Release binaries / Build macOS ARM64 (push) Has been cancelled
Release binaries / Build macOS x64 (push) Has been cancelled
Release binaries / Build Linux ARM64 (push) Has been cancelled
Release binaries / Build Linux musl ARM64 (push) Has been cancelled
Release binaries / Build Linux musl x64 (push) Has been cancelled
Release binaries / Build Linux x64 (push) Has been cancelled
Release binaries / Build Windows x64 (push) Has been cancelled
Release binaries / Attach binaries to GitHub Release (push) Has been cancelled
2026-06-15 16:47:14 +09:00
leeguooooo 2a338d4c29 diag(connect): log CDP transport mode to detect 'Allow remote debugging?' modal source (#31)
The consent modal only appears on a raw remote-debugging attach or a browser we
launched with a debug port — never on the ab-connect extension relay. Append one
line per connection to ~/.chrome-use/connect-mode.log (relay | raw-port-attach |
launched | remote-ws), flagging 'CONSENT-MODAL-RISK' when a raw-port/launch path
runs while the relay was available. Lets us tell a code regression from Chrome's
own extension-debugger consent UX when the modal reappears. Best-effort, never
fails a connection. Verified: normal 'open' logs mode=relay (consent-free).
2026-06-15 16:47:12 +09:00
leeguooooo 6fcf52db60 chore(release): 1.5.4 — get text --pierce (closed shadow DOM, #30)
Release binaries / Build macOS ARM64 (push) Has been cancelled
Release binaries / Build macOS x64 (push) Has been cancelled
Release binaries / Build Linux ARM64 (push) Has been cancelled
Release binaries / Build Linux musl ARM64 (push) Has been cancelled
Release binaries / Build Linux musl x64 (push) Has been cancelled
Release binaries / Build Linux x64 (push) Has been cancelled
Release binaries / Build Windows x64 (push) Has been cancelled
Release binaries / Attach binaries to GitHub Release (push) Has been cancelled
2026-06-15 15:46:31 +09:00
leeguooooo af8823b27b feat(text): 'get text --pierce' reads through CLOSED shadow DOM (#30)
Some injected UI (browser-extension debug panels, web components) renders into a
CLOSED shadow root that eval/innerText cannot read. --pierce walks the CDP DOM
tree (DOM.getDocument depth:-1 pierce:true), which includes closed shadow roots
and child documents, and collects text nodes (skipping script/style/etc).

Review-safe: rides the per-tab debugger session already attached, no new Chrome
permission and no ab-connect/extension change — so it works in extension-relay
mode without touching the published extension. Verified live: a closed-shadow
panel that main-world eval reports HIDDEN is read in full via --pierce.

First slice of #30 (read extension/injected-panel content). Deeper extension
introspection (background SW / chrome.storage) stays a launch-mode / raw-CDP
concern, deliberately NOT done by expanding ab-connect's debugger powers.
2026-06-15 15:46:23 +09:00
leeguooooo c85e3faa82 chore(release): 1.5.3 — get text defaults to cross-frame; #29 (sessions/did-you-mean/tab liveness); CI changelog fix
Release binaries / Build macOS ARM64 (push) Has been cancelled
Release binaries / Build macOS x64 (push) Has been cancelled
Release binaries / Build Linux ARM64 (push) Has been cancelled
Release binaries / Build Linux musl ARM64 (push) Has been cancelled
Release binaries / Build Linux musl x64 (push) Has been cancelled
Release binaries / Build Linux x64 (push) Has been cancelled
Release binaries / Build Windows x64 (push) Has been cancelled
Release binaries / Attach binaries to GitHub Release (push) Has been cancelled
2026-06-15 13:42:58 +09:00
leeguooooo b25958946c feat(text): 'get text' (no selector) defaults to cross-frame whole-page read
So an agent never silently misses iframed content (listing descriptions etc.)
without having to know the --all-frames flag. Single-frame pages are unchanged
(identical to the old body read); multi-frame pages now include child frames —
a strict superset. Skill + help updated to make the default and 'frames'/--main
discoverable.
2026-06-15 13:42:48 +09:00
leeguooooo d4ff49caa8 ci(release): don't let an empty changelog section abort the release (bash -e)
The changelog step runs under 'bash -e'. section() returned non-zero when a
commit category was empty (grep no-match / empty [ -n ] test), aborting the
script before the closing heredoc delimiter — so any release whose range lacked
a whole category (e.g. 1.5.2: only feat, no fix) failed to attach binaries.
Add '|| true' + 'return 0' so section() always succeeds.
2026-06-15 13:42:48 +09:00
leeguooooo 4e949fffbf chore(release): 1.5.2 — sessions command + did-you-mean + honest tab-switch liveness (#29) 2026-06-15 13:34:26 +09:00
leeguooooo af46490812 feat(cli): sessions command + 'did you mean' suggestions + honest tab-switch liveness (#29)
- chrome-use sessions: top-level alias for the daemon inventory (the skill
  advertises sessions, so it's a natural guess that used to error).
- Unknown commands now suggest the nearest valid one (Levenshtein + prefix
  match), staying silent when nothing is close (e.g. 'clik' -> click,
  'sesions' -> sessions, 'xyzzy' -> no suggestion).
- tab <id>: probe the switched session and show a warning indicator instead of
  a green check when it isn't responding yet, so a switch onto a re-attaching
  (churned-tabId) session no longer reports false success. The #24 targetId
  recovery self-heals within ~6s, hence a warning rather than a hard error.
2026-06-15 13:25:16 +09:00
leeguooooo 57c52d6517 chore(release): 1.5.1 — frame-aware text extraction (get text --all-frames/--main, frames; #27) + ab-connect 0.4.9 targetId recovery (#24)
Release binaries / Build macOS ARM64 (push) Has been cancelled
Release binaries / Build macOS x64 (push) Has been cancelled
Release binaries / Build Linux ARM64 (push) Has been cancelled
Release binaries / Build Linux musl ARM64 (push) Has been cancelled
Release binaries / Build Linux musl x64 (push) Has been cancelled
Release binaries / Build Linux x64 (push) Has been cancelled
Release binaries / Build Windows x64 (push) Has been cancelled
Release binaries / Attach binaries to GitHub Release (push) Has been cancelled
2026-06-15 13:15:47 +09:00
leeguooooo 2707ceb1c4 feat(text): frame-aware text extraction — get text --all-frames / --main + frames (#27)
On listing/marketplace pages (Yahoo Auctions, Rakuten, Mercari shops) the
seller's description lives in a child frame or under a related-items sidebar,
so 'get text body' returned only header/nav boilerplate.

- get text --all-frames: aggregate visible text across every reachable frame.
  Same-process child frames are read via Page.createIsolatedWorld; OOPIFs via
  their auto-attached debugger session (iframe_sessions). Each non-top frame is
  labelled with a '----- frame [kind] url -----' separator.
- get text --main: readability-lite — prefer the densest <main>/<article>
  region over the whole body, dropping global header/nav/footer chrome.
- frames: enumerate frames (kind + url + per-frame text length) so an agent can
  see where a page's text actually lives and pick the right read.

Verified live: inline srcdoc frame text aggregated through --all-frames; Yahoo
Auctions <main> (2881 chars) extracted via --main, stripping the Yahoo header.
2026-06-15 12:51:59 +09:00
leeguooooo f7a657ac46 ci(release): group changelog by type ( Features / 🐛 Fixes / 🔧 Other)
Release notes were a flat list of commit subjects — hard to tell at a glance what
was added vs fixed (recurring '看不出改了什么'). Group by conventional-commit type
so every future release auto-shows scannable Features/Fixes sections.
2026-06-15 12:17:38 +09:00
leeguooooo 4e295ce139 fix(ab-connect): recover a churned-tabId session by stable CDP targetId (0.4.9, #24)
Live-reproduced #24 on 0.4.8 driving the Mercari signin token-exchange hop
(login.jp.mercari.com): the cross-process nav gives the tab a NEW Chrome tabId
while the CDP targetId stays the same. So cb-tab-<oldTabId> can't be recovered —
recoverSessionTab parsed the old tabId, chrome.tabs.get(oldTabId) failed (gone),
and it gave up → permanent 'stale sessionId ... its tab is gone' until the page
settled ~6s later and something re-attached. current/tab <targetId>/daemon
restart all failed because the relay still mapped the targetId to the dead
session.

Fix: remember each session's targetId across detach (sessionTargets map). When
recoverSessionTab can't recover by the encoded tabId, fall back to the STABLE
targetId — chrome.debugger.getTargets() to find the tab now hosting that target,
attach it, and ALIAS the dead cb-tab-<oldTabId> session to the live tab so the
daemon's session id keeps resolving. Longer retry window (~6s) since this hop
takes seconds to settle. Builds on 0.4.6/0.4.8 reattach; covers the tabId-churn
case those missed.

Needs dogfood on the real Mercari flow (can't repro the tabId churn synthetically).
2026-06-15 12:08:32 +09:00
27 changed files with 5003 additions and 251 deletions
+19 -4
View File
@@ -147,16 +147,31 @@ jobs:
git fetch --tags --force --quiet origin 2>/dev/null || true
TAG="${{ github.event.inputs.tag || github.ref_name }}"
PREV="$(git describe --tags --abbrev=0 "${TAG}^" 2>/dev/null || true)"
RANGE="${TAG}"
[ -n "$PREV" ] && RANGE="${PREV}..${TAG}"
# Group commit subjects by conventional-commit type so the notes are
# scannable ("what's new / what's fixed") instead of a flat dev log.
LOG="$(git log "$RANGE" --no-merges --pretty='%s' | grep -v '^chore(release)' || true)"
# NOTE: the job runs under `bash -e`. grep returning 1 (no match) and
# the `[ -n "$body" ]` test returning 1 (empty section) must NOT abort
# the script — otherwise a release whose commit range lacks a whole
# category (e.g. only `feat`, no `fix`) dies before writing the closing
# heredoc delimiter and the whole release step fails. `|| true` +
# `return 0` keep section() always-succeeding.
section() { # $1=header $2=grep-pattern
local body; body="$(printf '%s\n' "$LOG" | grep -E "$2" | sed 's/^/- /' || true)"
[ -n "$body" ] && printf '\n### %s\n%s\n' "$1" "$body"
return 0
}
{
echo "notes<<__NOTES_EOF__"
echo "## What changed"
echo ""
section "✨ Features" '^feat'
section "🐛 Fixes" '^fix'
section "🔧 Other" '^(perf|refactor|docs|build|ci|test|style|revert)'
if [ -n "$PREV" ]; then
git log "${PREV}..${TAG}" --no-merges --pretty='- %s' | grep -v '^- chore(release)' || true
echo ""
echo "**Full changelog**: https://github.com/${{ github.repository }}/compare/${PREV}...${TAG}"
else
git log "${TAG}" --no-merges --pretty='- %s' | grep -v '^- chore(release)' || true
fi
echo "__NOTES_EOF__"
} >> "$GITHUB_OUTPUT"
+47
View File
@@ -205,6 +205,50 @@ chrome-use --launch --profile auto open https://x.com/home
In CI environments, standalone mode is used automatically.
## Site adapters — turn a website into a structured-data CLI
Most "read GitHub issues" / "search Reddit" / "get my Bilibili feed" tasks don't
need clicking and screenshotting at all — the site already has a JSON API behind
its own login. A **site adapter** is a tiny JS function that calls that API *from
inside your logged-in tab* (your cookies, same-origin `fetch`, the site's own
modules) and returns clean JSON. The site can't tell it apart from you, because it
*is* you.
chrome-use ships none of these adapters — `site update` fetches the community
[**bb-sites**](https://github.com/epiral/bb-sites) pack at runtime (like a package
manager pulling a dependency), then runs them over chrome-use's stealth transport:
```bash
chrome-use site update # fetch the adapter pack (~145 commands)
chrome-use site list # github/issues, reddit/search, bilibili/feed, …
chrome-use site info github/issues # see an adapter's args + domain
# Run one — navigates to the site (reusing the tab if you're already there) and returns JSON
chrome-use site github/issues epiral/bb-browser --json
chrome-use site reddit/search "rust async" --json
chrome-use site bilibili/feed --json # works because it's your logged-in session
```
Positional args fill the adapter's declared args in order; `--key value` overrides
by name. Adapters are authored by the bb-sites community and remain their authors'
property — chrome-use just runs them.
**Auto-sync + auto-suggest.** You rarely type `site update` yourself: chrome-use
syncs the pack on first use and refreshes it weekly in the background (tune with
`AGENT_BROWSER_SITES_TTL_DAYS`, disable with `AGENT_BROWSER_SITES_NO_AUTO_UPDATE=1`).
And when you `open`/`snapshot` a page whose domain has adapters, chrome-use surfaces
them right in the output — a `💡 site adapters for <domain>` line, plus a
`siteAdapters` field under `--json` — so an agent reaches for the structured-data
adapter instead of scraping the DOM:
```text
$ chrome-use open https://github.com
💡 site adapters for github.com — prefer these for structured data:
github/issues, github/me, github/repo, …
e.g. chrome-use site github/issues --json
✓ GitHub
```
## Automated testing (`chrome-use test`)
Turn the repetitive "open it, click around, check it's right" work into a
@@ -326,3 +370,6 @@ We deliberately **don't ship our own bot detector** — the strongest, most hone
## License
Apache-2.0
---
> Built by **leeguooooo** — field notes on AI agents, reverse engineering & Cloudflare Workers at **[blog.misonote.com](https://blog.misonote.com)** · follow on **[X @leeguooooo](https://x.com/leeguooooo)**
+138 -1
View File
@@ -63,6 +63,25 @@ chrome-use 让**任意** agentClaude Code、Cursor、Codex、你自己的脚
每个 `--session` 拿到**自己的彩色标签组**,多个 agent 共用同一个真实浏览器、互不干扰,也不动你自己的标签页。
## 为什么用扩展(而非裸调试端口)
其他本地工具走裸 `--remote-debugging-port`CDP)驱动 Chrome。从 **Chrome 136** 起,每次这样连接都会弹出一个阻塞式的 **"Allow remote debugging?"** 同意框 —— 而且端口得提前开好。我们的扩展改用原生消息:**装一次,之后零确认。**
| | **chrome-use**(本扩展) | web-access(裸 CDP 端口) | Claude in Chromechrome.debugger |
|---|---|---|---|
| 连接方式 | 原生消息 —— 无端口、无 token | `--remote-debugging-port` | `chrome.debugger` |
| **"Allow remote debugging?" 弹框** | **从不** ✅ | **每次连都弹** 🔴 | 无 |
| 复用你的真实登录 | 是 | 是 | 是 |
| `Runtime.enable`CDP)泄漏¹ | **默认关闭 → 干净** ✅ | 域已启用 | 不适用 |
| CreepJS 隐身分² | **0% stealth · 0% headless** ✅ | 真实 Chrome | 真实 Chrome |
| 每会话标签组 / 并发 agent | **支持** ✅ | 无 | 无 |
| 为 chrome-use CLI 打造 | 是 | 独立代理 | 单 app 助手 |
> ¹ 对 [rebrowser-bot-detector](https://bot-detector.rebrowser.net/) 实测:我们的中继报 `runtimeEnableLeak: 🟢 No leak`、`navigatorWebdriver: 🟢`。
> ² 对 [CreepJS](https://abrahamjuliot.github.io/creepjs/) 在「连接真实 Chrome」路径上实测 —— 见 [反检测](#反检测)。
>
> 同意框不是假想:裸端口工具**每次** attach 都会弹(Chrome 136+ 安全策略)。扩展路径从不弹。
## 安装
```bash
@@ -71,6 +90,15 @@ curl -fsSL https://raw.githubusercontent.com/leeguooooo/chrome-use/main/install.
从最新的 [GitHub Release](https://github.com/leeguooooo/chrome-use/releases) 下载对应平台的预编译二进制,安装 `chrome-use`(以及 `abs` 别名)。无需 npm,无需 token。
<details>
<summary>其他安装方式</summary>
- **锁定版本:** `AGENT_BROWSER_VERSION=v0.27.0-fork.12 curl -fsSL https://raw.githubusercontent.com/leeguooooo/chrome-use/main/install.sh | sh`
- **自定义路径:** `AGENT_BROWSER_BIN_DIR=$HOME/bin curl -fsSL … | sh`
- **Windows** 从 [Releases 页](https://github.com/leeguooooo/chrome-use/releases) 下载 `chrome-use-win32-x64.tar.gz`,把 `chrome-use.exe` 放进 PATH。
- **npm(旧渠道):** `npm install -g chrome-use` —— 仍在发布,但 GitHub Releases 现在是主渠道。
</details>
### 安装 AI agent skills
```bash
@@ -79,6 +107,10 @@ npx skills add leeguooooo/chrome-use
`skills/chrome-use` 拉进当前项目,让你的 AI agent 拿到正确的用法和预授权的 bash 权限。
## 命令名
`chrome-use``chrome-use``abs` 是**同一个二进制** —— `abs` 只是短别名。没有单独的「隐身可执行文件」;隐身是**运行时行为**(见下方 [反检测](#反检测)),根据你是连接真实 Chrome 还是 `--launch` 全新实例自动启用。
## 连接你的 Chrome
**推荐 —— 浏览器扩展(一键,无弹窗)。** 从 Chrome 应用商店安装 [**chrome-use** 扩展](https://chromewebstore.google.com/detail/chrome-use/knfcmbamhjmaonkfnjhldjedeobeafmk),再注册一次本地桥:
@@ -128,8 +160,83 @@ chrome-use --launch open https://example.com
# 保留登录:用你真实的 Chrome profile 启动
chrome-use --launch --profile auto open https://x.com/home
# 或显式指定:--profile Default / --profile "Profile 1"
```
## 站点适配器 —— 把一个网站变成「结构化数据 CLI」
大多数「读 GitHub issue」「搜 Reddit」「拉我的 B 站动态」这类任务,根本不需要点击 +
截图 —— 网站登录态背后本来就有 JSON 接口。**站点适配器**就是一小段 JS 函数,它在你
**已登录的标签页内**调用那个接口(用你的 cookie、同源 `fetch`、网站自己的模块),返回
干净的 JSON。网站分辨不出它和你的区别,因为它**就是你**。
chrome-use 本身不附带任何适配器 —— `site update` 会在运行时拉取社区的
[**bb-sites**](https://github.com/epiral/bb-sites) 适配器包(就像包管理器拉依赖),
然后在 chrome-use 的隐身通道上运行它们:
```bash
chrome-use site update # 拉取适配器包(约 145 条命令)
chrome-use site list # github/issues、reddit/search、bilibili/feed…
chrome-use site info github/issues # 查看某个适配器的参数 + 域名
# 运行一个 —— 会导航到对应站点(已在该站点则复用当前标签页)并返回 JSON
chrome-use site github/issues epiral/bb-browser --json
chrome-use site reddit/search "rust async" --json
chrome-use site bilibili/feed --json # 能用,因为走的是你的登录态
```
位置参数按适配器声明的参数顺序填入;`--key value` 按名覆盖。适配器由 bb-sites 社区编写、
版权归各自作者所有 —— chrome-use 只负责运行它们。
**自动同步 + 自动提示。** 你基本不用手动 `site update`:chrome-use 首次使用时自动拉取,
之后每周后台刷新一次(`AGENT_BROWSER_SITES_TTL_DAYS` 调周期,`AGENT_BROWSER_SITES_NO_AUTO_UPDATE=1`
关闭)。而当你 `open`/`snapshot` 一个有适配器的域名时,chrome-use 会在输出里直接把可用命令
亮出来 —— 一行 `💡 site adapters for <域名>`,`--json` 下则是 `siteAdapters` 字段 —— 这样
agent 会直接改用结构化适配器,而不是去扒 DOM:
```text
$ chrome-use open https://github.com
💡 site adapters for github.com — prefer these for structured data:
github/issues, github/me, github/repo, …
e.g. chrome-use site github/issues --json
✓ GitHub
```
## 自动化测试(`chrome-use test`
把反复的「打开它、点一圈、看对不对」变成**可重跑的测试套件** —— 前端的单元测试。用 YAML 写用例;步骤复用 chrome-use 自己的命令,断言编译成一次检查:
```yaml
# smoke.yaml
suite: chatgpt smoke
setup:
- account: chatgpt/huayue # 注入一个 cookie-use 登录(可选)
cases:
- name: home loads logged in
steps:
- open: https://chatgpt.com/
- wait: { load: networkidle }
assert:
- url: { contains: chatgpt.com }
- visible: "#prompt-textarea"
```
```bash
chrome-use test smoke.yaml # 启动隔离浏览器,跑用例
chrome-use test smoke.yaml --session default # …或对你已连接的 Chrome 跑
```
```
suite: chatgpt smoke (session cu-test)
✓ home loads logged in 1.2s
✗ composer takes text 0.8s
assert text "#prompt-textarea" contains "hi" → got ""
↳ cu-test-artifacts/composer-takes-text.png
2 cases · 1 passed · 1 failed
```
任一用例失败时退出码非零(可直接丢进 CI),失败用例会存截图。断言:`url` · `visible` · `hidden` · `text` · `count` · `eval`。步骤:`open` · `click` · `fill` · `type` · `press` · `wait` · `scroll` · `eval`。完整指南:`chrome-use skills get test`。发现回归?加个用例 —— 用得越多,套件越值钱。
## 反检测
连接你真实 Chrome 时,我们**零** JS 注入 —— 浏览器指纹完全是真的。指导原则是 **native CDP/Chrome 覆盖优先于 JS 谎言**:被重定义的 getter 本身可被检测,原生覆盖则不会。
@@ -146,7 +253,9 @@ chrome-use --launch --profile auto open https://x.com/home
| [rebrowser-bot-detector](https://bot-detector.rebrowser.net/) | `runtimeEnableLeak` 🟢 · `pwInitScripts` 🟢 |
| [bot.sannysoft.com](https://bot.sannysoft.com) | 全绿 |
`--launch` 独立模式下会改用一整套隐身补丁,同样过上述检测。
CreepJS 上的 `0% stealth` 是关键数字:因为连接路径**什么都不打补丁**,根本没有可供说谎检测器抓的 override。(读 `navigator.languages` 顺序或 IP 地理位置的面板可能给个软性的「navigator」/「location」标记 —— 那反映的是*你真实 Chrome* 的语言列表和网络,不是自动化破绽。)
`--launch` 独立模式(全新浏览器)会改用一整套隐身补丁,也能过上述检测 —— 唯一例外:CreepJS 报 **~20% stealth**,因为 srcdoc-iframe 的 `contentWindow` 补丁触发了它的 `hasIframeProxy` 探测(用来藏自动化的 proxy 本身成了破绽)。其余全干净(`0% headless`、sannysoft/browserscan 全绿、Cloudflare 通过)。设 **`AGENT_BROWSER_DISABLE_IFRAME_PROXY=1`** 去掉那个补丁即可拿到干净的 **0% stealth**(代价是放弃小众的 srcdoc-iframe 遮蔽)。**扩展连接路径**(你的真实 Chrome)零 JS 注入、不受影响 —— 它才是货真价实的 0% 路径。
### 类人输入(行为隐身)
@@ -167,6 +276,30 @@ chrome-use --launch --profile auto open https://x.com/home
操作你的真实 Chrome 不该打断你的工作。agent **全程在后台操作**:新标签后台打开(在自己的彩色会话标签组里),**从不强制把标签拽到前台**,并用 `Emulation.setFocusEmulationEnabled` 让每个 agent 标签照常渲染、`document.hasFocus()` / `visibilityState` 仍报 `visible`。于是截图正常、页面不被降频,"标签全程隐藏"也不会变成新的机器人信号。你在自己的标签里照常工作,agent 在旁边默默干活。(想置顶某个标签仍可显式调用命令。)
### 自己验证
别光听我们说 —— 把你连接的 Chrome 指向最硬的公开检测器,自己对比:
- **[CreepJS](https://abrahamjuliot.github.io/creepjs/)** —— 最全面的指纹 / 说谎检测器
- **[bot.incolumitas.com](https://bot.incolumitas.com/)** —— 行为 + 指纹打分,方法公开
- **[BrowserScan](https://www.browserscan.net/bot-detection)** —— Webdriver / User-Agent / CDP / Navigator
- **[bot.sannysoft.com](https://bot.sannysoft.com)** —— 经典自动化特征清单
- **[pixelscan.net](https://pixelscan.net/)** · **[iphey.com](https://iphey.com/)** —— 一致性与身份
我们故意**不自带 bot 检测器** —— 最强、最诚实的基准,就是拿市面上最好的检测器去测你的真实浏览器。
### 调参(环境变量)
| 变量 | 默认 | 作用 |
|---|---|---|
| `AGENT_BROWSER_CAPTURE_CONSOLE` | 关 | 启用 `Runtime` 域,让 `console` / `errors` 捕获页面输出。关闭可保持最隐身的画像。 |
| `AGENT_BROWSER_HUMANIZE` | 关 | 类人输入动作:`off`(瞬时)、`fast`(轻量缓动轨迹)、`human`(全套曲线轨迹 + 落点抖动 + 击键节奏 + 缓动滚动/拖拽)。也可用 `--humanize`。默认 `off`;自适应检测器会把 Akamai/PerimeterX/DataDome 守护的页面自动升到 `human`。 |
| `AGENT_BROWSER_TIMEZONE` | 未设 | 仅 `--launch`。IANA id(如 `Asia/Tokyo`)原生设置时区(Intl + Date 跟随,无 JS 谎言)以匹配代理;`auto` 按 locale 推导。 |
| `AGENT_BROWSER_BLOCK_WEBRTC` | auto | 仅 `--launch`。设了代理时自动强制 WebRTC 走代理(不泄漏真实 IP)。`1` 无代理时也隐藏本地 IP;`0` 退出。 |
| `AGENT_BROWSER_HIDE_CANVAS` | 关 | 仅 `--launch`。加入会话稳定的 canvas/audio 指纹噪声。默认关(噪声本身就是一种「谎言」)。 |
| `AGENT_BROWSER_ADAPTIVE_REF` | 开 | 当保存的 `@ref` 移动且 role/name 重查失败时,按指纹相似度重定位(需高分 + 明显领先,否则明确报错)。`0` 关闭。 |
| `AGENT_BROWSER_CLICK_MODE` | _(auto)_ | 点击策略。默认先滚动入视、派发坐标点击,若被浮层遮挡则回退 DOM `.click()``dom` 始终用 `.click()`(适合 blur 即关的自动补全/菜单项);`coord` 严格只用坐标(遮挡时硬失败)。 |
## chrome-use 的独特之处
- **默认 auto-connect** —— `chrome-use open` 连你现有的 Chrome 而非启新的
@@ -181,3 +314,7 @@ chrome-use --launch --profile auto open https://x.com/home
## License
Apache-2.0
---
> 由 **leeguooooo** 打造 —— AI agent、逆向工程与 Cloudflare Workers 的实战笔记见 **[blog.misonote.com](https://blog.misonote.com)** · 关注 **[X @leeguooooo](https://x.com/leeguooooo)**
+1 -1
View File
@@ -290,7 +290,7 @@ checksum = "613afe47fcd5fac7ccf1db93babcb082c5994d996f20b8b159f2ad1658eb5724"
[[package]]
name = "chrome-use"
version = "1.5.0"
version = "1.5.27"
dependencies = [
"aes",
"aes-gcm",
+1 -1
View File
@@ -1,6 +1,6 @@
[package]
name = "chrome-use"
version = "1.5.0"
version = "1.5.27"
edition = "2021"
description = "Fast browser automation CLI for AI agents"
license = "Apache-2.0"
+731 -62
View File
@@ -30,12 +30,111 @@ pub enum ParseError {
InvalidSessionName { name: String },
}
/// Top-level commands an agent is likely to mistype, used for "did you mean"
/// suggestions on an unknown command (issue #29). Not exhaustive — just the
/// common verbs plus a few known wrong-guesses mapped to the real command.
const KNOWN_COMMANDS: &[&str] = &[
"open",
"navigate",
"click",
"fill",
"type",
"press",
"snapshot",
"screenshot",
"eval",
"get",
"text",
"html",
"frames",
"find",
"wait",
"scroll",
"hover",
"select",
"check",
"uncheck",
"tab",
"tabs",
"close",
"back",
"forward",
"reload",
"sessions",
"status",
"daemon",
"doctor",
"upgrade",
"connect",
"cookies",
"mouse",
"keyboard",
"stream",
"frame",
"profiles",
"title",
"url",
"is",
"drag",
"dialog",
"upload",
"site",
"box",
"adopt",
"canvas",
"viewport",
"resize",
"keep",
];
/// Levenshtein distance, capped — small inputs only (command names).
fn edit_distance(a: &str, b: &str) -> usize {
let a: Vec<char> = a.chars().collect();
let b: Vec<char> = b.chars().collect();
let mut prev: Vec<usize> = (0..=b.len()).collect();
let mut curr = vec![0usize; b.len() + 1];
for (i, &ca) in a.iter().enumerate() {
curr[0] = i + 1;
for (j, &cb) in b.iter().enumerate() {
let cost = if ca == cb { 0 } else { 1 };
curr[j + 1] = (prev[j + 1] + 1).min(curr[j] + 1).min(prev[j] + cost);
}
std::mem::swap(&mut prev, &mut curr);
}
prev[b.len()]
}
/// Closest known command within a small edit distance, or a prefix/substring
/// match — `None` if nothing is close enough to suggest confidently.
fn nearest_command(input: &str) -> Option<String> {
let lower = input.to_lowercase();
// Exact prefix/substring hits first (e.g. "session" -> "sessions").
if let Some(c) = KNOWN_COMMANDS
.iter()
.find(|c| c.starts_with(&lower) || lower.starts_with(**c))
{
return Some(c.to_string());
}
// Tolerance scales with length: short words get distance 1, longer get 2.
let max_dist = if lower.len() <= 4 { 1 } else { 2 };
KNOWN_COMMANDS
.iter()
.map(|c| (*c, edit_distance(&lower, c)))
.filter(|(_, d)| *d <= max_dist)
.min_by_key(|(_, d)| *d)
.map(|(c, _)| c.to_string())
}
impl ParseError {
pub fn format(&self) -> String {
match self {
ParseError::UnknownCommand { command } => {
format!("Unknown command: {}", command)
}
ParseError::UnknownCommand { command } => match nearest_command(command) {
Some(suggestion) => format!(
"Unknown command: {}\nDid you mean: chrome-use {}?",
command, suggestion
),
None => format!("Unknown command: {}", command),
},
ParseError::UnknownSubcommand {
subcommand,
valid_options,
@@ -467,25 +566,64 @@ fn parse_command_inner(args: &[String], flags: &Flags) -> Result<Value, ParseErr
"fill" => {
let sel = rest.first().ok_or_else(|| ParseError::MissingArguments {
context: "fill".to_string(),
usage: "fill <selector> <text>",
usage: "fill <selector> <text> | fill <selector> --file <path> | fill <selector> --stdin",
})?;
Ok(json!({ "id": id, "action": "fill", "selector": sel, "value": rest[1..].join(" ") }))
// Large/multiline content without shell-escaping hell (issue #41):
// `fill <sel> --file <path>` reads the value from a UTF-8 file, and
// `fill <sel> --stdin` reads it from stdin — sent verbatim, so backticks,
// quotes, newlines and non-ASCII pass through untouched.
let value = match rest.get(1).copied() {
Some("--file") => {
let path = rest.get(2).ok_or(ParseError::InvalidValue {
message: "fill --file requires a path".to_string(),
usage: "fill <selector> --file <path>",
})?;
std::fs::read_to_string(path).map_err(|e| ParseError::InvalidValue {
message: format!("fill --file: cannot read {path}: {e}"),
usage: "fill <selector> --file <path>",
})?
}
Some("--stdin") => {
use std::io::Read;
let mut buf = String::new();
io::stdin()
.read_to_string(&mut buf)
.map_err(|e| ParseError::InvalidValue {
message: format!("fill --stdin: {e}"),
usage: "fill <selector> --stdin",
})?;
buf
}
_ => rest[1..].join(" "),
};
Ok(json!({ "id": id, "action": "fill", "selector": sel, "value": value }))
}
"type" => {
// `--key-events` (alias `--keys`): send real per-character keystrokes
// instead of Input.insertText, so autocomplete/combobox widgets that
// only react to key events fire (e.g. Google address postal lookup).
let key_events = rest.iter().any(|a| *a == "--key-events" || *a == "--keys");
let rest: Vec<&str> = rest
.iter()
.copied()
.filter(|a| *a != "--key-events" && *a != "--keys")
.collect();
// `type --focused <text>` types into whatever element currently has
// focus (no selector) — for custom widgets that move focus to a hidden
// input after you open them.
if rest.first() == Some(&"--focused") {
return Ok(json!({
"id": id, "action": "type", "focused": true,
"text": rest[1..].join(" "),
"text": rest[1..].join(" "), "keyEvents": key_events,
}));
}
let sel = rest.first().ok_or_else(|| ParseError::MissingArguments {
context: "type".to_string(),
usage: "type <selector> <text> (or: type --focused <text>)",
usage: "type <selector> <text> (or: type --focused <text>) [--key-events]",
})?;
Ok(json!({ "id": id, "action": "type", "selector": sel, "text": rest[1..].join(" ") }))
Ok(
json!({ "id": id, "action": "type", "selector": sel, "text": rest[1..].join(" "), "keyEvents": key_events }),
)
}
"pick" => {
// pick <selector|@ref> --option "<text>" — atomic combobox select:
@@ -676,10 +814,57 @@ fn parse_command_inner(args: &[String], flags: &Flags) -> Result<Value, ParseErr
} else {
return Err(ParseError::MissingArguments {
context: "scroll --selector".to_string(),
usage: "scroll [direction] [amount] [--selector <sel>]",
usage: "scroll [direction] [amount] [--selector <sel>] [--at <x,y>] [--frame <n>]",
});
}
}
"--at" => {
// `--at x,y`: dispatch the wheel at this viewport pixel, so it
// scrolls whatever element/iframe is under the pointer — including
// cross-origin iframes that `window.scrollBy` can't reach (#36).
let val = rest.get(i + 1).ok_or(ParseError::MissingArguments {
context: "scroll --at".to_string(),
usage: "scroll [direction] [amount] --at <x,y>",
})?;
let mut parts = val.split(',');
match (
parts.next().and_then(|s| s.trim().parse::<f64>().ok()),
parts.next().and_then(|s| s.trim().parse::<f64>().ok()),
) {
(Some(x), Some(y)) => {
obj.insert("at".to_string(), json!([x, y]));
}
_ => {
return Err(ParseError::InvalidValue {
message: format!("scroll --at: invalid coordinate `{}`", val),
usage:
"scroll [direction] [amount] --at <x,y> (e.g. --at 640,400)",
})
}
}
i += 1;
}
"--frame" => {
// `--frame n`: scroll the n-th frame from `chrome-use frames` by
// dispatching the wheel at that frame's center — reaches content in
// a cross-origin iframe without needing a selector into it (#36).
let val = rest.get(i + 1).ok_or(ParseError::MissingArguments {
context: "scroll --frame".to_string(),
usage: "scroll [direction] [amount] --frame <n>",
})?;
match val.trim().parse::<usize>() {
Ok(n) => {
obj.insert("frame".to_string(), json!(n));
}
Err(_) => {
return Err(ParseError::InvalidValue {
message: format!("scroll --frame: invalid index `{}`", val),
usage: "scroll [direction] [amount] --frame <n> (index from `chrome-use frames`)",
})
}
}
i += 1;
}
arg if arg.starts_with('-') => {}
_ => {
match positional_index {
@@ -854,17 +1039,85 @@ fn parse_command_inner(args: &[String], flags: &Flags) -> Result<Value, ParseErr
// selector: @ref or CSS selector
// path: file path (contains / or . or ends with known extension)
let mut full_page = false;
let positional: Vec<&str> = rest
.iter()
.filter(|arg| match **arg {
"--full" | "-f" => {
full_page = true;
false
let mut clip: Option<Value> = None;
let mut max_width: Option<u32> = None;
let mut max_height: Option<u32> = None;
let mut scale: Option<f64> = None;
let mut positional: Vec<&str> = Vec::new();
let mut i = 0;
// Parse a numeric value for a downscale flag (issue #42).
let parse_num = |i: &mut usize, flag: &str| -> Result<String, ParseError> {
let v = rest
.get(*i + 1)
.ok_or_else(|| ParseError::MissingArguments {
context: format!("screenshot {flag}"),
usage: "screenshot [--max-width <px>] [--max-height <px>] [--scale <0..1>]",
})?;
*i += 1;
Ok(v.to_string())
};
while i < rest.len() {
match rest[i] {
"--full" | "-f" => full_page = true,
// Downscale the saved image so retina/full-page shots fit an
// agent's image reader and screenshot px line up with click px (#42).
"--max-width" => {
let v = parse_num(&mut i, "--max-width")?;
max_width = Some(v.parse().map_err(|_| ParseError::InvalidValue {
message: format!("--max-width expects a number, got '{v}'"),
usage: "screenshot --max-width <px>",
})?);
}
_ => true,
})
.copied()
.collect();
"--max-height" => {
let v = parse_num(&mut i, "--max-height")?;
max_height = Some(v.parse().map_err(|_| ParseError::InvalidValue {
message: format!("--max-height expects a number, got '{v}'"),
usage: "screenshot --max-height <px>",
})?);
}
"--scale" => {
let v = parse_num(&mut i, "--scale")?;
let s: f64 = v.parse().map_err(|_| ParseError::InvalidValue {
message: format!("--scale expects a number like 0.5, got '{v}'"),
usage: "screenshot --scale <0..1>",
})?;
if s <= 0.0 || s > 1.0 {
return Err(ParseError::InvalidValue {
message: format!("--scale must be in (0, 1], got '{v}'"),
usage: "screenshot --scale <0..1>",
});
}
scale = Some(s);
}
// `--clip x,y,w,h` captures a pixel region (issue #34).
"--clip" => {
let raw = rest
.get(i + 1)
.ok_or_else(|| ParseError::MissingArguments {
context: "screenshot --clip".to_string(),
usage: "screenshot --clip <x,y,w,h> [path]",
})?;
let nums: Vec<f64> = raw
.split(',')
.filter_map(|n| n.trim().parse::<f64>().ok())
.collect();
if nums.len() != 4 {
return Err(ParseError::InvalidValue {
message: format!(
"--clip expects 'x,y,w,h' (4 numbers), got '{raw}'"
),
usage: "screenshot --clip <x,y,w,h> [path]",
});
}
clip = Some(json!({
"x": nums[0], "y": nums[1], "width": nums[2], "height": nums[3]
}));
i += 1;
}
other => positional.push(other),
}
i += 1;
}
let (selector, path) = match (positional.first(), positional.get(1)) {
(Some(first), Some(second)) => {
// Two args: first is selector, second is path
@@ -895,6 +1148,18 @@ fn parse_command_inner(args: &[String], flags: &Flags) -> Result<Value, ParseErr
"path": path, "selector": selector,
"fullPage": full_page, "annotate": flags.annotate
});
if let Some(c) = clip {
cmd["clip"] = c;
}
if let Some(w) = max_width {
cmd["maxWidth"] = json!(w);
}
if let Some(h) = max_height {
cmd["maxHeight"] = json!(h);
}
if let Some(s) = scale {
cmd["scale"] = json!(s);
}
if let Some(ref fmt) = flags.screenshot_format {
cmd["format"] = json!(fmt);
}
@@ -912,6 +1177,54 @@ fn parse_command_inner(args: &[String], flags: &Flags) -> Result<Value, ParseErr
}
Ok(cmd)
}
// Canvas/WebGL apps (Figma, games, maps, charts, drawing tools) render to a
// <canvas> with no DOM/refs to read. `canvas` extracts what's actually
// rendered: `list` enumerates canvases; `capture` saves a canvas to PNG via
// toDataURL (full backing-store resolution) with a CDP-screenshot fallback
// for WebGL contexts (no preserveDrawingBuffer) or cross-origin-tainted ones.
"canvas" => {
match rest.first().copied() {
Some("list") => Ok(json!({ "id": id, "action": "canvas_list" })),
Some("capture") => {
let force_screenshot = rest.contains(&"--screenshot");
let pos: Vec<&str> = rest[1..]
.iter()
.copied()
.filter(|a| !a.starts_with("--"))
.collect();
// `capture [selector] [path]`: a selector starts with . # @ or is
// a tag; a path contains / or ends in an image extension.
let is_path = |s: &str| {
s.contains('/')
|| s.ends_with(".png")
|| s.ends_with(".jpg")
|| s.ends_with(".jpeg")
|| s.ends_with(".webp")
};
let (selector, path) = match (pos.first(), pos.get(1)) {
(Some(a), Some(b)) => (Some(*a), Some(*b)),
(Some(a), None) if is_path(a) => (None, Some(*a)),
(Some(a), None) => (Some(*a), None),
_ => (None, None),
};
let mut cmd = json!({ "id": id, "action": "canvas_capture" });
if let Some(s) = selector {
cmd["selector"] = json!(s);
}
if let Some(p) = path {
cmd["path"] = json!(p);
}
if force_screenshot {
cmd["forceScreenshot"] = json!(true);
}
Ok(cmd)
}
_ => Err(ParseError::InvalidValue {
message: "canvas needs a subcommand".to_string(),
usage: "canvas list | canvas capture [selector] [path] [--screenshot]",
}),
}
}
"pdf" => {
let path = rest.first().ok_or_else(|| ParseError::MissingArguments {
context: "pdf".to_string(),
@@ -1016,6 +1329,57 @@ fn parse_command_inner(args: &[String], flags: &Flags) -> Result<Value, ParseErr
Ok(json!({ "id": id, "action": "evaluate", "script": script }))
}
"site" => {
// `site <name>/<command> [positional...] [--key value]`. The
// `update`/`list`/`info` subcommands are handled CLI-side (main.rs)
// and never reach here — by this point `rest[0]` is a `name/cmd`
// adapter spec. Load it, map the args onto the adapter's declared
// `args`, and emit a `site` action: the daemon navigates to the
// adapter's @meta.domain (reusing the tab if already there) and evals
// the adapter function in the site's own logged-in page.
let spec = rest.first().ok_or(ParseError::InvalidValue {
message: "site requires <name>/<command> (run `chrome-use site list`)".to_string(),
usage: "site <name>/<command> [args]",
})?;
let adapter =
crate::site::load_adapter(spec).map_err(|e| ParseError::InvalidValue {
message: e,
usage: "site <name>/<command> [args]",
})?;
let domain = adapter
.domain()
.ok_or(ParseError::InvalidValue {
message: format!("site: adapter `{spec}` @meta is missing a \"domain\""),
usage: "site <name>/<command>",
})?
.to_string();
// Split remaining args: `--key value` → named, everything else → positional.
let mut positional: Vec<String> = Vec::new();
let mut named: Vec<(String, String)> = Vec::new();
let mut it = rest[1..].iter();
while let Some(a) = it.next() {
if let Some(key) = a.strip_prefix("--") {
let val = it.next().map(|s| s.to_string()).unwrap_or_default();
named.push((key.to_string(), val));
} else {
positional.push(a.to_string());
}
}
let mapped = crate::site::map_args(&adapter, &positional, &named);
let script = crate::site::build_eval(&adapter, &mapped);
Ok(json!({ "id": id, "action": "site", "domain": domain, "script": script }))
}
// `adopt <url|targetId>`: the adoption happens at daemon connect (driven by
// the AGENT_BROWSER_ADOPT env main.rs set + a forced-fresh daemon), so by
// the time this command runs the tab is already attached. Resolve to a
// `url` read so the response confirms which tab got adopted.
"adopt" => Ok(json!({ "id": id, "action": "url" })),
// `keep`: leave the active tab for the user — exempt it from the daemon's
// auto-close/idle cleanup and remove it from the session's tab group.
"keep" => Ok(json!({ "id": id, "action": "keep" })),
// === Stealth self-check ===
"stealth" => {
// `stealth [status]` — local stealth self-check: mode, live probes
@@ -1025,6 +1389,14 @@ fn parse_command_inner(args: &[String], flags: &Flags) -> Result<Value, ParseErr
Ok(json!({ "id": id, "action": "stealth_status" }))
}
// `cf-status` — Cloudflare challenge/clearance preflight: is the page
// currently a CF challenge, and is there a still-valid cf_clearance (the
// HttpOnly persistence cookie)? Lets an agent SKIP re-solving when already
// cleared, and know when it must solve. Persistence optimization.
"cf-status" | "cf" | "cloudflare-status" | "clearance" => {
Ok(json!({ "id": id, "action": "cf_status" }))
}
// === Close ===
"close" | "quit" | "exit" => {
// `close <tab>` closes only that tab (and the output says "Tab
@@ -1296,6 +1668,11 @@ fn parse_command_inner(args: &[String], flags: &Flags) -> Result<Value, ParseErr
// === Get ===
"get" => parse_get(&rest, &id),
// List every frame the session can reach (top + same-process child
// frames + out-of-process iframes), with a text-length per frame so you
// can see where a listing's description actually lives (issue #27).
"frames" => Ok(json!({ "id": id, "action": "frames" })),
// Top-level shortcuts for `get <x>` status reads — users naturally type
// `chrome-use url` / `cdp-url` / `title` without the `get` prefix
// (and expect `cdp-url`/`cdp_url` to work interchangeably).
@@ -1326,6 +1703,13 @@ fn parse_command_inner(args: &[String], flags: &Flags) -> Result<Value, ParseErr
// === Mouse ===
"mouse" => parse_mouse(&rest, &id),
// === Viewport / window size ===
// Top-level shortcut for `set viewport` — agents (and the author, in
// issue #47) reach for `viewport`/`resize` first. Uses a CDP virtual
// viewport, so it also works on the extension relay without yanking the
// user's real window around.
"viewport" | "resize" => parse_viewport(&rest, &id),
// === Set (browser settings) ===
"set" => parse_set(&rest, &id),
@@ -2388,11 +2772,42 @@ fn parse_get(rest: &[&str], id: &str) -> Result<Value, ParseError> {
match rest.first().copied() {
Some("text") => {
// `get text` with no selector returns the whole page's text (body) —
// a common convenience; previously it errored without a selector
// (issue #24-D).
let sel = rest.get(1).copied().unwrap_or("body");
Ok(json!({ "id": id, "action": "gettext", "selector": sel }))
// `get text --all-frames` aggregates visible text across every
// frame, including out-of-process iframes invisible to the top
// document (issue #27). The selector is ignored in this mode.
let all_frames = rest[1..]
.iter()
.any(|a| matches!(*a, "--all-frames" | "--frames" | "-a"));
if all_frames {
return Ok(json!({ "id": id, "action": "gettext", "allFrames": true }));
}
// `get text --pierce` reads through CLOSED shadow DOM / child docs
// via the CDP DOM tree — content eval/innerText can't reach, e.g. an
// extension's injected panel in a closed shadow root (issue #30).
let pierce = rest[1..]
.iter()
.any(|a| matches!(*a, "--pierce" | "--shadow" | "--deep"));
if pierce {
return Ok(json!({ "id": id, "action": "gettext", "pierce": true }));
}
// `get text --main` returns the main-content region (readability),
// skipping header/nav/footer/sidebar boilerplate (issue #27).
let main = rest[1..]
.iter()
.any(|a| matches!(*a, "--main" | "--readable" | "-m"));
if main {
return Ok(json!({ "id": id, "action": "gettext", "main": true }));
}
// `get text` with no selector reads the WHOLE PAGE and now defaults
// to cross-frame aggregation, so an agent gets a page's iframed
// content (listing descriptions etc.) without having to know about
// `--all-frames` (#27). On a single-frame page this is identical to
// the old body read; multi-frame pages get the child frames too —
// a strict superset. An explicit selector stays element-scoped.
match rest.iter().skip(1).find(|a| !a.starts_with("--")).copied() {
Some(sel) => Ok(json!({ "id": id, "action": "gettext", "selector": sel })),
None => Ok(json!({ "id": id, "action": "gettext", "allFrames": true })),
}
}
Some("html") => {
let sel = rest.get(1).ok_or_else(|| ParseError::MissingArguments {
@@ -2743,6 +3158,97 @@ fn parse_mouse(rest: &[&str], id: &str) -> Result<Value, ParseError> {
}
}
/// Parse a viewport / resize spec, shared by the top-level `viewport` / `resize`
/// commands and `set viewport`. Sets a CDP device-metrics override
/// (`Emulation.setDeviceMetricsOverride`) — a *virtual* viewport for the tab, so
/// it works headless AND on the extension relay without physically resizing the
/// user's real Chrome window (issue #47). Forms:
/// viewport <width> <height> [scale] [--dpr N] [--mobile]
/// viewport <width>x<height> (e.g. 1280x800)
/// viewport reset | clear (drop the override, restore real size)
fn parse_viewport(rest: &[&str], id: &str) -> Result<Value, ParseError> {
const USAGE: &str = "viewport <width> <height> [scale] [--dpr N] [--mobile] | viewport reset";
if matches!(
rest.first().copied(),
Some("reset") | Some("clear") | Some("off")
) {
return Ok(json!({ "id": id, "action": "viewport", "reset": true }));
}
// Positional (non-flag) tokens. A `WxH` token counts as one positional.
let positionals: Vec<&str> = rest
.iter()
.copied()
.filter(|a| !a.starts_with("--"))
.collect();
let (w, h, scale_tok): (i32, i32, Option<&str>) = match positionals.first() {
Some(first) if first.contains('x') || first.contains('X') => {
let mut parts = first.split(['x', 'X']);
let w = parts.next().and_then(|s| s.parse::<i32>().ok());
let h = parts.next().and_then(|s| s.parse::<i32>().ok());
match (w, h) {
(Some(w), Some(h)) => (w, h, positionals.get(1).copied()),
_ => {
return Err(ParseError::InvalidValue {
message: format!("Invalid viewport size: {}", first),
usage: USAGE,
})
}
}
}
Some(w_str) => {
let h_str = positionals.get(1).ok_or(ParseError::MissingArguments {
context: "viewport".to_string(),
usage: USAGE,
})?;
let w = w_str.parse::<i32>().map_err(|_| ParseError::InvalidValue {
message: format!("Invalid width: {}", w_str),
usage: USAGE,
})?;
let h = h_str.parse::<i32>().map_err(|_| ParseError::InvalidValue {
message: format!("Invalid height: {}", h_str),
usage: USAGE,
})?;
(w, h, positionals.get(2).copied())
}
None => {
return Err(ParseError::MissingArguments {
context: "viewport".to_string(),
usage: USAGE,
})
}
};
let mut cmd = json!({ "id": id, "action": "viewport", "width": w, "height": h });
// Device-scale-factor: positional, overridden by --dpr / --scale.
let mut scale: Option<f64> = match scale_tok {
Some(s) => Some(s.parse::<f64>().map_err(|_| ParseError::InvalidValue {
message: format!("Invalid scale: {}", s),
usage: USAGE,
})?),
None => None,
};
if let Some(i) = rest.iter().position(|a| *a == "--dpr" || *a == "--scale") {
let v = rest.get(i + 1).and_then(|s| s.parse::<f64>().ok()).ok_or(
ParseError::InvalidValue {
message: "--dpr/--scale needs a number".to_string(),
usage: USAGE,
},
)?;
scale = Some(v);
}
if let Some(s) = scale {
cmd["deviceScaleFactor"] = json!(s);
}
if rest.contains(&"--mobile") {
cmd["mobile"] = json!(true);
}
Ok(cmd)
}
fn parse_set(rest: &[&str], id: &str) -> Result<Value, ParseError> {
const VALID: &[&str] = &[
"viewport",
@@ -2757,39 +3263,8 @@ fn parse_set(rest: &[&str], id: &str) -> Result<Value, ParseError> {
];
match rest.first().copied() {
Some("viewport") => {
let w_str = rest.get(1).ok_or_else(|| ParseError::MissingArguments {
context: "set viewport".to_string(),
usage: "set viewport <width> <height> [scale]",
})?;
let h_str = rest.get(2).ok_or_else(|| ParseError::MissingArguments {
context: "set viewport".to_string(),
usage: "set viewport <width> <height> [scale]",
})?;
let w = w_str
.parse::<i32>()
.map_err(|_| ParseError::MissingArguments {
context: "set viewport".to_string(),
usage: "set viewport <width> <height> [scale]",
})?;
let h = h_str
.parse::<i32>()
.map_err(|_| ParseError::MissingArguments {
context: "set viewport".to_string(),
usage: "set viewport <width> <height> [scale]",
})?;
let mut cmd = json!({ "id": id, "action": "viewport", "width": w, "height": h });
if let Some(scale_str) = rest.get(3) {
let scale = scale_str
.parse::<f64>()
.map_err(|_| ParseError::MissingArguments {
context: "set viewport".to_string(),
usage: "set viewport <width> <height> [scale]",
})?;
cmd["deviceScaleFactor"] = json!(scale);
}
Ok(cmd)
}
// `set viewport ...` is an alias for the top-level `viewport` command.
Some("viewport") => parse_viewport(&rest[1..], id),
Some("device") => {
let dev = rest.get(1).ok_or_else(|| ParseError::MissingArguments {
context: "set device".to_string(),
@@ -3946,6 +4421,28 @@ mod tests {
assert_eq!(cmd["action"], "type");
assert_eq!(cmd["selector"], "#input");
assert_eq!(cmd["text"], "some text");
assert_eq!(cmd["keyEvents"], false);
}
#[test]
fn test_type_key_events() {
// --key-events sends real keystrokes (for autocomplete/combobox) and must
// not be swallowed into the typed text.
let cmd = parse_command(
&args("type #postal 201-0001 --key-events"),
&default_flags(),
)
.unwrap();
assert_eq!(cmd["action"], "type");
assert_eq!(cmd["selector"], "#postal");
assert_eq!(cmd["text"], "201-0001");
assert_eq!(cmd["keyEvents"], true);
let focused =
parse_command(&args("type --focused 201-0001 --keys"), &default_flags()).unwrap();
assert_eq!(focused["focused"], true);
assert_eq!(focused["text"], "201-0001");
assert_eq!(focused["keyEvents"], true);
}
#[test]
@@ -4230,6 +4727,24 @@ mod tests {
assert_eq!(cmd["fullPage"], true);
}
#[test]
fn test_screenshot_clip() {
// `--clip x,y,w,h` captures a pixel region (issue #34); the path still parses.
let cmd = parse_command(
&args("screenshot --clip 10,20,200,40 out.png"),
&default_flags(),
)
.unwrap();
assert_eq!(cmd["action"], "screenshot");
assert_eq!(cmd["clip"]["x"], 10.0);
assert_eq!(cmd["clip"]["y"], 20.0);
assert_eq!(cmd["clip"]["width"], 200.0);
assert_eq!(cmd["clip"]["height"], 40.0);
assert_eq!(cmd["path"], "out.png");
// Bad clip is a clear error, not silent.
assert!(parse_command(&args("screenshot --clip 1,2,3"), &default_flags()).is_err());
}
#[test]
fn test_screenshot_with_ref() {
let cmd = parse_command(&args("screenshot @e1"), &default_flags()).unwrap();
@@ -4750,15 +5265,84 @@ mod tests {
}
#[test]
fn test_get_text_defaults_to_body() {
// `get text` with no selector now returns the whole page (body) instead
// of erroring (issue #24-D).
fn test_get_text_defaults_to_all_frames() {
// `get text` with no selector now reads the whole page across ALL frames
// by default (#27), so iframed content isn't silently missed. (Was: a
// top-frame `body` read, #24-D.)
let cmd = parse_command(&args("get text"), &default_flags()).unwrap();
assert_eq!(cmd["action"], "gettext");
assert_eq!(cmd["selector"], "body");
// An explicit selector still wins.
assert_eq!(cmd["allFrames"], true);
assert!(cmd.get("selector").is_none());
// An explicit selector still wins and stays element-scoped.
let cmd2 = parse_command(&args("get text h1"), &default_flags()).unwrap();
assert_eq!(cmd2["selector"], "h1");
assert!(cmd2.get("allFrames").is_none());
// `text` top-level shortcut behaves the same.
let cmd3 = parse_command(&args("text"), &default_flags()).unwrap();
assert_eq!(cmd3["allFrames"], true);
}
#[test]
fn test_get_text_all_frames() {
// `--all-frames` switches to whole-page, cross-frame aggregation and
// drops the selector (issue #27).
for variant in ["get text --all-frames", "get text --frames", "text -a"] {
let cmd = parse_command(&args(variant), &default_flags()).unwrap();
assert_eq!(cmd["action"], "gettext", "{variant}");
assert_eq!(cmd["allFrames"], true, "{variant}");
assert!(cmd.get("selector").is_none(), "{variant}");
}
// A flag mixed with a selector still triggers all-frames.
let cmd = parse_command(&args("get text body --all-frames"), &default_flags()).unwrap();
assert_eq!(cmd["allFrames"], true);
// Without the flag, a leading flag-like token is skipped for the selector.
let cmd = parse_command(&args("get text main"), &default_flags()).unwrap();
assert_eq!(cmd["selector"], "main");
assert!(cmd.get("allFrames").is_none());
}
#[test]
fn test_frames_command() {
let cmd = parse_command(&args("frames"), &default_flags()).unwrap();
assert_eq!(cmd["action"], "frames");
}
#[test]
fn test_nearest_command_suggestions() {
assert_eq!(nearest_command("sesions").as_deref(), Some("sessions"));
assert_eq!(nearest_command("session").as_deref(), Some("sessions"));
assert_eq!(nearest_command("clik").as_deref(), Some("click"));
assert_eq!(
nearest_command("screenshits").as_deref(),
Some("screenshot")
);
// Nonsense with no close match stays silent.
assert_eq!(nearest_command("xyzzy"), None);
// The unknown-command error embeds the suggestion.
let err = ParseError::UnknownCommand {
command: "sesions".to_string(),
};
assert!(err.format().contains("Did you mean: chrome-use sessions?"));
}
#[test]
fn test_get_text_main() {
for variant in ["get text --main", "get text --readable", "text -m"] {
let cmd = parse_command(&args(variant), &default_flags()).unwrap();
assert_eq!(cmd["action"], "gettext", "{variant}");
assert_eq!(cmd["main"], true, "{variant}");
assert!(cmd.get("selector").is_none(), "{variant}");
}
}
#[test]
fn test_get_text_pierce() {
for variant in ["get text --pierce", "get text --shadow", "text --deep"] {
let cmd = parse_command(&args(variant), &default_flags()).unwrap();
assert_eq!(cmd["action"], "gettext", "{variant}");
assert_eq!(cmd["pierce"], true, "{variant}");
assert!(cmd.get("selector").is_none(), "{variant}");
}
}
#[test]
@@ -4841,6 +5425,62 @@ mod tests {
assert!(result.is_err());
}
#[test]
fn test_viewport_toplevel() {
let cmd = parse_command(&args("viewport 1280 800"), &default_flags()).unwrap();
assert_eq!(cmd["action"], "viewport");
assert_eq!(cmd["width"], 1280);
assert_eq!(cmd["height"], 800);
assert!(cmd.get("deviceScaleFactor").is_none());
assert!(cmd.get("mobile").is_none());
}
#[test]
fn test_resize_alias() {
let cmd = parse_command(&args("resize 700 800"), &default_flags()).unwrap();
assert_eq!(cmd["action"], "viewport");
assert_eq!(cmd["width"], 700);
assert_eq!(cmd["height"], 800);
}
#[test]
fn test_viewport_wxh_form() {
let cmd = parse_command(&args("viewport 375x812"), &default_flags()).unwrap();
assert_eq!(cmd["action"], "viewport");
assert_eq!(cmd["width"], 375);
assert_eq!(cmd["height"], 812);
}
#[test]
fn test_viewport_dpr_and_mobile_flags() {
let cmd =
parse_command(&args("viewport 375 812 --dpr 3 --mobile"), &default_flags()).unwrap();
assert_eq!(cmd["action"], "viewport");
assert_eq!(cmd["width"], 375);
assert_eq!(cmd["height"], 812);
assert_eq!(cmd["deviceScaleFactor"], 3.0);
assert_eq!(cmd["mobile"], true);
}
#[test]
fn test_viewport_reset() {
for spec in ["viewport reset", "viewport clear", "resize reset"] {
let cmd = parse_command(&args(spec), &default_flags()).unwrap();
assert_eq!(cmd["action"], "viewport", "{spec}");
assert_eq!(cmd["reset"], true, "{spec}");
}
}
#[test]
fn test_viewport_missing_height() {
assert!(parse_command(&args("viewport 1280"), &default_flags()).is_err());
}
#[test]
fn test_viewport_invalid_width() {
assert!(parse_command(&args("viewport abc 800"), &default_flags()).is_err());
}
#[test]
fn test_find_first_no_value() {
let cmd = parse_command(&args("find first a click"), &default_flags()).unwrap();
@@ -5667,6 +6307,35 @@ mod tests {
assert_eq!(cmd["selector"], ".sidebar");
}
#[test]
fn test_scroll_at_coordinate() {
// `--at x,y` carries a [x, y] array for a wheel dispatched at that pixel
// (issue #36: cross-origin iframe scroll).
let cmd = parse_command(&args("scroll down 700 --at 640,400"), &default_flags()).unwrap();
assert_eq!(cmd["action"], "scroll");
assert_eq!(cmd["direction"], "down");
assert_eq!(cmd["amount"], 700);
assert_eq!(cmd["at"], json!([640.0, 400.0]));
}
#[test]
fn test_scroll_at_rejects_garbage() {
assert!(parse_command(&args("scroll --at nope"), &default_flags()).is_err());
assert!(parse_command(&args("scroll --at 1"), &default_flags()).is_err());
}
#[test]
fn test_scroll_frame_index() {
let cmd = parse_command(&args("scroll down 700 --frame 2"), &default_flags()).unwrap();
assert_eq!(cmd["action"], "scroll");
assert_eq!(cmd["frame"], 2);
}
#[test]
fn test_scroll_frame_rejects_non_integer() {
assert!(parse_command(&args("scroll --frame two"), &default_flags()).is_err());
}
#[test]
fn test_scroll_selector_before_positional() {
let cmd =
+41
View File
@@ -449,6 +449,47 @@ pub fn relay_url() -> Option<String> {
}
}
/// Append a one-line record of how a CDP connection was established, to
/// `~/.chrome-use/connect-mode.log`. This is the smoking-gun detector for the
/// "Allow remote debugging?" consent modal: that modal ONLY appears on a raw
/// remote-debugging attach / a browser we launched with a debug port — NEVER on
/// the extension relay. When the modal reappears, this log says which session
/// took which path and when, so we can tell a code regression (`raw-port` /
/// `launched` while the relay was up) from Chrome's own extension-debugger
/// consent UX. Low volume (one line per connection); best-effort, never fails a
/// connection.
pub fn log_connect_mode(ws_url: &str, launched: bool, session: &str) {
let relay = relay_url();
let relay_up = relay.is_some();
let mode = if launched {
"launched(debug-port)"
} else if relay.as_deref() == Some(ws_url) {
"relay"
} else if ws_url.contains("127.0.0.1") || ws_url.contains("localhost") {
"raw-port-attach"
} else {
"remote-ws"
};
// A raw-port attach or a self-launch while the relay was available is the
// exact thing that pops the consent modal — flag it loudly in the line.
let suspect = (mode == "raw-port-attach" || launched) && relay_up;
let line = format!(
"session={session} mode={mode} relay_up={relay_up}{} ws={ws_url}\n",
if suspect { " CONSENT-MODAL-RISK" } else { "" }
);
if let Some(home) = dirs::home_dir() {
let path = home.join(".chrome-use").join("connect-mode.log");
use std::io::Write;
if let Ok(mut f) = std::fs::OpenOptions::new()
.create(true)
.append(true)
.open(&path)
{
let _ = f.write_all(line.as_bytes());
}
}
}
/// Sidecar recording the connected extension's version, written by the host when
/// it receives the extension's `hello` (sibling of `relay-cdp-url`). Lets
/// `doctor` surface which extension build is live without a CDP round-trip.
+1 -1
View File
@@ -595,7 +595,7 @@ fn query_current_url(session: &str) -> Option<String> {
}
/// Kill a running daemon by reading its PID file and sending a kill signal.
fn kill_stale_daemon(session: &str) {
pub fn kill_stale_daemon(session: &str) {
// Remove the socket first so no new connections reach the old daemon
#[cfg(unix)]
{
+202
View File
@@ -10,6 +10,7 @@ mod flags;
mod install;
mod native;
mod output;
mod site;
mod skills;
mod test_runner;
#[cfg(test)]
@@ -304,6 +305,49 @@ fn run_session(args: &[String], session: &str, json_mode: bool) {
}
}
}
// Stop a specific session daemon (issue #48). Graceful: kill_stale_daemon
// sends SIGTERM first, so the daemon's shutdown handler runs `close()` and
// tidies the tabs IT created (its tab group) before exiting.
Some("stop") => {
let target = args.get(2).map(|s| s.as_str()).unwrap_or(session);
connection::kill_stale_daemon(target);
if json_mode {
print_json_value(json!({ "success": true, "data": { "stopped": target } }));
} else {
println!(
"{} stopped session daemon: {}",
color::success_indicator(),
target
);
}
}
// Reclaim ALL session daemons now (issue #48) — for clearing the pile of
// idle daemons left after a round of automation/debugging without waiting
// for the idle timeout. Each is stopped gracefully (closes its own tabs);
// they respawn clean on next use. The `__nm-host` relay is not a tracked
// session daemon, so the extension/live-Chrome connection survives.
Some("prune") => {
let sessions: Vec<String> = walk_daemons()
.sessions
.into_iter()
.map(|s| s.name)
.collect();
for s in &sessions {
connection::kill_stale_daemon(s);
}
if json_mode {
print_json_value(json!({ "success": true, "data": { "pruned": sessions } }));
} else if sessions.is_empty() {
println!("No session daemons to prune");
} else {
println!(
"{} pruned {} session daemon(s): {}",
color::success_indicator(),
sessions.len(),
sessions.join(", ")
);
}
}
None | Some(_) => {
// Just show current session
if json_mode {
@@ -834,6 +878,105 @@ fn main() {
exit(test_runner::run_test(suite, &flags));
}
// Handle `site`: site adapters — turn a website into a structured-data CLI by
// running a per-command JS adapter inside your logged-in tab. `update`/`list`/
// `info` are CLI-side (download/filesystem); `site <name>/<cmd> [args]` falls
// through to the daemon dispatch below (navigate to the adapter's domain + eval).
if clean.first().map(|s| s.as_str()) == Some("site") {
// Auto-sync the adapter pack on first use and periodically (TTL, default
// 7d) so adapters stay fresh without a manual `site update`. Skipped for an
// explicit `update` (full sync below). Best-effort: offline → cached pack.
// Disable with AGENT_BROWSER_SITES_NO_AUTO_UPDATE=1.
if clean.get(1).map(|s| s.as_str()) != Some("update") && site::needs_refresh() {
let rt = tokio::runtime::Runtime::new().expect("Failed to create tokio runtime");
match rt.block_on(site::update()) {
Ok(n) => {
eprintln!(
"{}",
color::dim(&format!("site: synced {n} adapters (auto)"))
)
}
Err(e) => eprintln!(
"{}",
color::dim(&format!(
"site: auto-sync skipped ({e}); using cached adapters"
))
),
}
}
match clean.get(1).map(|s| s.as_str()) {
Some("update") => {
let rt = tokio::runtime::Runtime::new().expect("Failed to create tokio runtime");
match rt.block_on(site::update()) {
Ok(n) if flags.json => {
println!("{}", json!({ "success": true, "adapters": n }))
}
Ok(n) => println!(
"{} synced {} site adapters → ~/.chrome-use/sites (run `chrome-use site list`)",
color::success_indicator(),
n
),
Err(e) => {
eprintln!("{} {}", color::error_indicator(), e);
exit(1);
}
}
return;
}
Some("list") => {
match site::list_adapters() {
Ok(list) if flags.json => {
println!("{}", json!({ "success": true, "adapters": list }))
}
Ok(list) if list.is_empty() => {
println!("no site adapters installed — run `chrome-use site update`")
}
Ok(list) => {
for a in &list {
println!("{a}");
}
eprintln!(
"{}",
color::dim(&format!(
"{} adapters · run: chrome-use site <name>/<cmd> [args]",
list.len()
))
);
}
Err(e) => {
eprintln!("{} {}", color::error_indicator(), e);
exit(1);
}
}
return;
}
Some("info") => {
let spec = clean.get(2).cloned().unwrap_or_default();
match site::load_adapter(&spec) {
Ok(a) => println!(
"{}",
serde_json::to_string_pretty(&a.meta).unwrap_or_default()
),
Err(e) => {
eprintln!("{} {}", color::error_indicator(), e);
exit(1);
}
}
return;
}
// `site <name>/<cmd> [args]` → fall through to the daemon dispatch.
Some(spec) if spec.contains('/') => {}
_ => {
eprintln!(
"{} usage: chrome-use site <name>/<cmd> [args] | site update | site list | \
site info <name>/<cmd>",
color::error_indicator()
);
exit(2);
}
}
}
// Handle skills command (doesn't need daemon)
if clean.first().map(|s| s.as_str()) == Some("skills") {
skills::run_skills(&clean, flags.json);
@@ -881,6 +1024,43 @@ fn main() {
}
}
// `adopt <url|targetId>`: read a PRE-EXISTING tab (the user's own, or another
// session's) WITHOUT opening a new one. Forces a fresh daemon and points it at
// the relay (like `extension connect`); the AGENT_BROWSER_ADOPT env makes the
// daemon's first connect ADOPT the matching tab instead of creating an
// about:blank. Rewrites into `connect <relay-url>` BEFORE parse_command so the
// daemon attaches to the user's real Chrome. Must run before parse_command.
if clean.first().map(|s| s.as_str()) == Some("adopt") {
match clean.get(1) {
Some(spec) if !spec.trim().is_empty() => {
std::env::set_var("AGENT_BROWSER_ADOPT", spec.trim());
connection::kill_stale_daemon(&flags.session);
match connect::relay_url() {
Some(url) => {
flags.cdp = Some(url.clone());
flags.auto_connect = false;
clean = vec!["connect".to_string(), url];
}
None => {
eprintln!(
"{} extension relay not connected — open Chrome with the ab-connect \
extension first (this command reads an EXISTING tab, it won't launch one).",
color::error_indicator()
);
exit(1);
}
}
}
_ => {
eprintln!(
"{} usage: chrome-use adopt <url-substring|targetId> (reads an existing tab, no new tab)",
color::error_indicator()
);
exit(2);
}
}
}
// Handle session separately (doesn't need daemon)
if clean.first().map(|s| s.as_str()) == Some("session") {
run_session(&clean, &flags.session, flags.json);
@@ -893,6 +1073,14 @@ fn main() {
return;
}
// `sessions` is a natural top-level guess for "list my sessions" (the skill
// advertises sessions as a feature) — route it to the daemon inventory the
// same way `daemon status` does (issue #29).
if clean.first().map(|s| s.as_str()) == Some("sessions") {
run_daemon(&["sessions".to_string(), "status".to_string()], flags.json);
return;
}
// Handle close --all: close all active sessions
if matches!(
clean.first().map(|s| s.as_str()),
@@ -1361,6 +1549,20 @@ fn main() {
&& flags.provider.is_none()
&& (flags.force_launch || !flags.auto_connect)
{
// Launching a debug-port Chrome pops Chrome's "Allow remote debugging?"
// consent modal (Chrome 136+). When the ab-connect relay is already up,
// this is almost always unintended — the relay drives the user's real
// Chrome with NO modal. Warn so the modal is self-explained and the
// caller (often a stray --launch / --no-auto-connect) is fixable (#32).
if !flags.json && connect::relay_url().is_some() {
eprintln!(
"{} launching a new Chrome with a debug port — this pops Chrome's \
\"Allow remote debugging?\" modal.\n The ab-connect relay is up; \
drop --launch/--new (and don't pass --no-auto-connect) to drive your \
real Chrome with no modal.",
color::warning_indicator()
);
}
let mut launch_cmd = json!({
"id": gen_id(),
"action": "launch",
+804 -11
View File
@@ -1315,10 +1315,14 @@ pub async fn execute_command(cmd: &Value, state: &mut DaemonState) -> Value {
"title" => handle_title(state).await,
"content" => handle_content(state).await,
"evaluate" => handle_evaluate(cmd, state).await,
"site" => handle_site(cmd, state).await,
"close" => handle_close(state).await,
"keep" => handle_keep(state).await,
"stealth_status" => handle_stealth_status(state).await,
"snapshot" => handle_snapshot(cmd, state).await,
"screenshot" => handle_screenshot(cmd, state).await,
"canvas_list" => handle_canvas_list(state).await,
"canvas_capture" => handle_canvas_capture(cmd, state).await,
"click" => handle_click(cmd, state).await,
"dblclick" => handle_dblclick(cmd, state).await,
"fill" => handle_fill(cmd, state).await,
@@ -1332,6 +1336,7 @@ pub async fn execute_command(cmd: &Value, state: &mut DaemonState) -> Value {
"uncheck" => handle_uncheck(cmd, state).await,
"wait" => handle_wait(cmd, state).await,
"gettext" => handle_gettext(cmd, state).await,
"frames" => handle_frames(cmd, state).await,
"getattribute" => handle_getattribute(cmd, state).await,
"isvisible" => handle_isvisible(cmd, state).await,
"isenabled" => handle_isenabled(cmd, state).await,
@@ -1340,6 +1345,7 @@ pub async fn execute_command(cmd: &Value, state: &mut DaemonState) -> Value {
"forward" => handle_forward(state).await,
"reload" => handle_reload(state).await,
"cookies_get" => handle_cookies_get(cmd, state).await,
"cf_status" => handle_cf_status(cmd, state).await,
"cookies_set" => handle_cookies_set(cmd, state).await,
"cookies_clear" => handle_cookies_clear(state).await,
"storage_get" => handle_storage_get(cmd, state).await,
@@ -2475,7 +2481,10 @@ async fn handle_navigate(cmd: &Value, state: &mut DaemonState) -> Result<Value,
wb.navigate(url).await?;
let new_url = wb.get_url().await.unwrap_or_else(|_| url.to_string());
let title = wb.get_title().await.unwrap_or_default();
return Ok(json!({ "url": new_url, "title": title }));
return Ok(with_site_hint(
json!({ "url": new_url, "title": title }),
url,
));
}
}
@@ -2542,7 +2551,7 @@ async fn handle_navigate(cmd: &Value, state: &mut DaemonState) -> Result<Value,
.unwrap_or(false)
{
if let Ok(Some(switched)) = mgr.reuse_tab_for_url(url).await {
return Ok(switched);
return Ok(with_site_hint(switched, url));
}
}
@@ -2550,7 +2559,34 @@ async fn handle_navigate(cmd: &Value, state: &mut DaemonState) -> Result<Value,
// Adaptive humanize: sample the freshly loaded page for known behavioural
// anti-bot vendors and escalate this session to Human if any are present.
detect_and_set_humanize(mgr).await;
Ok(result)
Ok(with_site_hint(result, url))
}
/// Annotate a navigation/snapshot result with the `site` adapters available for
/// the page's domain (auto-trigger): when you land on e.g. github.com, the
/// response carries `siteAdapters: { domain, commands: ["github/issues", …] }` so
/// the agent reaches for a structured-data adapter instead of scraping. No-op
/// when nothing matches or the pack isn't synced yet.
fn with_site_hint(mut result: Value, fallback_url: &str) -> Value {
let url = result
.get("url")
.and_then(|v| v.as_str())
.unwrap_or(fallback_url);
let host = url::Url::parse(url)
.ok()
.and_then(|u| u.host_str().map(String::from));
if let Some(host) = host {
let adapters = crate::site::adapters_for_domain(&host);
if !adapters.is_empty() {
if let Some(obj) = result.as_object_mut() {
obj.insert(
"siteAdapters".to_string(),
json!({ "domain": host, "commands": adapters }),
);
}
}
}
result
}
/// After navigation, probe the page for known anti-bot vendor fingerprints
@@ -2696,6 +2732,47 @@ async fn handle_evaluate(cmd: &Value, state: &DaemonState) -> Result<Value, Stri
Ok(json!({ "result": result, "origin": url }))
}
/// Run a site adapter: navigate to its `@meta.domain` (only if we're not already
/// there — the point is to run as you, in the page that's already open) and eval
/// the adapter function in the site's own logged-in page. The CLI/commands.rs has
/// already loaded the adapter and built the `script`; here we just place the page
/// and evaluate. Never disrupts the user's foreground tab — navigation happens on
/// the daemon's own tab (same as every other command on the relay).
async fn handle_site(cmd: &Value, state: &mut DaemonState) -> Result<Value, String> {
let domain = cmd
.get("domain")
.and_then(|v| v.as_str())
.ok_or("site: missing 'domain'")?
.to_string();
let script = cmd
.get("script")
.and_then(|v| v.as_str())
.ok_or("site: missing 'script'")?
.to_string();
let current = match state.browser.as_ref() {
Some(mgr) => mgr.get_url().await.unwrap_or_default(),
None => String::new(),
};
let on_domain = url::Url::parse(&current)
.ok()
.and_then(|u| u.host_str().map(|h| h.to_string()))
.map(|h| h == domain || h.ends_with(&format!(".{domain}")))
.unwrap_or(false);
if !on_domain {
let nav = json!({ "url": format!("https://{domain}/") });
handle_navigate(&nav, state).await?;
}
let eval_cmd = json!({ "script": script });
let out = handle_evaluate(&eval_cmd, state).await?;
Ok(json!({
"result": out.get("result").cloned().unwrap_or(Value::Null),
"origin": out.get("origin").cloned().unwrap_or(Value::Null),
"domain": domain,
}))
}
/// Local stealth self-check: reports the active mode, live fingerprint probes,
/// and the list of applied overrides — so an agent (or human) can confirm
/// stealth is working without driving an external detector, and audit exactly
@@ -2779,6 +2856,34 @@ async fn handle_stealth_status(state: &DaemonState) -> Result<Value, String> {
}))
}
/// `keep` — leave the ACTIVE tab for the user: stop owning it (so the daemon's
/// `close()`/idle-shutdown won't close it) and best-effort remove it from this
/// session's tab group so it looks like a normal user tab. The "leave for the
/// user" half of the auto-close-on-idle cleanup: scratch tabs get closed, tabs
/// the agent explicitly `keep`s stay. (Adopted user tabs are never owned, so
/// they're already safe.)
async fn handle_keep(state: &mut DaemonState) -> Result<Value, String> {
let mgr = state.browser.as_mut().ok_or("Browser not launched")?;
let target_id = mgr.active_target_id()?.to_string();
let session_id = mgr.active_session_id()?.to_string();
let was_owned = mgr.unown_target(&target_id);
// Best-effort: ask the extension to ungroup the tab (relay only; no-ops on a
// launched browser or an older extension that doesn't know ABExt.ungroupTab).
let _ = mgr
.client
.send_command_typed::<_, Value>(
"ABExt.ungroupTab",
&json!({ "sessionId": session_id, "targetId": target_id }),
None,
)
.await;
Ok(json!({
"kept": target_id,
"wasOwned": was_owned,
"note": "tab left for the user — exempt from auto-close, removed from the session tab group",
}))
}
async fn handle_close(state: &mut DaemonState) -> Result<Value, String> {
if let Some(ref mgr) = state.browser {
if let Some(ref session_name) = state.session_name {
@@ -2892,6 +2997,13 @@ async fn handle_snapshot(cmd: &Value, state: &mut DaemonState) -> Result<Value,
let ref_count = refs.len();
let mut out = json!({ "snapshot": tree, "origin": url, "refs": refs });
// Auto-trigger: if this domain has site adapters, surface them so the agent
// pulls structured data instead of walking the tree. `with_site_hint` reads
// the `url` field, so pass it under that key.
if let Some(hint) = with_site_hint(json!({ "url": url }), &url).get("siteAdapters") {
out["siteAdapters"] = hint.clone();
}
// Canvas/WebGL apps (games, map/3D viewers, drawing tools) paint to a
// <canvas> and expose almost no accessibility tree, so `snapshot` comes back
// near-empty and agents get stuck looking for refs that will never exist
@@ -2998,6 +3110,14 @@ async fn handle_screenshot(cmd: &Value, state: &mut DaemonState) -> Result<Value
.get("screenshotDir")
.and_then(|v| v.as_str())
.map(String::from),
clip: cmd.get("clip").and_then(|c| {
Some((
c.get("x")?.as_f64()?,
c.get("y")?.as_f64()?,
c.get("width")?.as_f64()?,
c.get("height")?.as_f64()?,
))
}),
};
if annotate {
@@ -3025,7 +3145,40 @@ async fn handle_screenshot(cmd: &Value, state: &mut DaemonState) -> Result<Value
)
.await?;
// Downscale the saved image so retina/full-page shots fit an agent's image
// reader and screenshot pixels line up with `click x y` CSS px (issue #42).
// Explicit --scale / --max-width / --max-height win; otherwise a default cap
// (2000px longest edge, AGENT_BROWSER_SCREENSHOT_MAX_EDGE overrides, 0 = off)
// applies. Annotated shots are left untouched so ref overlays stay aligned.
let mut resized: Option<(u32, u32)> = None;
if !annotate {
let scale = cmd.get("scale").and_then(|v| v.as_f64());
let max_w = cmd
.get("maxWidth")
.and_then(|v| v.as_u64())
.map(|v| v as u32);
let max_h = cmd
.get("maxHeight")
.and_then(|v| v.as_u64())
.map(|v| v as u32);
let default_edge = if scale.is_none() && max_w.is_none() && max_h.is_none() {
std::env::var("AGENT_BROWSER_SCREENSHOT_MAX_EDGE")
.ok()
.and_then(|s| s.parse::<u32>().ok())
.or(Some(2000))
.filter(|&e| e > 0)
} else {
None
};
resized = downscale_screenshot(&result.path, scale, max_w, max_h, default_edge);
}
let mut response = json!({ "path": absolutize_saved_path(&result.path) });
if let Some((w, h)) = resized {
response["width"] = json!(w);
response["height"] = json!(h);
response["resized"] = json!(true);
}
if !result.annotations.is_empty() {
response["annotations"] = serde_json::to_value(&result.annotations)
.map_err(|e| format!("Failed to serialize annotations: {}", e))?;
@@ -3041,6 +3194,204 @@ async fn handle_screenshot(cmd: &Value, state: &mut DaemonState) -> Result<Value
Ok(response)
}
/// Downscale a saved screenshot in place (issue #42). Resolves the target longest
/// edge from `scale` (fraction of current), explicit `max_w`/`max_h` caps, or a
/// `default_edge` cap — whichever yields the smaller image. Only ever shrinks;
/// no-op (returns None) if the image is already within bounds or can't be read.
/// Returns the new (width, height) when it actually resized.
fn downscale_screenshot(
path: &str,
scale: Option<f64>,
max_w: Option<u32>,
max_h: Option<u32>,
default_edge: Option<u32>,
) -> Option<(u32, u32)> {
let img = image::open(path).ok()?;
let (w, h) = (img.width(), img.height());
if w == 0 || h == 0 {
return None;
}
// Collect candidate scale factors (≤ 1.0); the smallest wins.
let mut factor = 1.0f64;
if let Some(s) = scale {
factor = factor.min(s);
}
if let Some(mw) = max_w {
if w > mw {
factor = factor.min(mw as f64 / w as f64);
}
}
if let Some(mh) = max_h {
if h > mh {
factor = factor.min(mh as f64 / h as f64);
}
}
if let Some(edge) = default_edge {
let longest = w.max(h);
if longest > edge {
factor = factor.min(edge as f64 / longest as f64);
}
}
if factor >= 1.0 {
return None; // already within bounds — never upscale
}
let nw = ((w as f64 * factor).round() as u32).max(1);
let nh = ((h as f64 * factor).round() as u32).max(1);
let resized = img.resize(nw, nh, image::imageops::FilterType::Lanczos3);
resized.save(path).ok()?;
Some((resized.width(), resized.height()))
}
/// `canvas list` — enumerate <canvas> elements (size, visibility, whether
/// toDataURL is usable) so an agent can pick one to capture on a canvas/WebGL app
/// where snapshot/DOM reads see nothing.
async fn handle_canvas_list(state: &DaemonState) -> Result<Value, String> {
let mgr = state.browser.as_ref().ok_or("Browser not launched")?;
let js = r#"(() => [...document.querySelectorAll('canvas')].map((c, i) => {
const r = c.getBoundingClientRect();
let toDataUrl = true, tainted = false;
try { c.toDataURL('image/png'); } catch (e) { toDataUrl = false; tainted = true; }
return {
index: i,
backingWidth: c.width, backingHeight: c.height,
cssWidth: Math.round(r.width), cssHeight: Math.round(r.height),
visible: r.width > 0 && r.height > 0 && r.bottom > 0 && r.top < innerHeight,
id: c.id || null, className: c.className || null,
toDataUrl, tainted,
};
}))()"#;
let canvases = mgr.evaluate(js, None).await?;
Ok(json!({ "canvases": canvases }))
}
/// `canvas capture [selector] [path]` — save a canvas's rendered pixels to PNG.
/// Prefers `toDataURL` (full backing-store resolution); falls back to a CDP
/// screenshot of the canvas element when toDataURL is blank (WebGL without
/// preserveDrawingBuffer, e.g. Figma) or throws (cross-origin tainted). This is
/// how chrome-use "sees" canvas/WebGL apps that expose no DOM or refs.
async fn handle_canvas_capture(cmd: &Value, state: &mut DaemonState) -> Result<Value, String> {
let selector = cmd
.get("selector")
.and_then(|v| v.as_str())
.map(String::from);
let force_screenshot = cmd
.get("forceScreenshot")
.and_then(|v| v.as_bool())
.unwrap_or(false);
let out_path = cmd.get("path").and_then(|v| v.as_str()).map(String::from);
let id = cmd.get("id").and_then(|v| v.as_str()).unwrap_or("0");
// Probe: locate the canvas, get its bbox, and try toDataURL (unless forced).
let probe = {
let mgr = state.browser.as_ref().ok_or("Browser not launched")?;
let sel_lit = match &selector {
Some(s) => serde_json::to_string(s).unwrap_or_else(|_| "null".to_string()),
None => "null".to_string(),
};
let js = format!(
r#"(() => {{
const sel = {sel_lit};
const c = sel ? document.querySelector(sel)
: [...document.querySelectorAll('canvas')].sort((a,b)=>(b.width*b.height)-(a.width*a.height))[0];
if (!c) return {{ found: false, count: document.querySelectorAll('canvas').length }};
const r = c.getBoundingClientRect();
let dataUrl = null, err = null;
if ({try_data}) {{ try {{ dataUrl = c.toDataURL('image/png'); }} catch (e) {{ err = String(e && e.message || e); }} }}
return {{ found: true, w: c.width, h: c.height, x: r.x, y: r.y, cw: r.width, ch: r.height, dataUrl, err }};
}})()"#,
try_data = !force_screenshot
);
mgr.evaluate(&js, None).await?
};
if !probe
.get("found")
.and_then(|v| v.as_bool())
.unwrap_or(false)
{
let count = probe.get("count").and_then(|v| v.as_u64()).unwrap_or(0);
return Err(format!(
"canvas: no canvas matched{} ({count} canvas element(s) on the page — try `canvas list`)",
selector
.as_deref()
.map(|s| format!(" `{s}`"))
.unwrap_or_default()
));
}
let backing_w = probe.get("w").and_then(|v| v.as_u64()).unwrap_or(0);
let backing_h = probe.get("h").and_then(|v| v.as_u64()).unwrap_or(0);
// Decode toDataURL if present; a WebGL canvas without preserveDrawingBuffer
// returns a blank PNG (tiny when compressed), so reject suspiciously small
// results and fall back to the screenshot path.
let decoded: Option<Vec<u8>> = probe
.get("dataUrl")
.and_then(|v| v.as_str())
.and_then(|u| u.split_once(",").map(|(_, b)| b.to_string()))
.and_then(|b64| {
base64::Engine::decode(&base64::engine::general_purpose::STANDARD, b64.as_bytes()).ok()
});
let use_data = !force_screenshot && decoded.as_ref().map(|d| d.len() > 1024).unwrap_or(false);
if use_data {
let bytes = decoded.unwrap();
let path = out_path.unwrap_or_else(|| {
std::env::temp_dir()
.join(format!("canvas-{id}.png"))
.to_string_lossy()
.into_owned()
});
std::fs::write(&path, &bytes).map_err(|e| format!("canvas: write {path}: {e}"))?;
Ok(json!({
"path": absolutize_saved_path(&path),
"method": "toDataURL",
"width": backing_w,
"height": backing_h,
}))
} else {
// Fallback: screenshot the canvas element (or its bbox clip).
let clip = if selector.is_none() {
let x = probe.get("x").and_then(|v| v.as_f64()).unwrap_or(0.0);
let y = probe.get("y").and_then(|v| v.as_f64()).unwrap_or(0.0);
let cw = probe.get("cw").and_then(|v| v.as_f64()).unwrap_or(0.0);
let ch = probe.get("ch").and_then(|v| v.as_f64()).unwrap_or(0.0);
Some((x, y, cw, ch))
} else {
None
};
let options = ScreenshotOptions {
selector: selector.clone(),
path: out_path,
clip,
..ScreenshotOptions::default()
};
let mgr = state.browser.as_ref().ok_or("Browser not launched")?;
let session_id = mgr.active_session_id()?.to_string();
let result = screenshot::take_screenshot(
&mgr.client,
&session_id,
&state.ref_map,
&options,
&state.iframe_sessions,
)
.await?;
let why = probe
.get("err")
.and_then(|v| v.as_str())
.map(|e| format!("toDataURL failed ({e})"))
.unwrap_or_else(|| {
"toDataURL blank/unavailable (WebGL no preserveDrawingBuffer)".into()
});
Ok(json!({
"path": absolutize_saved_path(&result.path),
"method": "screenshot",
"note": format!("captured rendered pixels via screenshot — {why}"),
}))
}
}
async fn handle_click(cmd: &Value, state: &mut DaemonState) -> Result<Value, String> {
// First-class coordinate click (issue #8.4): click a raw viewport point with
// no element resolution. Parsed from `click <x> <y>` / `click --coords x,y`.
@@ -3199,7 +3550,7 @@ async fn handle_fill(cmd: &Value, state: &mut DaemonState) -> Result<Value, Stri
let mgr = state.browser.as_ref().ok_or("Browser not launched")?;
let session_id = mgr.active_session_id()?.to_string();
interaction::fill(
let engine = interaction::fill(
&mgr.client,
&session_id,
&state.ref_map,
@@ -3208,13 +3559,23 @@ async fn handle_fill(cmd: &Value, state: &mut DaemonState) -> Result<Value, Stri
&state.iframe_sessions,
)
.await?;
Ok(json!({ "filled": selector }))
// Echo the input path used (input/contenteditable/codemirror5/monaco/select)
// so the agent can confirm a rich editor was handled, not silently no-op'd (#41).
Ok(json!({ "filled": selector, "engine": engine }))
}
async fn handle_type(cmd: &Value, state: &mut DaemonState) -> Result<Value, String> {
let mgr = state.browser.as_ref().ok_or("Browser not launched")?;
let session_id = mgr.active_session_id()?.to_string();
// `--key-events`: dispatch real per-character keyDown/keyUp instead of
// Input.insertText, so autocomplete/combobox widgets that only react to key
// events fire (e.g. Google's address postal-code lookup) (issue #4/#36).
let key_events = cmd
.get("keyEvents")
.and_then(|v| v.as_bool())
.unwrap_or(false);
// `type --focused <text>`: type into the currently-focused element without a
// selector (custom widgets that move focus to a hidden input on open).
if cmd
@@ -3226,7 +3587,14 @@ async fn handle_type(cmd: &Value, state: &mut DaemonState) -> Result<Value, Stri
.get("text")
.and_then(|v| v.as_str())
.ok_or("Missing 'text' parameter")?;
interaction::type_text_into_active_context(&mgr.client, &session_id, text, None).await?;
interaction::type_text_into_active_context(
&mgr.client,
&session_id,
text,
None,
key_events,
)
.await?;
return Ok(json!({ "typed": text, "focused": true }));
}
@@ -3250,6 +3618,7 @@ async fn handle_type(cmd: &Value, state: &mut DaemonState) -> Result<Value, Stri
clear,
delay,
&state.iframe_sessions,
key_events,
)
.await?;
Ok(json!({ "typed": text }))
@@ -3457,17 +3826,182 @@ async fn handle_scroll(cmd: &Value, state: &mut DaemonState) -> Result<Value, St
}
}
// An explicit `--selector` keeps the precise element-scroll path (scrollBy on
// the resolved node, same-origin only).
if let Some(sel) = selector {
interaction::scroll(
&mgr.client,
&session_id,
&state.ref_map,
Some(sel),
dx,
dy,
&state.iframe_sessions,
)
.await?;
return Ok(json!({ "scrolled": true, "via": "selector" }));
}
// `--at x,y` / `--frame n`: dispatch a real (isTrusted) wheel at a viewport
// coordinate. This hits the compositor and scrolls whatever scroll container
// is under the pointer — including cross-origin iframes that `window.scrollBy`
// on the top document silently no-ops on (issue #36).
if cmd.get("at").is_some() || cmd.get("frame").is_some() {
let (x, y, via) = if let Some(at) = cmd.get("at").and_then(|v| v.as_array()) {
let x = at.first().and_then(|v| v.as_f64()).unwrap_or(0.0);
let y = at.get(1).and_then(|v| v.as_f64()).unwrap_or(0.0);
(x, y, "at")
} else {
let n = cmd.get("frame").and_then(|v| v.as_u64()).unwrap_or(0);
let (x, y) = frame_center(mgr, &session_id, &state.iframe_sessions, n as usize).await?;
(x, y, "frame")
};
dispatch_wheel(&mgr.client, &session_id, x, y, dx, dy).await?;
return Ok(json!({ "scrolled": true, "via": via, "at": [x, y] }));
}
// Default (no selector/at/frame): scroll the page with `window.scrollBy`. This
// is the reliable path for ordinary page scrolling; a coordinate wheel at the
// viewport centre is NOT a dependable substitute (it no-ops on some pages,
// e.g. headless), so the wheel stays opt-in via `--at`/`--frame` for the
// cross-origin-iframe case (issue #36).
interaction::scroll(
&mgr.client,
&session_id,
&state.ref_map,
selector,
None,
dx,
dy,
&state.iframe_sessions,
)
.await?;
Ok(json!({ "scrolled": true }))
Ok(json!({ "scrolled": true, "via": "page" }))
}
/// Viewport center in CSS pixels, used as the default wheel landing point for
/// `scroll` (issue #36). Falls back to a sane 640×400 center if the page can't
/// be evaluated (e.g. a restricted document).
async fn viewport_center(mgr: &BrowserManager, session_id: &str) -> Result<(f64, f64), String> {
let dims = mgr
.client
.send_command_typed::<_, Value>(
"Runtime.evaluate",
&super::cdp::types::EvaluateParams {
expression: "[window.innerWidth, window.innerHeight]".to_string(),
return_by_value: Some(true),
await_promise: Some(false),
},
Some(session_id),
)
.await
.ok();
let arr = dims
.as_ref()
.and_then(|v| v.get("result"))
.and_then(|v| v.get("value"))
.and_then(|v| v.as_array());
let w = arr
.and_then(|a| a.first())
.and_then(|v| v.as_f64())
.filter(|w| *w > 0.0)
.unwrap_or(1280.0);
let h = arr
.and_then(|a| a.get(1))
.and_then(|v| v.as_f64())
.filter(|h| *h > 0.0)
.unwrap_or(800.0);
Ok((w / 2.0, h / 2.0))
}
/// Center of the `n`-th frame (as listed by `chrome-use frames`) in top-viewport
/// CSS pixels, so `scroll --frame n` lands its wheel inside a cross-origin iframe
/// without needing a selector into it (issue #36). Resolves the frame's owning
/// `<iframe>` element box via `DOM.getFrameOwner` + `DOM.getBoxModel` — exact for
/// a frame nested directly under the top document; for a deeper nesting the box is
/// relative to the intermediate frame, so prefer `--at x,y` from a screenshot.
async fn frame_center(
mgr: &BrowserManager,
session_id: &str,
iframe_sessions: &HashMap<String, String>,
n: usize,
) -> Result<(f64, f64), String> {
let frames =
super::element::collect_all_frames_text(&mgr.client, session_id, iframe_sessions).await?;
let frame = frames.get(n).ok_or_else(|| {
format!(
"frame index {} out of range (run `chrome-use frames`: {} frame(s))",
n,
frames.len()
)
})?;
if n == 0 {
// Frame 0 is the top document — there's no owner element; scroll its center.
return viewport_center(mgr, session_id).await;
}
let owner = mgr
.client
.send_command_typed::<_, Value>(
"DOM.getFrameOwner",
&json!({ "frameId": frame.frame_id }),
Some(session_id),
)
.await
.map_err(|e| format!("can't locate frame {}'s owner element: {}", n, e))?;
let backend_node_id = owner
.get("backendNodeId")
.and_then(|v| v.as_i64())
.ok_or_else(|| format!("frame {} has no owner <iframe> element", n))?;
let box_model = mgr
.client
.send_command_typed::<_, Value>(
"DOM.getBoxModel",
&json!({ "backendNodeId": backend_node_id }),
Some(session_id),
)
.await
.map_err(|e| format!("can't measure frame {}'s box: {}", n, e))?;
let content = box_model
.get("model")
.and_then(|m| m.get("content"))
.and_then(|c| c.as_array())
.ok_or_else(|| format!("frame {} box model has no content quad", n))?;
let coord = |i: usize| content.get(i).and_then(|v| v.as_f64()).unwrap_or(0.0);
// content quad is [x1,y1, x2,y2, x3,y3, x4,y4]; opposite corners are 0 and 2.
let cx = (coord(0) + coord(4)) / 2.0;
let cy = (coord(1) + coord(5)) / 2.0;
Ok((cx, cy))
}
/// Dispatch a trusted mouse wheel at `(x, y)`, humanized like `handle_wheel`.
async fn dispatch_wheel(
client: &super::cdp::client::CdpClient,
session_id: &str,
x: f64,
y: f64,
delta_x: f64,
delta_y: f64,
) -> Result<(), String> {
let level = humanize::active_level();
let seed = humanize::next_seed();
for (dx, dy, delay) in humanize::scroll_segments(delta_x, delta_y, level, seed) {
client
.send_command(
"Input.dispatchMouseEvent",
Some(json!({
"type": "mouseWheel",
"x": x,
"y": y,
"deltaX": dx,
"deltaY": dy,
})),
Some(session_id),
)
.await?;
if !delay.is_zero() {
tokio::time::sleep(delay).await;
}
}
Ok(())
}
async fn handle_select(cmd: &Value, state: &mut DaemonState) -> Result<Value, String> {
@@ -3594,6 +4128,56 @@ async fn handle_wait(cmd: &Value, state: &mut DaemonState) -> Result<Value, Stri
async fn handle_gettext(cmd: &Value, state: &mut DaemonState) -> Result<Value, String> {
let mgr = state.browser.as_ref().ok_or("Browser not launched")?;
let session_id = mgr.active_session_id()?.to_string();
// `get text --all-frames` aggregates visible text across every frame the
// session can reach — including out-of-process iframes that never show up
// in the top document (#27: Yahoo/Rakuten/Mercari listing descriptions).
if cmd.get("allFrames").and_then(|v| v.as_bool()) == Some(true) {
let frames = super::element::collect_all_frames_text(
&mgr.client,
&session_id,
&state.iframe_sessions,
)
.await?;
let mut combined = String::new();
let mut frame_count = 0usize;
for f in &frames {
let t = f.text.trim();
if t.is_empty() {
continue;
}
frame_count += 1;
if f.kind != "top" {
combined.push_str(&format!("\n\n----- frame [{}] {} -----\n", f.kind, f.url));
}
combined.push_str(t);
}
let url = mgr.get_url().await.unwrap_or_default();
return Ok(json!({
"text": combined,
"origin": url,
"frames": frame_count,
"allFrames": true,
}));
}
// `get text --pierce` reads text through CLOSED shadow roots and child
// documents via the CDP DOM tree — content `innerText`/`eval` can't see,
// e.g. an extension's injected panel in a closed shadow DOM (#30).
if cmd.get("pierce").and_then(|v| v.as_bool()) == Some(true) {
let text = super::element::get_pierced_text(&mgr.client, &session_id).await?;
let url = mgr.get_url().await.unwrap_or_default();
return Ok(json!({ "text": text, "origin": url, "pierce": true }));
}
// `get text --main` returns the page's main-content region (readability-lite),
// skipping global header/nav/footer/sidebar boilerplate (#27).
if cmd.get("main").and_then(|v| v.as_bool()) == Some(true) {
let text = super::element::get_main_content_text(&mgr.client, &session_id).await?;
let url = mgr.get_url().await.unwrap_or_default();
return Ok(json!({ "text": text, "origin": url, "main": true }));
}
let selector = cmd
.get("selector")
.and_then(|v| v.as_str())
@@ -3611,6 +4195,29 @@ async fn handle_gettext(cmd: &Value, state: &mut DaemonState) -> Result<Value, S
Ok(json!({ "text": text, "origin": url }))
}
async fn handle_frames(_cmd: &Value, state: &mut DaemonState) -> Result<Value, String> {
let mgr = state.browser.as_ref().ok_or("Browser not launched")?;
let session_id = mgr.active_session_id()?.to_string();
let frames =
super::element::collect_all_frames_text(&mgr.client, &session_id, &state.iframe_sessions)
.await?;
let list: Vec<Value> = frames
.iter()
.enumerate()
.map(|(i, f)| {
json!({
"index": i,
"kind": f.kind,
"url": f.url,
"frameId": f.frame_id,
"textLen": f.text.trim().chars().count(),
})
})
.collect();
let url = mgr.get_url().await.unwrap_or_default();
Ok(json!({ "frames": list, "count": list.len(), "origin": url }))
}
async fn handle_getattribute(cmd: &Value, state: &mut DaemonState) -> Result<Value, String> {
let mgr = state.browser.as_ref().ok_or("Browser not launched")?;
let session_id = mgr.active_session_id()?.to_string();
@@ -4045,6 +4652,111 @@ async fn handle_cookies_clear(state: &DaemonState) -> Result<Value, String> {
Ok(json!({ "cleared": true }))
}
// Detect whether the active page is *currently* a Cloudflare challenge
// (the full-page "Just a moment…" / "正在进行安全验证" interstitial), so an agent
// knows whether it must solve or can proceed. Runs in the top frame main world.
const CF_CHALLENGE_JS: &str = r#"(function(){
var t = document.title || '';
var challenged =
/just a moment|attention required|checking (your|if)|verify you are human|||||/i.test(t) ||
!!document.querySelector('#challenge-form, #challenge-running, #cf-challenge-running, [id^="cf-chl"], script[src*="/cdn-cgi/challenge-platform/"]');
var turnstile = !!document.querySelector('.cf-turnstile, [data-sitekey]');
return JSON.stringify({ title: t, challenged: challenged, turnstile: turnstile, readyState: document.readyState });
})()"#;
/// Recommendation for a Cloudflare-gated page, from the current challenge state
/// and whether a still-valid `cf_clearance` exists. Pure so it's unit-testable.
/// - not challenged → "proceed" (the page is cleared/loaded)
/// - challenged, valid cookie → "reissue" (clearance present but page still
/// blocks → it's stale or the IP/UA no longer matches what it was issued for)
/// - challenged, no cookie → "solve"
fn cf_recommendation(challenged: bool, clearance_valid: bool) -> &'static str {
if !challenged {
"proceed"
} else if clearance_valid {
"reissue"
} else {
"solve"
}
}
/// `cf_clearance` validity for a cookie's expiry (epoch seconds; <=0 = session
/// cookie, treated as non-expiring). Returns (present, expired). Pure.
fn clearance_state(expires: Option<f64>, now: f64) -> (bool, bool) {
match expires {
None => (false, false),
Some(e) if e <= 0.0 => (true, false), // session cookie: no expiry
Some(e) => (true, e < now),
}
}
async fn handle_cf_status(_cmd: &Value, state: &mut DaemonState) -> Result<Value, String> {
let mgr = state.browser.as_ref().ok_or("Browser not launched")?;
let session_id = mgr.active_session_id()?.to_string();
let url = mgr.get_url().await.unwrap_or_default();
// 1. Is the page a Cloudflare challenge right now?
let probe_raw = mgr
.evaluate(CF_CHALLENGE_JS, None)
.await
.unwrap_or(Value::Null);
let probe = parse_json_string(probe_raw, "cf challenge probe").unwrap_or(Value::Null);
let challenged = probe
.get("challenged")
.and_then(|v| v.as_bool())
.unwrap_or(false);
let turnstile = probe
.get("turnstile")
.and_then(|v| v.as_bool())
.unwrap_or(false);
let title = probe
.get("title")
.and_then(|v| v.as_str())
.unwrap_or("")
.to_string();
// 2. Persistence artifacts: cf_clearance (HttpOnly → must read via CDP, not
// document.cookie) + CF_VERIFIED_DEVICE. Scope to the current URL.
let urls = if url.is_empty() {
None
} else {
Some(vec![url.clone()])
};
let cookies = super::cookies::get_cookies(&mgr.client, &session_id, urls)
.await
.unwrap_or_default();
let clearance = cookies.iter().find(|c| c.name == "cf_clearance");
let now = std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.map(|d| d.as_secs_f64())
.unwrap_or(0.0);
let (present, expired) = clearance_state(clearance.map(|c| c.expires), now);
let expires_in = clearance
.filter(|_| present && !expired)
.map(|c| (c.expires - now).max(0.0) as i64);
let device_verified = cookies
.iter()
.any(|c| c.name.starts_with("CF_VERIFIED_DEVICE"));
let clearance_valid = present && !expired;
let recommendation = cf_recommendation(challenged, clearance_valid);
Ok(json!({
"url": url,
"title": title,
"challenged": challenged,
"turnstile": turnstile,
"clearance": {
"present": present,
"expired": expired,
"expiresIn": expires_in,
"httpOnly": clearance.map(|c| c.http_only).unwrap_or(false),
},
"deviceVerified": device_verified,
"recommendation": recommendation,
}))
}
async fn handle_storage_get(cmd: &Value, state: &DaemonState) -> Result<Value, String> {
let mgr = state.browser.as_ref().ok_or("Browser not launched")?;
let session_id = mgr.active_session_id()?.to_string();
@@ -4382,8 +5094,18 @@ async fn handle_keyboard(cmd: &Value, state: &DaemonState) -> Result<Value, Stri
.get("text")
.and_then(|v| v.as_str())
.ok_or("Missing 'text' parameter")?;
interaction::type_text_into_active_context(&mgr.client, &session_id, text, None)
.await?;
let key_events = cmd
.get("keyEvents")
.and_then(|v| v.as_bool())
.unwrap_or(false);
interaction::type_text_into_active_context(
&mgr.client,
&session_id,
text,
None,
key_events,
)
.await?;
return Ok(json!({ "typed": text }));
}
Some("insertText") => {
@@ -4493,7 +5215,22 @@ async fn handle_tab_switch(cmd: &Value, state: &mut DaemonState) -> Result<Value
state.ref_map.clear();
state.iframe_sessions.clear();
state.active_frame_id = None;
let result = mgr.tab_switch_by_id(tab_id).await?;
let mut result = mgr.tab_switch_by_id(tab_id).await?;
// Liveness probe: confirm the new session actually answers before we report
// success, so `tab <id>` doesn't print a misleading ✓ for a session that's
// stale and will fail on the very next command (issue #29.3). On the churned
// -tabId case the ext-0.4.9 targetId recovery (#24) self-heals within ~6s, so
// we surface a warning rather than a hard error to avoid a false failure
// during that window.
if mgr.evaluate("1", None).await.is_err() {
if let Some(obj) = result.as_object_mut() {
obj.insert(
"warning".to_string(),
json!("switched tab is not responding yet (session re-attaching); retry the next command"),
);
}
}
// `--activate`: raise this tab to the foreground (the switch made it active;
// bring_to_front acts on the active tab) — for handing a specific tab to the
@@ -4544,6 +5281,19 @@ async fn handle_tab_close(cmd: &Value, state: &mut DaemonState) -> Result<Value,
async fn handle_viewport(cmd: &Value, state: &mut DaemonState) -> Result<Value, String> {
let mgr = state.browser.as_ref().ok_or("Browser not launched")?;
// `viewport reset` clears the device-metrics override and restores the real
// layout viewport (the launched window's size, or — over the relay — the
// user's actual Chrome window).
if cmd.get("reset").and_then(|v| v.as_bool()).unwrap_or(false) {
mgr.clear_viewport().await?;
state.viewport = None;
if let Some(ref server) = state.stream_server {
server.set_viewport(1280, 720).await;
}
return Ok(json!({ "reset": true }));
}
let width = cmd.get("width").and_then(|v| v.as_i64()).unwrap_or(1280) as i32;
let height = cmd.get("height").and_then(|v| v.as_i64()).unwrap_or(720) as i32;
let scale = cmd
@@ -6897,6 +7647,28 @@ async fn handle_drag(cmd: &Value, state: &mut DaemonState) -> Result<Value, Stri
.and_then(|v| v.as_str())
.ok_or("Missing 'target' parameter")?;
// Over the relay (or into an iframe) a coordinate drag drifts to the
// foreground tab and can't reach an OOPIF — DOM-dispatch an HTML5 drag in the
// element's own session instead (issues #31/#36). `coord` mode forces the
// coordinate path for pointer-driven drags (canvas/sliders) on a launched
// browser.
if std::env::var("AGENT_BROWSER_CLICK_MODE").as_deref() != Ok("coord")
&& (crate::connect::relay_url().is_some()
|| state.ref_map.ref_is_in_iframe(source)
|| state.ref_map.ref_is_in_iframe(target))
{
super::interaction::dom_drag(
&mgr.client,
&session_id,
&state.ref_map,
source,
target,
&state.iframe_sessions,
)
.await?;
return Ok(json!({ "dragged": { "source": source, "target": target }, "via": "dom" }));
}
let (sx, sy, _, _, source_session_id) = super::element::resolve_element_center(
&mgr.client,
&session_id,
@@ -7251,6 +8023,7 @@ async fn handle_diff_screenshot(cmd: &Value, state: &DaemonState) -> Result<Valu
quality: None,
annotate: false,
output_dir: None,
clip: None,
};
let result = screenshot::take_screenshot(
@@ -8977,6 +9750,26 @@ mod tests {
use crate::test_utils::EnvGuard;
use std::fs;
#[test]
fn test_cf_recommendation() {
assert_eq!(cf_recommendation(false, false), "proceed");
assert_eq!(cf_recommendation(false, true), "proceed");
assert_eq!(cf_recommendation(true, false), "solve");
assert_eq!(cf_recommendation(true, true), "reissue");
}
#[test]
fn test_clearance_state() {
// no cookie
assert_eq!(clearance_state(None, 1000.0), (false, false));
// session cookie (expires <= 0) → present, never expired
assert_eq!(clearance_state(Some(-1.0), 1000.0), (true, false));
// valid: expiry in the future
assert_eq!(clearance_state(Some(2000.0), 1000.0), (true, false));
// expired: expiry in the past
assert_eq!(clearance_state(Some(500.0), 1000.0), (true, true));
}
#[test]
fn test_url_glob_to_regex() {
assert_eq!(url_glob_to_regex("**/dashboard"), "^.*/dashboard$");
+853 -85
View File
File diff suppressed because it is too large Load Diff
+25 -6
View File
@@ -21,6 +21,16 @@ pub async fn run_daemon(session: &str) {
// (via the ab-connect extension) land in a per-session Chrome tab group.
let _ = super::browser::DAEMON_SESSION.set(session.to_string());
// Bootstrap / refresh the site-adapter pack in the background (first-run +
// periodic TTL). This populates ~/.chrome-use/sites/.index.json so navigation
// can auto-suggest `site` commands for the page you land on, with zero added
// latency to any command. Best-effort; offline is a no-op.
if crate::site::needs_refresh() {
tokio::spawn(async {
let _ = crate::site::update().await;
});
}
let socket_dir = get_daemon_socket_dir();
if !socket_dir.exists() {
let _ = fs::create_dir_all(&socket_dir);
@@ -120,12 +130,21 @@ pub async fn run_daemon(session: &str) {
}
}
// Auto-shutdown the daemon after this many ms of inactivity (no commands received).
// Disabled when unset or 0.
let idle_timeout_ms = env::var("AGENT_BROWSER_IDLE_TIMEOUT_MS")
.ok()
.and_then(|s| s.parse::<u64>().ok())
.filter(|&ms| ms > 0);
// Auto-shutdown the daemon after this many ms of inactivity (no commands
// received). On shutdown the daemon closes the tabs IT created (its per-session
// tab group), so an agent that finishes a task and just stops — without ever
// calling `close` — no longer leaves a pile of scratch tabs and a lingering
// tab group in the user's Chrome. The timer resets on every command, so active
// sessions are never interrupted; only genuinely-idle ones clean up.
//
// Defaults to 10 minutes. Set AGENT_BROWSER_IDLE_TIMEOUT_MS to override, or 0
// to disable (keep the daemon alive forever — the old behaviour). Adopted
// tabs (the user's own, via `adopt`) are never closed: only `created_targets`.
const DEFAULT_IDLE_TIMEOUT_MS: u64 = 600_000;
let idle_timeout_ms = match env::var("AGENT_BROWSER_IDLE_TIMEOUT_MS") {
Ok(s) => s.trim().parse::<u64>().ok().filter(|&ms| ms > 0),
Err(_) => Some(DEFAULT_IDLE_TIMEOUT_MS),
};
let result = run_socket_server(
&socket_path,
+71
View File
@@ -35,6 +35,7 @@ fn native_test_fixture_html(name: &str) -> &'static str {
"html5_drag_probe" => include_str!("test_fixtures/html5_drag_probe.html"),
"pointer_capture_probe" => include_str!("test_fixtures/pointer_capture_probe.html"),
"upload_probe" => include_str!("test_fixtures/upload_probe.html"),
"iframe_button_probe" => include_str!("test_fixtures/iframe_button_probe.html"),
_ => panic!("Unknown native test fixture: {}", name),
}
}
@@ -573,6 +574,76 @@ async fn e2e_snapshot_and_click_ref() {
assert_success(&resp);
}
/// Clicking a button INSIDE an iframe by `@ref` must deliver a TRUSTED activation
/// (`event.isTrusted === true`), not a synthetic DOM `.click()`. Security-sensitive
/// embedded forms (Google Payments' `保存`) reject `isTrusted:false` clicks, so an
/// enabled submit button silently no-op'd (issue #39). The fix routes iframe-ref
/// clicks to a real `Input.dispatchMouseEvent` on the element's own frame session.
/// The fixture's iframe button writes `clicked:<isTrusted>` into its own text on
/// click, which the cross-frame snapshot reads back.
#[tokio::test]
#[ignore]
async fn e2e_iframe_button_click_is_trusted() {
let mut state = DaemonState::new();
let resp = execute_command(
&json!({ "id": "1", "action": "launch", "headless": true }),
&mut state,
)
.await;
assert_success(&resp);
let resp = execute_command(
&json!({ "id": "2", "action": "navigate", "url": native_test_fixture_url("iframe_button_probe") }),
&mut state,
)
.await;
assert_success(&resp);
// Snapshot (interactive) — the button lives in the iframe and must appear with
// a ref; that ref carries the frame_id so the click resolves into the frame.
let resp = execute_command(
&json!({ "id": "3", "action": "snapshot", "interactive": true }),
&mut state,
)
.await;
assert_success(&resp);
let snapshot = get_data(&resp)["snapshot"].as_str().unwrap_or("");
let ref_id = snapshot
.lines()
.find(|l| l.contains("button \"save\""))
.and_then(|l| l.split("ref=").nth(1))
.map(|r| r.trim_end_matches(']').trim())
.unwrap_or_else(|| panic!("iframe button not found in snapshot:\n{snapshot}"));
// Click it by ref.
let resp = execute_command(
&json!({ "id": "4", "action": "click", "selector": ref_id }),
&mut state,
)
.await;
assert_success(&resp);
tokio::time::sleep(tokio::time::Duration::from_millis(300)).await;
// The button rewrote its own text with the click's isTrusted flag; read it
// back across frames.
let resp = execute_command(
&json!({ "id": "5", "action": "snapshot", "interactive": true }),
&mut state,
)
.await;
assert_success(&resp);
let after = get_data(&resp)["snapshot"].as_str().unwrap_or("");
assert!(
after.contains("clicked:true"),
"iframe button click must be trusted (isTrusted:true); snapshot:\n{after}"
);
let resp = execute_command(&json!({ "id": "99", "action": "close" }), &mut state).await;
assert_success(&resp);
}
// ---------------------------------------------------------------------------
// Screenshot
// ---------------------------------------------------------------------------
+326 -4
View File
@@ -100,6 +100,15 @@ impl RefMap {
self.map.get(ref_id)
}
/// Whether `selector_or_ref` is a `@ref` whose snapshot entry lives inside an
/// iframe (has a `frame_id`). Pointer interactions use this to choose
/// DOM-dispatch over coordinates for OOPIF elements (issue #36).
pub fn ref_is_in_iframe(&self, selector_or_ref: &str) -> bool {
parse_ref(selector_or_ref)
.and_then(|r| self.map.get(&r).map(|e| e.frame_id.is_some()))
.unwrap_or(false)
}
pub fn entries_sorted(&self) -> Vec<(String, RefEntry)> {
let mut entries = self
.map
@@ -975,6 +984,268 @@ pub async fn get_element_text(
.unwrap_or_default())
}
/// Text content collected from a single frame of the page.
#[derive(Debug, Clone)]
pub struct FrameText {
pub frame_id: String,
pub url: String,
/// "top" | "inline" (same-process child frame) | "oopif" (out-of-process).
pub kind: &'static str,
pub text: String,
}
// The expression we run in every frame to read its visible text. innerText
// honors CSS visibility (skips display:none), textContent is the fallback.
const FRAME_INNERTEXT_JS: &str = "(function(){try{var b=document.body||document.documentElement;return b?(b.innerText||b.textContent||''):'';}catch(e){return '';}})()";
async fn eval_text_default(client: &CdpClient, session_id: &str) -> String {
let res = client
.send_command(
"Runtime.evaluate",
Some(serde_json::json!({
"expression": FRAME_INNERTEXT_JS,
"returnByValue": true,
})),
Some(session_id),
)
.await;
res.ok()
.and_then(|v| v.get("result").and_then(|r| r.get("value")).cloned())
.and_then(|v| v.as_str().map(|s| s.to_string()))
.unwrap_or_default()
}
// Same-process child frames share the top renderer but live in their own
// execution context. Page.createIsolatedWorld hands us a context id bound to
// that frame so Runtime.evaluate reads the child document, not the parent.
async fn eval_text_in_frame(client: &CdpClient, session_id: &str, frame_id: &str) -> String {
let ctx = client
.send_command(
"Page.createIsolatedWorld",
Some(serde_json::json!({ "frameId": frame_id, "worldName": "chrome_use_text" })),
Some(session_id),
)
.await
.ok()
.and_then(|v| v.get("executionContextId").and_then(|c| c.as_i64()));
let Some(ctx_id) = ctx else {
return String::new();
};
let res = client
.send_command(
"Runtime.evaluate",
Some(serde_json::json!({
"expression": FRAME_INNERTEXT_JS,
"returnByValue": true,
"contextId": ctx_id,
})),
Some(session_id),
)
.await;
res.ok()
.and_then(|v| v.get("result").and_then(|r| r.get("value")).cloned())
.and_then(|v| v.as_str().map(|s| s.to_string()))
.unwrap_or_default()
}
fn flatten_frame_tree(node: &Value, is_top: bool, out: &mut Vec<(String, String, bool)>) {
if let Some(frame) = node.get("frame") {
if let Some(id) = frame.get("id").and_then(|v| v.as_str()) {
let url = frame
.get("url")
.and_then(|v| v.as_str())
.unwrap_or("")
.to_string();
out.push((id.to_string(), url, is_top));
}
}
if let Some(children) = node.get("childFrames").and_then(|v| v.as_array()) {
for child in children {
flatten_frame_tree(child, false, out);
}
}
}
/// Collect visible text from every frame reachable in the active session,
/// including out-of-process iframes (which never appear in the top frame's
/// `Page.getFrameTree` and so are invisible to `document.body.innerText`).
///
/// Same-process child frames are read through `Page.createIsolatedWorld`;
/// OOPIFs are read through their own auto-attached debugger session
/// (`iframe_sessions`, keyed by frameId == targetId). This is the engine
/// behind `get text --all-frames` and `chrome-use frames` — the fix for
/// listing/marketplace pages whose description lives in a child frame (#27).
pub async fn collect_all_frames_text(
client: &CdpClient,
top_session: &str,
iframe_sessions: &HashMap<String, String>,
) -> Result<Vec<FrameText>, String> {
let mut out: Vec<FrameText> = Vec::new();
let mut seen: std::collections::HashSet<String> = std::collections::HashSet::new();
// 1. Top session: the top frame plus its same-process descendants. OOPIF
// frames that happen to surface here are skipped — they're read via
// their dedicated session in step 2 (cross-process isolated worlds fail).
let tree = client
.send_command_no_params("Page.getFrameTree", Some(top_session))
.await?;
let mut frames: Vec<(String, String, bool)> = Vec::new();
flatten_frame_tree(&tree["frameTree"], true, &mut frames);
for (fid, url, is_top) in frames {
if iframe_sessions.contains_key(&fid) {
continue;
}
if !seen.insert(fid.clone()) {
continue;
}
let (kind, text) = if is_top {
("top", eval_text_default(client, top_session).await)
} else {
(
"inline",
eval_text_in_frame(client, top_session, &fid).await,
)
};
out.push(FrameText {
frame_id: fid,
url,
kind,
text,
});
}
// 2. Each out-of-process iframe, read through its own session.
for (fid, sid) in iframe_sessions {
if !seen.insert(fid.clone()) {
continue;
}
let url = client
.send_command_no_params("Page.getFrameTree", Some(sid))
.await
.ok()
.and_then(|t| {
t.get("frameTree")
.and_then(|ft| ft.get("frame"))
.and_then(|f| f.get("url"))
.and_then(|v| v.as_str())
.map(|s| s.to_string())
})
.unwrap_or_default();
let text = eval_text_default(client, sid).await;
out.push(FrameText {
frame_id: fid.clone(),
url,
kind: "oopif",
text,
});
}
Ok(out)
}
// Readability-lite: prefer the page's semantic main-content region over the
// whole body so global header/nav/footer chrome (and, on many listing pages,
// the "related items" sidebar) doesn't drown out the actual content. Runs on
// the live, rendered tree (innerText needs layout — a detached clone returns
// empty), so we pick the densest <main>/<article> region rather than cloning
// and stripping. Falls back to <body> when no substantial main region exists.
const MAIN_CONTENT_JS: &str = r#"(function(){
function txt(el){try{return (el.innerText||'').trim();}catch(e){return '';}}
var sels=['main','[role=main]','article','#main','#contents','#l-content'];
var best=null,bestLen=0;
for(var i=0;i<sels.length;i++){
var els=document.querySelectorAll(sels[i]);
for(var j=0;j<els.length;j++){var l=txt(els[j]).length;if(l>bestLen){bestLen=l;best=els[j];}}
}
if(best&&bestLen>200)return txt(best);
return txt(document.body);
})()"#;
/// Extract the page's main-content text (readability-lite), preferring a
/// semantic `<main>`/`<article>` region over the full body. Used by
/// `get text --main` to avoid header/nav/sidebar boilerplate (#27).
pub async fn get_main_content_text(client: &CdpClient, session_id: &str) -> Result<String, String> {
let res = client
.send_command(
"Runtime.evaluate",
Some(serde_json::json!({
"expression": MAIN_CONTENT_JS,
"returnByValue": true,
})),
Some(session_id),
)
.await?;
Ok(res
.get("result")
.and_then(|r| r.get("value"))
.and_then(|v| v.as_str())
.unwrap_or_default()
.to_string())
}
// Text nodes whose parent is one of these carry no visible content.
fn is_noise_tag(name: &str) -> bool {
matches!(name, "SCRIPT" | "STYLE" | "NOSCRIPT" | "TEMPLATE" | "HEAD")
}
// Walk a CDP DOM.Node tree, collecting text-node values. Unlike `innerText`
// (JS, blocked by CLOSED shadow roots), the CDP DOM tree from
// `DOM.getDocument(pierce:true)` includes closed shadow roots and child
// documents — so this reaches text JS can't. `parent_noise` carries whether an
// ancestor was <script>/<style>/etc so their text is skipped.
fn collect_dom_text(node: &Value, parent_noise: bool, out: &mut String) {
let node_type = node.get("nodeType").and_then(|v| v.as_i64()).unwrap_or(0);
let node_name = node.get("nodeName").and_then(|v| v.as_str()).unwrap_or("");
if node_type == 3 {
if !parent_noise {
if let Some(t) = node.get("nodeValue").and_then(|v| v.as_str()) {
let t = t.trim();
if !t.is_empty() {
if !out.is_empty() {
out.push(' ');
}
out.push_str(t);
}
}
}
return;
}
let noise = parent_noise || is_noise_tag(node_name);
if let Some(children) = node.get("children").and_then(|v| v.as_array()) {
for child in children {
collect_dom_text(child, noise, out);
}
}
if let Some(shadow) = node.get("shadowRoots").and_then(|v| v.as_array()) {
for sr in shadow {
collect_dom_text(sr, noise, out);
}
}
if let Some(doc) = node.get("contentDocument") {
collect_dom_text(doc, noise, out);
}
}
/// Extract text from the page via the CDP DOM tree with `pierce:true`, which
/// reaches into CLOSED shadow roots and child documents that `innerText`/`eval`
/// cannot. Lets an agent read content rendered into a closed shadow DOM (e.g. an
/// extension's injected debug panel) without any extra Chrome permission — it
/// rides the per-tab debugger session that's already attached (#30).
pub async fn get_pierced_text(client: &CdpClient, session_id: &str) -> Result<String, String> {
let doc = client
.send_command(
"DOM.getDocument",
Some(serde_json::json!({ "depth": -1, "pierce": true })),
Some(session_id),
)
.await?;
let mut out = String::new();
if let Some(root) = doc.get("root") {
collect_dom_text(root, false, &mut out);
}
Ok(out)
}
pub async fn get_element_attribute(
client: &CdpClient,
session_id: &str,
@@ -1258,9 +1529,27 @@ pub async fn get_element_input_value(
.send_command_typed(
"Runtime.callFunctionOn",
&CallFunctionOnParams {
function_declaration:
"function() { return typeof this.value === 'string' ? this.value : ''; }"
.to_string(),
// Read rich-editor content too (issue #41): CodeMirror 5 / Monaco
// keep their text in a model, not `.value`; contenteditable keeps
// it as innerText. Falls back to `.value` for plain inputs.
function_declaration: r#"function() {
const el = this;
const cm5 = el.closest && el.closest('.CodeMirror');
if (cm5 && cm5.CodeMirror) return cm5.CodeMirror.getValue();
if (window.monaco && monaco.editor) {
try {
const eds = monaco.editor.getEditors ? monaco.editor.getEditors() : [];
const ed = eds.find(e => e.getDomNode && e.getDomNode().contains(el)) || eds[0];
if (ed) return ed.getValue();
const m = monaco.editor.getModels ? monaco.editor.getModels() : [];
if (m[0]) return m[0].getValue();
} catch (e) {}
}
if (typeof el.value === 'string') return el.value;
if (el.isContentEditable) return el.innerText;
return '';
}"#
.to_string(),
object_id: Some(object_id),
arguments: None,
return_by_value: Some(true),
@@ -1338,7 +1627,15 @@ pub async fn get_element_bounding_box(
&CallFunctionOnParams {
function_declaration: r#"function() {
const r = this.getBoundingClientRect();
return { x: r.x, y: r.y, width: r.width, height: r.height };
const inViewport = r.bottom > 0 && r.right > 0
&& r.top < (innerHeight || document.documentElement.clientHeight)
&& r.left < (innerWidth || document.documentElement.clientWidth);
return {
x: r.x, y: r.y, width: r.width, height: r.height,
centerX: Math.round(r.x + r.width / 2),
centerY: Math.round(r.y + r.height / 2),
inViewport,
};
}"#
.to_string(),
object_id: Some(object_id),
@@ -1448,6 +1745,31 @@ mod tests {
assert_eq!(parse_ref("@e123"), Some("e123".to_string()));
}
#[test]
fn test_collect_dom_text_pierces_closed_shadow_and_skips_noise() {
// A CDP DOM.Node tree: a host element whose CLOSED shadow root holds the
// text, plus a <script> whose text must be skipped.
let tree = serde_json::json!({
"nodeType": 1, "nodeName": "BODY",
"children": [
{ "nodeType": 1, "nodeName": "SCRIPT",
"children": [ { "nodeType": 3, "nodeName": "#text", "nodeValue": "var secret=1;" } ] },
{ "nodeType": 1, "nodeName": "DIV",
"shadowRoots": [
{ "nodeType": 11, "nodeName": "#document-fragment",
"children": [
{ "nodeType": 1, "nodeName": "SPAN",
"children": [ { "nodeType": 3, "nodeName": "#text", "nodeValue": "DECRYPTED 42" } ] }
] }
] }
]
});
let mut out = String::new();
collect_dom_text(&tree, false, &mut out);
assert_eq!(out, "DECRYPTED 42");
assert!(!out.contains("secret"), "script text must be skipped");
}
#[test]
fn test_parse_ref_equals_prefix() {
assert_eq!(parse_ref("ref=e1"), Some("e1".to_string()));
+378 -18
View File
@@ -7,6 +7,17 @@ use super::cdp::types::*;
use super::element::{parse_ref, resolve_element_center, resolve_element_object_id, RefMap};
use super::humanize;
/// Whether a pointer interaction should be DOM-dispatched (invoke the event on
/// the element in its own session) rather than dispatched at a viewport
/// coordinate via `Input.dispatchMouseEvent`. True when the target is inside an
/// iframe (an OOPIF element's box can't be mapped to a top-viewport point) or we
/// drive over the extension relay (a coordinate Input event isn't confined to the
/// target tab on a busy real Chrome — it drifts onto the foreground tab; issues
/// #31/#36). DOM-dispatch always hits the right element in the right tab.
fn prefer_dom_dispatch(ref_map: &RefMap, selector_or_ref: &str) -> bool {
ref_map.ref_is_in_iframe(selector_or_ref) || crate::connect::relay_url().is_some()
}
pub async fn click(
client: &CdpClient,
session_id: &str,
@@ -45,6 +56,45 @@ pub async fn click(
.await;
}
// An element INSIDE an iframe needs a TRUSTED activation: a DOM `.click()` is
// `isTrusted:false`, which security-sensitive embedded forms reject — Google
// Payments' enabled `保存` button silently no-ops on a synthetic click (issue
// #39). A coordinate `Input.dispatchMouseEvent` can't help either: `getBoxModel`
// for a sub-frame node returns frame-local coordinates that don't compose the
// iframe's offset, so the click lands in the wrong place. The frame-agnostic
// trusted path is keyboard activation — focus the element in its own frame, then
// dispatch a real Enter on the page session; Chrome routes the key to the
// focused element regardless of frame (same as `type --focused`), and Enter on a
// focused button/link fires a trusted `click`. `coord` mode opts out.
let in_iframe = ref_map.ref_is_in_iframe(selector_or_ref);
if mode != "coord" && button == "left" && click_count == 1 && in_iframe {
return dom_activate(
client,
session_id,
ref_map,
selector_or_ref,
iframe_sessions,
)
.await;
}
// On the relay (the user's real Chrome) a TOP-document coordinate click used to
// drift onto the foreground tab; that root cause is fixed (#5: the agent drives
// its own pinned tab), but DOM-dispatch stays the conservative default here.
if mode != "coord"
&& button == "left"
&& click_count == 1
&& crate::connect::relay_url().is_some()
{
return dom_click(
client,
session_id,
ref_map,
selector_or_ref,
iframe_sessions,
)
.await;
}
let resolved = resolve_element_center(
client,
session_id,
@@ -235,6 +285,112 @@ async fn dom_click(
Ok(())
}
/// Trusted activation of an element inside an iframe (issue #39). Focuses the
/// element in its own frame session, then dispatches a real Enter/Space on the
/// page session — Chrome routes the key to the focused element across frames, and
/// Enter/Space on a focused button/link/checkbox fires a `click` with
/// `isTrusted: true`, which security-sensitive embedded forms (Google Payments
/// `保存`) require. Non-activatable roles (a `div[onclick]`) can't be keyboard-
/// activated, so they fall back to a DOM `.click()`.
async fn dom_activate(
client: &CdpClient,
session_id: &str,
ref_map: &RefMap,
selector_or_ref: &str,
iframe_sessions: &HashMap<String, String>,
) -> Result<(), String> {
let role = parse_ref(selector_or_ref)
.and_then(|r| ref_map.get(&r).map(|e| e.role.clone()))
.unwrap_or_default();
// Space toggles checkbox-like controls; Enter activates buttons/links/menus.
let key = match role.as_str() {
"checkbox" | "radio" | "switch" | "option" | "menuitemcheckbox" | "menuitemradio" => {
Some("space")
}
"button" | "link" | "menuitem" | "tab" | "treeitem" => Some("enter"),
_ => None,
};
let Some(key) = key else {
// Not keyboard-activatable — best effort via DOM .click() (untrusted).
return dom_click(
client,
session_id,
ref_map,
selector_or_ref,
iframe_sessions,
)
.await;
};
let (object_id, effective_session_id) = resolve_element_object_id(
client,
session_id,
ref_map,
selector_or_ref,
iframe_sessions,
)
.await?;
// Focus the element in its OWN frame session so the keystroke lands on it.
client
.send_command_typed::<_, Value>(
"Runtime.callFunctionOn",
&CallFunctionOnParams {
function_declaration: "function() { this.focus(); }".to_string(),
object_id: Some(object_id),
arguments: None,
return_by_value: Some(true),
await_promise: Some(false),
},
Some(&effective_session_id),
)
.await?;
// Trusted key on the page session — routed to the focused (in-frame) element.
press_key(client, session_id, key).await?;
wait_for_paint_settled(client, &effective_session_id).await;
Ok(())
}
/// DOM-dispatch a double-click on the element in its own session (no coordinates)
/// — the relay/iframe-safe counterpart to a coordinate dblclick. Fires the full
/// click,click,dblclick sequence so handlers bound to any of them respond.
async fn dom_dblclick(
client: &CdpClient,
session_id: &str,
ref_map: &RefMap,
selector_or_ref: &str,
iframe_sessions: &HashMap<String, String>,
) -> Result<(), String> {
let (object_id, effective_session_id) = resolve_element_object_id(
client,
session_id,
ref_map,
selector_or_ref,
iframe_sessions,
)
.await?;
client
.send_command_typed::<_, Value>(
"Runtime.callFunctionOn",
&CallFunctionOnParams {
function_declaration: r#"function() {
const opts = { bubbles: true, cancelable: true, view: window };
this.dispatchEvent(new MouseEvent('click', opts));
this.dispatchEvent(new MouseEvent('click', { ...opts, detail: 2 }));
this.dispatchEvent(new MouseEvent('dblclick', opts));
}"#
.to_string(),
object_id: Some(object_id),
arguments: None,
return_by_value: Some(true),
await_promise: Some(false),
},
Some(&effective_session_id),
)
.await?;
wait_for_paint_settled(client, &effective_session_id).await;
Ok(())
}
pub async fn dblclick(
client: &CdpClient,
session_id: &str,
@@ -242,6 +398,20 @@ pub async fn dblclick(
selector_or_ref: &str,
iframe_sessions: &HashMap<String, String>,
) -> Result<(), String> {
// Same relay/iframe drift hazard as a single click — DOM-dispatch the
// double-click there instead of a coordinate one (issues #31/#36).
if std::env::var("AGENT_BROWSER_CLICK_MODE").as_deref() != Ok("coord")
&& prefer_dom_dispatch(ref_map, selector_or_ref)
{
return dom_dblclick(
client,
session_id,
ref_map,
selector_or_ref,
iframe_sessions,
)
.await;
}
click(
client,
session_id,
@@ -254,6 +424,50 @@ pub async fn dblclick(
.await
}
/// DOM-dispatch a hover (pointer/mouse enter+move) on the element in its own
/// session — reaches OOPIF elements and never drifts to the foreground tab over
/// the relay, unlike a coordinate `mouseMoved` (issues #31/#36).
async fn dom_hover(
client: &CdpClient,
session_id: &str,
ref_map: &RefMap,
selector_or_ref: &str,
iframe_sessions: &HashMap<String, String>,
) -> Result<(), String> {
let (object_id, effective_session_id) = resolve_element_object_id(
client,
session_id,
ref_map,
selector_or_ref,
iframe_sessions,
)
.await?;
client
.send_command_typed::<_, Value>(
"Runtime.callFunctionOn",
&CallFunctionOnParams {
function_declaration: r#"function() {
const r = this.getBoundingClientRect();
const cx = r.left + r.width / 2, cy = r.top + r.height / 2;
const base = { bubbles: true, cancelable: true, view: window, clientX: cx, clientY: cy };
this.dispatchEvent(new PointerEvent('pointerover', base));
this.dispatchEvent(new PointerEvent('pointerenter', { ...base, bubbles: false }));
this.dispatchEvent(new MouseEvent('mouseover', base));
this.dispatchEvent(new MouseEvent('mouseenter', { ...base, bubbles: false }));
this.dispatchEvent(new MouseEvent('mousemove', base));
}"#
.to_string(),
object_id: Some(object_id),
arguments: None,
return_by_value: Some(true),
await_promise: Some(false),
},
Some(&effective_session_id),
)
.await?;
Ok(())
}
pub async fn hover(
client: &CdpClient,
session_id: &str,
@@ -261,6 +475,18 @@ pub async fn hover(
selector_or_ref: &str,
iframe_sessions: &HashMap<String, String>,
) -> Result<(), String> {
// Coordinate `mouseMoved` drifts to the foreground tab over the relay and
// can't reach an OOPIF — DOM-dispatch the hover there (issues #31/#36).
if prefer_dom_dispatch(ref_map, selector_or_ref) {
return dom_hover(
client,
session_id,
ref_map,
selector_or_ref,
iframe_sessions,
)
.await;
}
let (x, y, _w, _h, effective_session_id) = resolve_element_center(
client,
session_id,
@@ -289,6 +515,63 @@ pub async fn hover(
Ok(())
}
/// DOM-dispatch an HTML5 drag-and-drop from `source` to `target` in their shared
/// session — the relay/iframe-safe counterpart to the coordinate drag, which
/// drifts to the foreground tab over the relay and can't reach an OOPIF (issues
/// #31/#36). Covers HTML5 DnD (sortable lists, file/card boards); pointer-driven
/// drag (canvas, sliders) still needs the coordinate path. Errors if source and
/// target live in different frames — a synthetic cross-frame DnD isn't reliable.
pub async fn dom_drag(
client: &CdpClient,
session_id: &str,
ref_map: &RefMap,
source: &str,
target: &str,
iframe_sessions: &HashMap<String, String>,
) -> Result<(), String> {
let (src_obj, src_session) =
resolve_element_object_id(client, session_id, ref_map, source, iframe_sessions).await?;
let (tgt_obj, tgt_session) =
resolve_element_object_id(client, session_id, ref_map, target, iframe_sessions).await?;
if src_session != tgt_session {
return Err(
"drag source and target are in different frames; cross-frame drag-and-drop over the \
relay isn't supported drag within a single frame, or use a launched browser with \
AGENT_BROWSER_CLICK_MODE=coord"
.to_string(),
);
}
client
.send_command_typed::<_, Value>(
"Runtime.callFunctionOn",
&CallFunctionOnParams {
function_declaration: r#"function(target) {
const dt = new DataTransfer();
const ev = (type, el) => el.dispatchEvent(
new DragEvent(type, { bubbles: true, cancelable: true, dataTransfer: dt }));
ev('dragstart', this);
ev('drag', this);
ev('dragenter', target);
ev('dragover', target);
ev('drop', target);
ev('dragend', this);
}"#
.to_string(),
object_id: Some(src_obj),
arguments: Some(vec![CallArgument {
value: None,
object_id: Some(tgt_obj),
}]),
return_by_value: Some(true),
await_promise: Some(false),
},
Some(&src_session),
)
.await?;
wait_for_paint_settled(client, &src_session).await;
Ok(())
}
pub async fn fill(
client: &CdpClient,
session_id: &str,
@@ -296,7 +579,7 @@ pub async fn fill(
selector_or_ref: &str,
value: &str,
iframe_sessions: &HashMap<String, String>,
) -> Result<(), String> {
) -> Result<String, String> {
let (object_id, effective_session_id) = resolve_element_object_id(
client,
session_id,
@@ -307,14 +590,15 @@ pub async fn fill(
.await?;
// Emulate a real edit so framework-controlled inputs (React/Vue) and
// site-side listeners actually see the change (issue #25): the old path set
// `this.value` directly and used Input.insertText, which left React's
// internal value-tracker out of sync and never fired change/blur — so
// dependent logic (e.g. Mercari's postal-code → 都道府県 autocomplete) never
// ran even though the value was visible. Set the value through the element's
// PROTOTYPE setter (which React's _valueTracker hooks), then dispatch
// input → change → blur/focusout. `type <sel> <text>` remains for sites that
// need per-keystroke events.
// site-side listeners actually see the change (issue #25): set the value
// through the element's PROTOTYPE setter (which React's _valueTracker hooks),
// then dispatch input → changeblur/focusout. Beyond plain inputs, detect
// rich editors and use their own API/events (issue #41): CodeMirror 5 and
// Monaco have a model that `.value`/`textContent` can't touch; ProseMirror /
// contenteditable need `execCommand('insertText')` so beforeinput/input fire
// (a raw `textContent =` corrupts PM's doc and skips React composers).
// Returns the engine used so the caller can report it. `type <sel> <text>`
// remains for sites that need per-keystroke events.
let fill_js = format!(
r#"function() {{
const el = this;
@@ -322,13 +606,43 @@ pub async fn fill(
try {{ el.focus(); }} catch (e) {{}}
const tag = el.tagName;
const fire = (type, ctor) => el.dispatchEvent(new (ctor || Event)(type, {{ bubbles: true }}));
if (tag === 'SELECT') {{
el.value = v; fire('input'); fire('change'); return true;
// CodeMirror 5: a hidden <textarea> inside .CodeMirror with a live instance.
const cm5 = el.closest && el.closest('.CodeMirror');
if (cm5 && cm5.CodeMirror) {{ cm5.CodeMirror.setValue(v); return 'codemirror5'; }}
// Monaco: global `monaco`; prefer the editor whose DOM contains el.
if (window.monaco && monaco.editor) {{
try {{
const eds = monaco.editor.getEditors ? monaco.editor.getEditors() : [];
const ed = eds.find(e => e.getDomNode && e.getDomNode().contains(el)) || eds[0];
if (ed) {{ ed.setValue(v); return 'monaco'; }}
const models = monaco.editor.getModels ? monaco.editor.getModels() : [];
if (models[0]) {{ models[0].setValue(v); return 'monaco'; }}
}} catch (e) {{}}
}}
if (tag === 'SELECT') {{ el.value = v; fire('input'); fire('change'); return 'select'; }}
if (el.isContentEditable) {{
el.textContent = v; fire('input', window.InputEvent || Event); fire('change');
try {{ el.blur(); }} catch (e) {{}} fire('focusout'); return true;
// ProseMirror / contenteditable: select-all then insertText fires
// beforeinput/input that PM and React composers listen for.
let ok = false;
try {{
const sel = window.getSelection();
const range = document.createRange();
range.selectNodeContents(el);
sel.removeAllRanges();
sel.addRange(range);
ok = document.execCommand('insertText', false, v);
}} catch (e) {{}}
if (!ok) {{ el.textContent = v; fire('input', window.InputEvent || Event); }}
fire('change');
try {{ el.blur(); }} catch (e) {{}}
fire('focusout');
return ok ? 'contenteditable' : 'contenteditable-fallback';
}}
const proto = tag === 'TEXTAREA' ? window.HTMLTextAreaElement.prototype
: window.HTMLInputElement.prototype;
const desc = Object.getOwnPropertyDescriptor(proto, 'value');
@@ -340,13 +654,13 @@ pub async fn fill(
fire('change');
try {{ el.blur(); }} catch (e) {{}}
fire('focusout'); // blur-triggered lookups/validation
return true;
return 'input';
}}"#,
val = serde_json::to_string(value).unwrap_or_default()
);
client
.send_command_typed::<_, Value>(
let result: EvaluateResult = client
.send_command_typed(
"Runtime.callFunctionOn",
&CallFunctionOnParams {
function_declaration: fill_js,
@@ -359,7 +673,11 @@ pub async fn fill(
)
.await?;
Ok(())
Ok(result
.result
.value
.and_then(|v| v.as_str().map(String::from))
.unwrap_or_else(|| "input".to_string()))
}
#[allow(clippy::too_many_arguments)]
@@ -372,6 +690,7 @@ pub async fn type_text(
clear: bool,
delay_ms: Option<u64>,
iframe_sessions: &HashMap<String, String>,
key_events: bool,
) -> Result<(), String> {
let (object_id, effective_session_id) = resolve_element_object_id(
client,
@@ -418,7 +737,7 @@ pub async fn type_text(
.await?;
}
type_text_into_active_context(client, session_id, text, delay_ms).await
type_text_into_active_context(client, session_id, text, delay_ms, key_events).await
}
pub async fn type_text_into_active_context(
@@ -426,6 +745,7 @@ pub async fn type_text_into_active_context(
session_id: &str,
text: &str,
delay_ms: Option<u64>,
key_events: bool,
) -> Result<(), String> {
// Per-character timing: an explicit `delay_ms` wins (caller asked for a
// fixed cadence); otherwise fall back to humanize — variable, human-like
@@ -475,6 +795,46 @@ pub async fn type_text_into_active_context(
Some(session_id),
)
.await?;
} else if key_events {
// Real keystrokes (keyDown+keyUp carrying `text`) for autocomplete /
// combobox widgets that only react to key events and ignore the
// `input` that `Input.insertText` fires — e.g. Google's address
// postal-code → city/prefecture lookup (issue #36 / #4). The keyDown's
// `text` still inserts the character, so the field also fills.
let (key, code, key_code) = char_to_key_info(ch);
let s = ch.to_string();
client
.send_command_typed::<_, Value>(
"Input.dispatchKeyEvent",
&DispatchKeyEventParams {
event_type: "keyDown".to_string(),
key: Some(key.clone()),
code: Some(code.clone()),
text: Some(s.clone()),
unmodified_text: Some(s),
windows_virtual_key_code: Some(key_code),
native_virtual_key_code: Some(key_code),
modifiers: None,
},
Some(session_id),
)
.await?;
client
.send_command_typed::<_, Value>(
"Input.dispatchKeyEvent",
&DispatchKeyEventParams {
event_type: "keyUp".to_string(),
key: Some(key),
code: Some(code),
text: None,
unmodified_text: None,
windows_virtual_key_code: Some(key_code),
native_virtual_key_code: Some(key_code),
modifiers: None,
},
Some(session_id),
)
.await?;
} else {
// VS Code/Electron webviews reject repeated dispatchKeyEvent calls
// carrying printable `text`. Insert printable characters directly
+273 -4
View File
@@ -52,6 +52,22 @@ pub struct RelayState {
pending: HashMap<i64, (ClientId, Value)>,
/// monotonic source of relay-global command ids
next_global_id: i64,
/// Group-scoped isolation (issue #40). A tab group belongs to exactly one
/// agent/session; the relay scopes `Target.getTargets` per client to its own
/// group so the daemon can safely adopt new tabs (follow-popup, cross-session
/// adopt) without ever seeing the user's or another agent's tabs.
///
/// clientId -> group name. A client that never announced a group (older
/// daemon) is absent here and gets the full, UNSCOPED target list — so this
/// is fully backward-compatible.
client_groups: HashMap<ClientId, String>,
/// targetId -> group name. Created tabs are tagged from `Target.createTarget`'s
/// `agentGroup`; an explicitly adopted tab is tagged to the adopter; a pop-up
/// inherits its opener's group (needs the extension to report `openerTargetId`).
target_group: HashMap<String, String>,
/// relay-global id of an in-flight `Target.createTarget` -> the `agentGroup`
/// it carried, so the reply's `targetId` can be tagged with that group.
pending_create: HashMap<i64, String>,
}
/// What to do with a raw CDP command received from a `CdpClient`.
@@ -95,6 +111,7 @@ impl RelayState {
/// `pending` entries don't leak.
pub fn drop_client(&mut self, client_id: ClientId) {
self.pending.retain(|_, (cid, _)| *cid != client_id);
self.client_groups.remove(&client_id);
}
/// Route a raw CDP command `{id, method, params?, sessionId?}` from a
@@ -123,12 +140,28 @@ impl RelayState {
"jsVersion": ""
}
})),
// Non-CDP control message: a daemon announces which tab group
// (session) it owns, so getTargets can be scoped to it (issue #40).
"ABRelay.setGroup" => {
if let Some(g) = params.get("group").and_then(|g| g.as_str()) {
if !g.is_empty() {
self.client_groups.insert(client_id, g.to_string());
}
}
ClientRoute::Local(json!({ "id": id, "result": {} }))
}
// Discovery is best-effort and event-driven in real CDP; abs only
// reads the getTargets result, so an empty ack is enough here.
"Target.setDiscoverTargets" | "Target.setAutoAttach" => {
ClientRoute::Local(json!({ "id": id, "result": {} }))
}
"Target.getTargets" => {
// Unscoped discovery for EXPLICIT cross-group adoption (`chrome-use
// adopt`): returns every target the extension has attached, ignoring
// group scoping, so an agent can find a specific pre-existing tab (the
// user's, another session's) by URL/targetId and adopt it. Isolation
// is preserved because the daemon only acts on the one tab it then
// attaches (which the relay re-tags into the adopter's group).
"ABRelay.getAllTargets" => {
let infos: Vec<Value> = self
.targets
.values()
@@ -136,15 +169,44 @@ impl RelayState {
.collect();
ClientRoute::Local(json!({ "id": id, "result": { "targetInfos": infos } }))
}
"Target.getTargets" => {
// Scope to the client's own group when it announced one; an
// un-announced (legacy) client gets the full list (back-compat).
let scoped = self.client_groups.get(&client_id).cloned();
let infos: Vec<Value> = self
.targets
.iter()
.filter(|(tid, _)| match &scoped {
Some(g) => self
.target_group
.get(*tid)
.map(|tg| tg == g)
.unwrap_or(false),
None => true,
})
.map(|(_, t)| t.target_info.clone())
.collect();
ClientRoute::Local(json!({ "id": id, "result": { "targetInfos": infos } }))
}
"Target.attachToTarget" => {
let target_id = params
.get("targetId")
.and_then(|t| t.as_str())
.unwrap_or("");
match self.targets.get(target_id) {
Some(entry) => ClientRoute::Local(
json!({ "id": id, "result": { "sessionId": entry.session_id } }),
),
Some(entry) => {
let session_id = entry.session_id.clone();
// Explicitly adopting a target makes it this client's
// (cross-session adopt, #21) — tag it into the adopter's
// group so it stays in that client's scoped getTargets and
// isn't churn-pruned.
if let Some(g) = self.client_groups.get(&client_id).cloned() {
self.target_group.insert(target_id.to_string(), g);
}
ClientRoute::Local(
json!({ "id": id, "result": { "sessionId": session_id } }),
)
}
None => ClientRoute::Local(json!({
"id": id,
"error": { "code": -32602, "message": format!("No such target {target_id}") }
@@ -157,6 +219,18 @@ impl RelayState {
self.next_global_id += 1;
let gid = self.next_global_id;
self.pending.insert(gid, (client_id, id));
// Remember the group a createTarget carries so the reply's
// targetId can be tagged to the creating session (issue #40).
if method == "Target.createTarget" {
if let Some(g) = params.get("agentGroup").and_then(|g| g.as_str()) {
if !g.is_empty() {
self.pending_create.insert(gid, g.to_string());
self.client_groups
.entry(client_id)
.or_insert_with(|| g.to_string());
}
}
}
ClientRoute::Forward(json!({
"id": gid,
"method": "forwardCDPCommand",
@@ -201,6 +275,17 @@ impl RelayState {
&& msg.get("method").is_none()
{
let gid = msg.get("id").and_then(|i| i.as_i64());
// A createTarget reply: tag the new tab's targetId with the group the
// command carried, so it lands in the creating session's scope (#40).
if let Some(g) = gid.and_then(|g| self.pending_create.remove(&g)) {
if let Some(tid) = msg
.get("result")
.and_then(|r| r.get("targetId"))
.and_then(|t| t.as_str())
{
self.target_group.insert(tid.to_string(), g);
}
}
let (to, orig_id) = match gid.and_then(|g| self.pending.remove(&g)) {
Some((client_id, orig)) => (Some(client_id), orig),
// No mapping (stale/unknown id) — fall back to broadcasting with
@@ -241,6 +326,27 @@ impl RelayState {
.and_then(|s| s.as_str())
.unwrap_or("")
.to_string();
// Attribute the tab to a group for scoping (issue #40),
// unless we already know it (createTarget tag). An
// explicit `abGroup` from the extension wins; otherwise a
// pop-up inherits its opener's group via `openerTargetId`.
if !self.target_group.contains_key(tid) {
if let Some(g) = info
.get("abGroup")
.and_then(|g| g.as_str())
.filter(|g| !g.is_empty())
{
self.target_group.insert(tid.to_string(), g.to_string());
} else if let Some(opener) = info
.get("openerTargetId")
.and_then(|o| o.as_str())
.filter(|o| !o.is_empty())
{
if let Some(g) = self.target_group.get(opener).cloned() {
self.target_group.insert(tid.to_string(), g);
}
}
}
self.targets.insert(
tid.to_string(),
TargetEntry {
@@ -255,6 +361,15 @@ impl RelayState {
"Target.detachedFromTarget" => {
let gone = inner_params.get("sessionId").and_then(|s| s.as_str());
if let Some(gone) = gone {
let gone_tids: Vec<String> = self
.targets
.iter()
.filter(|(_, e)| e.session_id == gone)
.map(|(tid, _)| tid.clone())
.collect();
for tid in gone_tids {
self.target_group.remove(&tid);
}
self.targets.retain(|_, e| e.session_id != gone);
}
return vec![];
@@ -557,4 +672,158 @@ mod tests {
_ => panic!("expected ToExt"),
}
}
// === Group-scoped isolation (issue #40) ===
/// Drive the real create path: announce group, createTarget(agentGroup), feed
/// the ext reply (tags target→group) + the attachedToTarget event (creates the
/// entry). Returns nothing; mutates `s`.
fn create_in_group(s: &mut RelayState, client: ClientId, group: &str, tid: &str, sid: &str) {
s.route_client_command(
client,
&json!({ "id": 1, "method": "ABRelay.setGroup", "params": { "group": group } }),
);
let route = s.route_client_command(
client,
&json!({ "id": 2, "method": "Target.createTarget",
"params": { "url": "about:blank", "agentGroup": group } }),
);
let gid = match route {
ClientRoute::Forward(env) => env["id"].as_i64().unwrap(),
_ => panic!("createTarget must forward"),
};
s.handle_ext_message(&json!({ "id": gid, "result": { "targetId": tid } }), "");
s.handle_ext_message(
&json!({ "method": "forwardCDPEvent", "params": {
"method": "Target.attachedToTarget",
"params": { "sessionId": sid, "targetInfo": {
"targetId": tid, "type": "page", "url": "about:blank", "attached": true } } } }),
"",
);
}
fn get_target_ids(s: &mut RelayState, client: ClientId) -> Vec<String> {
match s.route_client_command(client, &json!({ "id": 9, "method": "Target.getTargets" })) {
ClientRoute::Local(v) => v["result"]["targetInfos"]
.as_array()
.unwrap()
.iter()
.map(|t| t["targetId"].as_str().unwrap().to_string())
.collect(),
_ => panic!("getTargets must be local"),
}
}
#[test]
fn get_targets_is_scoped_to_each_clients_group() {
let mut s = RelayState::new();
create_in_group(&mut s, 1, "agent-a", "ta", "sa");
create_in_group(&mut s, 2, "agent-b", "tb", "sb");
// Each client sees ONLY its own group's tab — never the other agent's.
assert_eq!(get_target_ids(&mut s, 1), vec!["ta"]);
assert_eq!(get_target_ids(&mut s, 2), vec!["tb"]);
}
#[test]
fn legacy_client_without_group_sees_all_targets() {
let mut s = RelayState::new();
create_in_group(&mut s, 1, "agent-a", "ta", "sa");
create_in_group(&mut s, 2, "agent-b", "tb", "sb");
// Client 3 never announced a group → full, unscoped list (back-compat).
let mut all = get_target_ids(&mut s, 3);
all.sort();
assert_eq!(all, vec!["ta", "tb"]);
}
#[test]
fn popup_inherits_opener_group_and_is_visible_to_that_client_only() {
let mut s = RelayState::new();
create_in_group(&mut s, 1, "agent-a", "ta", "sa");
create_in_group(&mut s, 2, "agent-b", "tb", "sb");
// A pop-up that agent-a's tab opened: extension reports openerTargetId=ta.
s.handle_ext_message(
&json!({ "method": "forwardCDPEvent", "params": {
"method": "Target.attachedToTarget",
"params": { "sessionId": "sp", "targetInfo": {
"targetId": "tp", "type": "page", "url": "https://oauth.example/",
"attached": true, "openerTargetId": "ta" } } } }),
"",
);
// Only agent-a sees the pop-up; agent-b never does.
let mut a = get_target_ids(&mut s, 1);
a.sort();
assert_eq!(a, vec!["ta", "tp"]);
assert_eq!(get_target_ids(&mut s, 2), vec!["tb"]);
}
#[test]
fn explicit_attach_tags_target_into_adopter_group() {
let mut s = RelayState::new();
// A pre-existing, ungrouped tab the extension reported (e.g. user's tab).
s.handle_ext_message(
&json!({ "method": "forwardCDPEvent", "params": {
"method": "Target.attachedToTarget",
"params": { "sessionId": "su", "targetInfo": {
"targetId": "tu", "type": "page", "url": "https://user.example/", "attached": true } } } }),
"",
);
// Client 1 (group agent-a) explicitly adopts it by targetId (#21).
s.route_client_command(
1,
&json!({ "id": 1, "method": "ABRelay.setGroup", "params": { "group": "agent-a" } }),
);
s.route_client_command(
1,
&json!({ "id": 2, "method": "Target.attachToTarget", "params": { "targetId": "tu" } }),
);
// Now it's in agent-a's scope and survives the scoped getTargets.
assert_eq!(get_target_ids(&mut s, 1), vec!["tu"]);
// A different agent still doesn't see it.
s.route_client_command(
2,
&json!({ "id": 1, "method": "ABRelay.setGroup", "params": { "group": "agent-b" } }),
);
assert!(get_target_ids(&mut s, 2).is_empty());
}
#[test]
fn get_all_targets_is_unscoped() {
let mut s = RelayState::new();
create_in_group(&mut s, 1, "agent-a", "ta", "sa");
create_in_group(&mut s, 2, "agent-b", "tb", "sb");
// Client 1's scoped getTargets sees only its own group...
assert_eq!(get_target_ids(&mut s, 1), vec!["ta"]);
// ...but ABRelay.getAllTargets returns EVERY target regardless of group
// (for explicit cross-group adoption).
let all = match s
.route_client_command(1, &json!({ "id": 1, "method": "ABRelay.getAllTargets" }))
{
ClientRoute::Local(v) => {
let mut ids: Vec<String> = v["result"]["targetInfos"]
.as_array()
.unwrap()
.iter()
.map(|t| t["targetId"].as_str().unwrap().to_string())
.collect();
ids.sort();
ids
}
_ => panic!("getAllTargets must be local"),
};
assert_eq!(all, vec!["ta", "tb"]);
}
#[test]
fn detach_clears_target_group() {
let mut s = RelayState::new();
create_in_group(&mut s, 1, "agent-a", "ta", "sa");
assert_eq!(get_target_ids(&mut s, 1), vec!["ta"]);
s.handle_ext_message(
&json!({ "method": "forwardCDPEvent", "params": {
"method": "Target.detachedFromTarget", "params": { "sessionId": "sa" } } }),
"",
);
assert!(get_target_ids(&mut s, 1).is_empty());
assert!(!s.target_group.contains_key("ta"));
}
}
+14 -1
View File
@@ -60,6 +60,9 @@ pub struct ScreenshotOptions {
pub quality: Option<i32>,
pub annotate: bool,
pub output_dir: Option<String>,
/// Explicit pixel region (x, y, width, height) — `--clip` (issue #34). Takes
/// precedence over selector/full_page.
pub clip: Option<(f64, f64, f64, f64)>,
}
impl Default for ScreenshotOptions {
@@ -72,6 +75,7 @@ impl Default for ScreenshotOptions {
quality: None,
annotate: false,
output_dir: None,
clip: None,
}
}
}
@@ -187,7 +191,16 @@ async fn capture_screenshot_base64(
capture_beyond_viewport: if options.full_page { Some(true) } else { None },
};
if options.full_page {
if let Some((x, y, width, height)) = options.clip {
// Explicit pixel region wins over selector/full_page (issue #34).
params.clip = Some(Viewport {
x,
y,
width,
height,
scale: 1.0,
});
} else if options.full_page {
let metrics: Value = client
.send_command_no_params("Page.getLayoutMetrics", Some(session_id))
.await?;
+36 -5
View File
@@ -330,6 +330,13 @@ impl RoleNameTracker {
}
}
/// Max iframe nesting depth `take_snapshot` expands. Embedded payment/checkout
/// widgets nest a few frames deep (e.g. AdSense → payments.google.com → an inner
/// form frame); expanding past the first level is what gives those inner refs a
/// `frame_id` so clicks resolve into the right frame (issue #36). Capped to keep
/// a pathological frame tree from blowing up the snapshot.
const MAX_IFRAME_DEPTH: usize = 3;
pub async fn take_snapshot(
client: &CdpClient,
session_id: &str,
@@ -337,6 +344,28 @@ pub async fn take_snapshot(
ref_map: &mut RefMap,
frame_id: Option<&str>,
iframe_sessions: &HashMap<String, String>,
) -> Result<String, String> {
take_snapshot_at_depth(
client,
session_id,
options,
ref_map,
frame_id,
iframe_sessions,
0,
)
.await
}
#[allow(clippy::too_many_arguments)]
async fn take_snapshot_at_depth(
client: &CdpClient,
session_id: &str,
options: &SnapshotOptions,
ref_map: &mut RefMap,
frame_id: Option<&str>,
iframe_sessions: &HashMap<String, String>,
depth: usize,
) -> Result<String, String> {
client
.send_command_no_params("DOM.enable", Some(session_id))
@@ -606,10 +635,11 @@ pub async fn take_snapshot(
}
// Recurse into child iframes: for each Iframe node with a backend_node_id,
// resolve the child frame ID and take a snapshot of its content.
// We only recurse from the main frame (frame_id == None) to avoid
// unbounded depth; nested iframes within iframes are not expanded.
if frame_id.is_none() {
// resolve the child frame ID and snapshot its content. Recurse to
// MAX_IFRAME_DEPTH (not just the main frame) so refs inside nested
// payment/checkout widgets get a `frame_id` and clicks resolve into the right
// frame (issue #36); the cap bounds a pathological frame tree.
if depth < MAX_IFRAME_DEPTH {
let mut iframe_snapshots: Vec<(String, String)> = Vec::new(); // (ref_id, child_snapshot)
for node in tree_nodes.iter() {
if node.role != "Iframe" || !node.has_ref {
@@ -622,13 +652,14 @@ pub async fn take_snapshot(
if let Ok(child_fid) = resolve_iframe_frame_id(client, session_id, bid).await {
// Snapshot the child frame; errors are silently ignored
// (e.g. cross-origin iframes)
if let Ok(child_text) = Box::pin(take_snapshot(
if let Ok(child_text) = Box::pin(take_snapshot_at_depth(
client,
session_id,
options,
ref_map,
Some(&child_fid),
iframe_sessions,
depth + 1,
))
.await
{
@@ -0,0 +1,28 @@
<!doctype html>
<html>
<head>
<meta charset="utf-8" />
<title>iframe button probe</title>
</head>
<body>
<h1>iframe button probe</h1>
<iframe
id="frame"
width="320"
height="140"
srcdoc="
<!doctype html>
<html>
<body style='margin:24px'>
<button id='b' style='padding:24px;font-size:22px'>save</button>
<script>
document.getElementById('b').addEventListener('click', function (e) {
this.textContent = 'clicked:' + e.isTrusted;
});
</script>
</body>
</html>
"
></iframe>
</body>
</html>
+237 -20
View File
@@ -186,6 +186,26 @@ pub fn print_response_with_opts(resp: &Response, action: Option<&str>, opts: &Ou
}
if let Some(data) = &resp.data {
// Auto-trigger: when you land on / read a page whose domain has site
// adapters, surface them so the agent pulls structured data via
// `chrome-use site <name>/<cmd>` instead of scraping the DOM. (In --json
// mode this same info rides along in the `siteAdapters` field above.)
if let Some(hint) = data.get("siteAdapters") {
let domain = hint.get("domain").and_then(|v| v.as_str()).unwrap_or("");
let cmds: Vec<&str> = hint
.get("commands")
.and_then(|v| v.as_array())
.map(|a| a.iter().filter_map(|v| v.as_str()).collect())
.unwrap_or_default();
if !cmds.is_empty() {
eprintln!("💡 site adapters for {domain} — prefer these for structured data:");
eprintln!(" {}", color::dim(&cmds.join(", ")));
eprintln!(
" {}",
color::dim(&format!("e.g. chrome-use site {} --json", cmds[0]))
);
}
}
// A click that opened a new tab: surface it so the agent doesn't read the
// unchanged old page as a failed click (issue #24-A).
if let Some(opened) = data.get("openedTab") {
@@ -224,6 +244,65 @@ pub fn print_response_with_opts(resp: &Response, action: Option<&str>, opts: &Ou
return;
}
// Cloudflare challenge/clearance preflight (`cf-status`). Checked early
// because its response carries `url`/`title`, which later generic
// renderers would otherwise swallow.
if action == Some("cf_status") {
let challenged = data
.get("challenged")
.and_then(|v| v.as_bool())
.unwrap_or(false);
let rec = data
.get("recommendation")
.and_then(|v| v.as_str())
.unwrap_or("?");
let cl = data.get("clearance");
let present = cl
.and_then(|c| c.get("present"))
.and_then(|v| v.as_bool())
.unwrap_or(false);
let expired = cl
.and_then(|c| c.get("expired"))
.and_then(|v| v.as_bool())
.unwrap_or(false);
let expires_in = cl.and_then(|c| c.get("expiresIn")).and_then(|v| v.as_i64());
let device = data
.get("deviceVerified")
.and_then(|v| v.as_bool())
.unwrap_or(false);
let (icon, headline) = match rec {
"proceed" => (color::success_indicator().to_string(), "cleared — no challenge, proceed"),
"solve" => (color::warning_indicator().to_string(), "Cloudflare challenge active, no valid clearance — solve it"),
"reissue" => (color::warning_indicator().to_string(), "challenge active but a clearance cookie exists — stale (IP/UA changed?), re-solve"),
_ => (color::cyan("").to_string(), "unknown"),
};
println!("{} {}", icon, headline);
println!(
" challenged: {}",
if challenged { "yes" } else { "no" }
);
let cl_desc = if !present {
"absent".to_string()
} else if expired {
"present but EXPIRED".to_string()
} else if let Some(s) = expires_in {
format!("valid, expires in {}m {}s", s / 60, s % 60)
} else {
"present (session)".to_string()
};
println!(" cf_clearance: {}", cl_desc);
println!(
" device trusted: {}",
if device {
"yes (CF_VERIFIED_DEVICE)"
} else {
"no"
}
);
return;
}
// Dialog status response
if action == Some("dialog") {
if let Some(has_dialog) = data.get("hasDialog").and_then(|v| v.as_bool()) {
@@ -342,6 +421,38 @@ pub fn print_response_with_opts(resp: &Response, action: Option<&str>, opts: &Ou
}
return;
}
// Frame list (`chrome-use frames`)
if action == Some("frames") {
if let Some(list) = data.get("frames").and_then(|v| v.as_array()) {
let count = list.len();
println!(
"{}",
color::bold(&format!(
"{} frame{}",
count,
if count == 1 { "" } else { "s" }
))
);
for f in list {
let idx = f.get("index").and_then(|v| v.as_i64()).unwrap_or(0);
let kind = f.get("kind").and_then(|v| v.as_str()).unwrap_or("?");
let url = f.get("url").and_then(|v| v.as_str()).unwrap_or("");
let len = f.get("textLen").and_then(|v| v.as_i64()).unwrap_or(0);
println!(
" [{}] {:<6} {} chars {}",
idx,
kind,
len,
color::dim(if url.is_empty() { "(about:blank)" } else { url })
);
}
eprintln!(
"{}",
color::dim("read everything with: chrome-use get text --all-frames")
);
}
return;
}
// Title
if let Some(title) = data.get("title").and_then(|v| v.as_str()) {
println!("{}", title);
@@ -378,6 +489,17 @@ pub fn print_response_with_opts(resp: &Response, action: Option<&str>, opts: &Ou
println!("y: {}", y);
println!("width: {}", w);
println!("height: {}", h);
if let (Some(cx), Some(cy)) = (
obj.get("centerX").and_then(|v| v.as_i64()),
obj.get("centerY").and_then(|v| v.as_i64()),
) {
// Echoed in click-ready CSS px so the agent can paste straight
// into `click <centerX> <centerY>` (issue #43).
println!("center: {} {}", cx, cy);
}
if let Some(iv) = obj.get("inViewport").and_then(|v| v.as_bool()) {
println!("inViewport: {}", iv);
}
}
return;
}
@@ -569,19 +691,21 @@ pub fn print_response_with_opts(resp: &Response, action: Option<&str>, opts: &Ou
// Tab switch
if action == Some("tab_switch") {
if let Some(tab_id) = data.get("tabId").and_then(|v| v.as_str()) {
if let Some(url) = data.get("url").and_then(|v| v.as_str()) {
println!(
"{} Switched to tab [{}] ({})",
color::success_indicator(),
tab_id,
url
);
let warning = data.get("warning").and_then(|v| v.as_str());
// A non-responding session isn't a real success — show a warning
// indicator instead of the green ✓ (issue #29.3).
let indicator = if warning.is_some() {
color::warning_indicator()
} else {
println!(
"{} Switched to tab [{}]",
color::success_indicator(),
tab_id
);
color::success_indicator()
};
if let Some(url) = data.get("url").and_then(|v| v.as_str()) {
println!("{} Switched to tab [{}] ({})", indicator, tab_id, url);
} else {
println!("{} Switched to tab [{}]", indicator, tab_id);
}
if let Some(w) = warning {
eprintln!("{}", color::dim(w));
}
return;
}
@@ -1377,9 +1501,20 @@ Examples:
chrome-use fill - Clear and fill an input field
Usage: chrome-use fill <selector> <text>
chrome-use fill <selector> --file <path>
chrome-use fill <selector> --stdin
Clears the input field and fills it with the specified text.
This replaces any existing content in the field.
Clears the field and fills it with the text, replacing existing content.
Works on rich editors too (issue #41): CodeMirror 5, Monaco, ProseMirror and
plain contenteditable are detected and set via their own API / input events,
not a raw `.value` write and the response echoes which `engine` was used.
For framework inputs (React/Vue/Angular) the value goes through the native
setter so the form registers it (no more "pristine" Save no-ops).
Options:
--file <path> Read the value from a UTF-8 file (large/multiline text,
backticks/quotes/newlines/non-ASCII no shell escaping)
--stdin Read the value from stdin
Global Options:
--json Output as JSON
@@ -1388,7 +1523,8 @@ Global Options:
Examples:
chrome-use fill "#email" "user@example.com"
chrome-use fill @e3 "Hello World"
chrome-use fill "input[name='search']" "query"
chrome-use fill ".CodeMirror" --file ./article.md # set a CodeMirror editor
cat post.md | chrome-use fill @e7 --stdin
"##
}
"type" => {
@@ -1400,6 +1536,12 @@ Usage: chrome-use type <selector> <text>
Types text into the specified element character by character.
Unlike fill, this does not clear existing content first.
Options:
--key-events Send real per-character keyDown/keyUp instead of
(alias --keys) Input.insertText. Use for autocomplete / combobox fields
that only react to key events e.g. a postal-code box
that auto-fills city/prefecture, or Google Places.
Global Options:
--json Output as JSON
--session <name> Use specific session
@@ -1407,6 +1549,7 @@ Global Options:
Examples:
chrome-use type "#search" "hello"
chrome-use type @e2 "additional text"
chrome-use type @e5 "201-0001" --key-events # trigger the address autocomplete
See Also:
For typing into contenteditable editors (Lexical, ProseMirror, etc.)
@@ -1671,12 +1814,23 @@ Usage: chrome-use scroll [direction] [amount] [options]
Scrolls the page or a specific element in the specified direction.
Without --selector, scroll dispatches a real (isTrusted) mouse wheel at a
viewport coordinate, so it scrolls whatever container is under the pointer
including cross-origin iframes (Google Payments, Stripe, embedded checkout/KYC)
that plain page scroll can't reach.
Arguments:
direction up, down, left, right (default: down)
amount Pixels to scroll (default: 300)
Options:
-s, --selector <sel> CSS selector for a scrollable container
-s, --selector <sel> CSS selector for a scrollable container (same-origin)
--at <x,y> Dispatch the wheel at this viewport pixel (read it from a
screenshot) precise way into a cross-origin iframe
--frame <n> Scroll the n-th frame from `chrome-use frames` (wheel at
that frame's center)
Without --selector/--at/--frame the wheel lands at the viewport center.
Global Options:
--json Output as JSON
@@ -1688,6 +1842,8 @@ Examples:
chrome-use scroll up 200
chrome-use scroll left 100
chrome-use scroll down 500 --selector "div.scroll-container"
chrome-use scroll down 700 --at 640,400 # wheel at a pixel over an iframe
chrome-use scroll down 700 --frame 2 # scroll frame 2 from `frames`
"##
}
"scrollintoview" | "scrollinto" => {
@@ -1768,6 +1924,15 @@ Pass --hide-scrollbars false when launching to keep native scrollbars visible.
Options:
--full, -f Capture full page (not just viewport)
[selector] Capture just an element (CSS or @ref), e.g. `screenshot ".header" h.png`
--clip <x,y,w,h> Capture a pixel region, e.g. `screenshot --clip 0,0,200,40 corner.png`
--max-width <px> Downscale so the image's width px (preserves aspect)
--max-height <px> Downscale so the image's height px
--scale <0..1> Downscale by a factor, e.g. 0.5 (DPR-1, so screenshot px
line up 1:1 with `click x y`)
Default: capped at 2000px longest edge unless overridden
(AGENT_BROWSER_SCREENSHOT_MAX_EDGE; 0 disables). Annotated
shots are never downscaled, so ref overlays stay aligned.
--annotate Overlay numbered labels on interactive elements.
Each label [N] corresponds to ref @eN from snapshot.
Prints a legend mapping labels to element roles/names.
@@ -1788,6 +1953,10 @@ Examples:
chrome-use screenshot
chrome-use screenshot ./screenshot.png
chrome-use screenshot --full ./full-page.png
chrome-use screenshot ".header .indicator" corner.png # just one element
chrome-use screenshot --clip 1600,0,200,40 corner.png # a pixel region
chrome-use screenshot --scale 0.5 ./half.png # DPR-1: screenshot px == click px
chrome-use screenshot --max-width 1400 ./shot.png # cap width for image readers
chrome-use screenshot --annotate # Labeled screenshot + legend
chrome-use screenshot --annotate ./page.png # Save annotated screenshot
chrome-use screenshot --annotate --json # JSON output with annotations
@@ -1929,7 +2098,9 @@ Usage: chrome-use get <subcommand> [args]
Retrieves various types of information from elements or the page.
Subcommands:
text <selector> Get text content of element
text [selector] Element text; no selector = WHOLE PAGE, all frames
text --main Main-content text only (skip nav/header/sidebar)
text --pierce Read through CLOSED shadow DOM (injected panels)
html <selector> Get inner HTML of element
value <selector> Get value of input element
attr <selector> <name> Get attribute value
@@ -1945,7 +2116,10 @@ Global Options:
--session <name> Use specific session
Examples:
chrome-use get text @e1
chrome-use get text # whole page across ALL frames (default)
chrome-use get text @e1 # one element
chrome-use get text --main # main content, no nav/sidebar boilerplate
chrome-use frames # list frames + where the text lives
chrome-use get html "#content"
chrome-use get value "#email-input"
chrome-use get attr "#link" href
@@ -3121,7 +3295,8 @@ Core Commands:
click <sel|x y> Click element/@ref, or a viewport coordinate
dblclick <sel> Double-click element
type <sel> <text> Type into element
fill <sel> <text> Clear and fill
fill <sel> <text> Clear and fill (handles CodeMirror/Monaco/ProseMirror/
contenteditable; `--file <path>`/`--stdin` for large text)
press <key> [--hold <ms>] Press key (Enter, Tab, Control+a). --hold keeps it
down <ms> then releases precise (in-daemon), for
games/charge: `press d --hold 800`
@@ -3141,11 +3316,19 @@ Core Commands:
scroll <dir> [px] Scroll (up/down/left/right)
scrollintoview <sel> Scroll element into view
wait <sel|ms> Wait for element or time
screenshot [path] Take screenshot
screenshot [path] Take screenshot (auto-downscaled to 2000px long edge;
--max-width/--max-height/--scale to override)
pdf <path> Save as PDF
canvas list List <canvas> elements (size, type) on the page
canvas capture [sel] [path] Save a canvas's rendered pixels to PNG for
WebGL/canvas apps (Figma, games, maps, charts) that
expose no DOM. toDataURL, with a screenshot fallback.
snapshot Accessibility tree with refs (for AI)
eval <js> Run JavaScript
connect <port|url> Connect to browser via CDP
keep Leave the active tab for the user exempt it from
auto-close/idle cleanup + remove it from the session
tab group (so scratch tabs get cleaned, this one stays)
close [--all] Close browser (--all closes every session)
Navigation:
@@ -3155,10 +3338,17 @@ Navigation:
Get Info: chrome-use get <what> [selector]
text, html, value, attr <name>, title, url, count, box, styles, cdp-url
box <sel> x,y,width,height,centerX,centerY,inViewport in CSS px (feed
centerX/centerY into `click x y`); value reads CodeMirror/Monaco too
text (no selector = whole page, all frames), text --main, frames (list)
Check State: chrome-use is <what> <selector>
visible, enabled, checked
Anti-bot: chrome-use stealth | cf-status
stealth stealth self-check (webdriver/UA/plugins + overrides)
cf-status Cloudflare challenge + cf_clearance preflight (skip re-solving)
Find Elements: chrome-use find <locator> <value> <action> [text]
role, text, label, placeholder, alt, title, testid, first, last, nth
@@ -3192,6 +3382,10 @@ Tabs:
stable targetId, no reload preserves in-page state
open <url> --reuse-tab Reuse an existing tab on that URL instead of spawning
a duplicate (matches origin+path; preserves state)
adopt <url|targetId> Read a PRE-EXISTING tab (the user's own, or another
session's) WITHOUT opening a new one matches by URL
substring or stable targetId, then drives it. e.g.
`adopt "github.com/owner/repo"`
Diff:
diff snapshot Compare current vs last snapshot
@@ -3239,6 +3433,14 @@ Batch:
batch [--bail] ["cmd" ...] Execute multiple commands sequentially (args or stdin)
--bail stops on first error (default: continue all)
Site adapters: turn a website into a structured-data CLI (runs as you, in your tab)
site update Fetch the community adapter pack into ~/.chrome-use/sites
site list List installed adapters (name/cmd)
site info <name>/<cmd> Show an adapter's @meta (args, domain, capabilities)
site <name>/<cmd> [args] Run an adapter: navigate to its site + return JSON
e.g. site github/issues epiral/repo, site reddit/search rust
Positional args fill declared args in order; --key value overrides
Auth Vault:
auth save <name> [opts] Save auth profile (--url, --username, --password/--password-stdin)
auth login <name> Login using saved credentials (waits for form fields)
@@ -3253,10 +3455,21 @@ Confirmation:
Sessions:
session Show current session name
session list List active sessions
session stop [name] Stop one session daemon (default: current) graceful,
closes the tabs it created
session prune Stop ALL session daemons now (closes their tabs; they
respawn clean on next use). For clearing idle daemons.
sessions List running session daemons (alias of daemon status)
daemon status List running session daemons (+ relay state)
daemon restart Kill all session daemons; keeps the extension relay
up. Clears stale/cross-leaked state after an upgrade.
Lifecycle: each --session <name> spawns a background daemon that drives that
session's tabs. A daemon auto-shuts-down after 10 min idle (no commands)
AGENT_BROWSER_IDLE_TIMEOUT_MS overrides, 0 disables and on shutdown closes
the scratch tabs IT created (its tab group). Use `keep` to leave a tab for the
user (exempt from auto-close), `session stop/prune` to reclaim now.
Chat (AI):
chat <message> Send a natural language instruction (single-shot)
chat Start interactive chat (REPL mode when stdin is a TTY)
@@ -3449,6 +3662,9 @@ iOS Simulator (requires Xcode and Appium):
chrome-use -p ios device list # List simulators
chrome-use -p ios swipe up # Swipe gesture
chrome-use -p ios tap @e1 # Touch element
Hit a bug or rough edge? A 30-second issue genuinely sharpens this tool:
https://github.com/leeguooooo/chrome-use/issues
"#
);
}
@@ -3531,6 +3747,7 @@ fn print_screenshot_diff(data: &serde_json::Map<String, serde_json::Value>) {
pub fn print_version() {
println!("chrome-use {}", env!("CARGO_PKG_VERSION"));
println!("report bugs / rough edges: https://github.com/leeguooooo/chrome-use/issues");
}
#[cfg(test)]
+465
View File
@@ -0,0 +1,465 @@
//! Site adapters: turn any website into a structured-data CLI by running a small
//! per-command JS adapter inside your real, logged-in browser tab (it reuses the
//! site's cookies / same-origin fetch / its own webpack modules — the site thinks
//! it's you, because it is).
//!
//! The adapter format is the community **bb-sites** convention
//! (<https://github.com/epiral/bb-sites>): one `.js` file per command, a
//! `/* @meta {...} */` JSON header (name, description, domain, args), then an
//! `async function(args){ ... return {...} }`. chrome-use ships none of those
//! adapters — `chrome-use site update` fetches the upstream repo at runtime into
//! `~/.chrome-use/sites` (like a package manager pulling a dependency), so the
//! adapters stay the property of their authors. Running an adapter navigates to
//! its `@meta.domain` and `eval`s the function in the site's own logged-in page.
use std::path::PathBuf;
use serde_json::Value;
const SITES_ZIP_URL: &str = "https://github.com/epiral/bb-sites/archive/refs/heads/main.zip";
/// `~/.chrome-use/sites` — where synced adapters live.
pub fn sites_dir() -> Option<PathBuf> {
dirs_home().map(|h| h.join(".chrome-use").join("sites"))
}
fn dirs_home() -> Option<PathBuf> {
std::env::var_os("HOME").map(PathBuf::from)
}
/// Parsed adapter: its `@meta` JSON and the raw `async function(args){...}` source.
pub struct Adapter {
pub meta: Value,
pub func_src: String,
/// The adapter's declared `args` keys in DECLARATION order. Parsed from the
/// raw @meta text because `serde_json` sorts object keys alphabetically, which
/// would otherwise scramble positional-arg mapping for multi-arg adapters.
pub arg_order: Vec<String>,
}
impl Adapter {
pub fn domain(&self) -> Option<&str> {
self.meta.get("domain").and_then(|v| v.as_str())
}
}
/// Load `<sites>/<name>/<cmd>.js`, splitting the `/* @meta {...} */` header from
/// the function body. `spec` is `name/cmd`.
pub fn load_adapter(spec: &str) -> Result<Adapter, String> {
let (name, cmd) = spec
.split_once('/')
.ok_or_else(|| format!("site: expected <name>/<command>, got `{spec}`"))?;
if name.is_empty()
|| cmd.is_empty()
|| name.contains("..")
|| cmd.contains("..")
|| name.contains('/')
|| cmd.contains('/')
{
return Err(format!("site: invalid adapter spec `{spec}`"));
}
let dir = sites_dir().ok_or("site: cannot resolve home dir")?;
let path = dir.join(name).join(format!("{cmd}.js"));
if !path.exists() {
return Err(format!(
"site: adapter `{spec}` not found. Run `chrome-use site update` to sync adapters, \
or `chrome-use site list` to see what's installed."
));
}
let raw = std::fs::read_to_string(&path).map_err(|e| format!("site: read {spec}: {e}"))?;
parse_adapter(&raw, spec)
}
/// Split the `@meta` JSON block and the function source from an adapter file.
pub fn parse_adapter(raw: &str, spec: &str) -> Result<Adapter, String> {
let start = raw
.find("@meta")
.and_then(|i| raw[i..].find('{').map(|j| i + j))
.ok_or_else(|| format!("site: {spec} missing /* @meta {{...}} */ header"))?;
// Find the matching close brace for the @meta object (brace-count, string-aware).
let bytes = raw.as_bytes();
let mut depth = 0i32;
let mut in_str = false;
let mut esc = false;
let mut end = None;
for (k, &b) in bytes.iter().enumerate().skip(start) {
if in_str {
if esc {
esc = false;
} else if b == b'\\' {
esc = true;
} else if b == b'"' {
in_str = false;
}
continue;
}
match b {
b'"' => in_str = true,
b'{' => depth += 1,
b'}' => {
depth -= 1;
if depth == 0 {
end = Some(k + 1);
break;
}
}
_ => {}
}
}
let end = end.ok_or_else(|| format!("site: {spec} @meta header has no closing brace"))?;
let meta: Value = serde_json::from_str(&raw[start..end])
.map_err(|e| format!("site: {spec} @meta is not valid JSON: {e}"))?;
// The function is everything after the meta comment's closing `*/`.
let after = raw[end..].find("*/").map(|i| end + i + 2).unwrap_or(end);
let func_src = raw[after..].trim().to_string();
if func_src.is_empty() {
return Err(format!("site: {spec} has no function body after @meta"));
}
let arg_order = arg_order_from_meta(&raw[start..end]);
Ok(Adapter {
meta,
func_src,
arg_order,
})
}
/// Extract the `args` object's keys in DECLARATION order from the raw @meta JSON
/// text (serde sorts them, losing order). Brace/string-aware: finds the `"args"`
/// value object and collects only its top-level keys.
fn arg_order_from_meta(meta_json: &str) -> Vec<String> {
let bytes = meta_json.as_bytes();
// Locate the `"args"` key, then the `{` that opens its value object.
let Some(args_pos) = meta_json.find("\"args\"") else {
return Vec::new();
};
let Some(brace_off) = meta_json[args_pos..].find('{') else {
return Vec::new();
};
let open = args_pos + brace_off;
let mut keys = Vec::new();
let mut depth = 0i32;
let mut in_str = false;
let mut esc = false;
let mut cur = String::new();
let mut last_str: Option<String> = None;
for &b in bytes.iter().skip(open) {
if in_str {
if esc {
esc = false;
} else if b == b'\\' {
esc = true;
} else if b == b'"' {
in_str = false;
last_str = Some(std::mem::take(&mut cur));
} else {
cur.push(b as char);
}
continue;
}
match b {
b'"' => in_str = true,
b'{' => depth += 1,
b'}' => {
depth -= 1;
if depth == 0 {
break; // end of the args object
}
}
// A `:` at depth 1 means the preceding string was a key of `args`.
b':' if depth == 1 => {
if let Some(k) = last_str.take() {
keys.push(k);
}
}
_ => {}
}
}
keys
}
/// Build the JS to eval: `(<adapter function>)(<args JSON>)`. The adapter's
/// `async function(args)` returns a promise; chrome-use's eval awaits it.
pub fn build_eval(adapter: &Adapter, args: &Value) -> String {
let args_json = serde_json::to_string(args).unwrap_or_else(|_| "{}".to_string());
format!("({})({})", adapter.func_src, args_json)
}
/// List installed adapters as `name/cmd` strings (sorted).
pub fn list_adapters() -> Result<Vec<String>, String> {
let dir = sites_dir().ok_or("site: cannot resolve home dir")?;
if !dir.exists() {
return Ok(Vec::new());
}
let mut out = Vec::new();
for site in std::fs::read_dir(&dir)
.map_err(|e| e.to_string())?
.flatten()
{
if !site.path().is_dir() {
continue;
}
let name = site.file_name().to_string_lossy().to_string();
for cmd in std::fs::read_dir(site.path())
.map_err(|e| e.to_string())?
.flatten()
{
let p = cmd.path();
if p.extension().and_then(|e| e.to_str()) == Some("js") {
if let Some(stem) = p.file_stem().and_then(|s| s.to_str()) {
out.push(format!("{name}/{stem}"));
}
}
}
}
out.sort();
Ok(out)
}
/// Download the bb-sites repo zip and extract its adapters into `~/.chrome-use/sites`.
pub async fn update() -> Result<usize, String> {
let dir = sites_dir().ok_or("site: cannot resolve home dir")?;
let client = reqwest::Client::builder()
.user_agent("chrome-use")
.build()
.map_err(|e| e.to_string())?;
let bytes = client
.get(SITES_ZIP_URL)
.send()
.await
.map_err(|e| format!("site update: download failed: {e}"))?
.error_for_status()
.map_err(|e| format!("site update: {e}"))?
.bytes()
.await
.map_err(|e| format!("site update: read body: {e}"))?;
let cursor = std::io::Cursor::new(bytes);
let mut zip = zip::ZipArchive::new(cursor).map_err(|e| format!("site update: bad zip: {e}"))?;
std::fs::create_dir_all(&dir).map_err(|e| e.to_string())?;
let mut count = 0usize;
for i in 0..zip.len() {
let mut f = zip.by_index(i).map_err(|e| e.to_string())?;
let Some(enclosed) = f.enclosed_name() else {
continue;
};
// Strip the top-level `bb-sites-main/` component from the archive path.
let rel: PathBuf = enclosed.components().skip(1).collect();
if rel.as_os_str().is_empty() {
continue;
}
let out = dir.join(&rel);
if f.is_dir() {
let _ = std::fs::create_dir_all(&out);
continue;
}
if let Some(parent) = out.parent() {
let _ = std::fs::create_dir_all(parent);
}
let mut buf = Vec::new();
std::io::copy(&mut f, &mut buf).map_err(|e| e.to_string())?;
std::fs::write(&out, &buf).map_err(|e| e.to_string())?;
if out.extension().and_then(|e| e.to_str()) == Some("js") {
count += 1;
}
}
// Build the domain→adapters index and stamp the sync time so navigation can
// suggest adapters (auto-trigger) and `needs_refresh` can pace re-syncs.
write_domain_index(&dir);
if let Some(p) = last_update_path() {
let _ = std::fs::write(p, now_secs().to_string());
}
Ok(count)
}
/// `~/.chrome-use/sites/.last_update` — unix-seconds marker of the last sync.
fn last_update_path() -> Option<PathBuf> {
sites_dir().map(|d| d.join(".last_update"))
}
/// `~/.chrome-use/sites/.index.json` — `{ "github.com": ["github/issues", …], … }`,
/// built on `update` so navigation can look up adapters by domain without parsing
/// all ~145 adapter files on every command.
fn index_path() -> Option<PathBuf> {
sites_dir().map(|d| d.join(".index.json"))
}
fn now_secs() -> u64 {
std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.map(|d| d.as_secs())
.unwrap_or(0)
}
/// Parse every installed adapter and write the domain→adapters index. Within a
/// domain, read-only adapters are listed first (then alphabetical) so the
/// auto-suggested example leads with a safe read, not a write action.
fn write_domain_index(dir: &std::path::Path) {
let mut by_domain: std::collections::BTreeMap<String, Vec<(bool, String)>> = Default::default();
for spec in list_adapters().unwrap_or_default() {
if let Ok(a) = load_adapter(&spec) {
if let Some(d) = a.domain() {
let read_only = a
.meta
.get("readOnly")
.and_then(|v| v.as_bool())
.unwrap_or(false);
by_domain
.entry(d.to_string())
.or_default()
.push((read_only, spec));
}
}
}
let ordered: std::collections::BTreeMap<String, Vec<String>> = by_domain
.into_iter()
.map(|(domain, mut v)| {
// read-only (true) first, then by spec name
v.sort_by(|a, b| b.0.cmp(&a.0).then_with(|| a.1.cmp(&b.1)));
(domain, v.into_iter().map(|(_, s)| s).collect())
})
.collect();
if let Ok(json) = serde_json::to_string(&ordered) {
let _ = std::fs::write(dir.join(".index.json"), json);
}
}
const DEFAULT_TTL_DAYS: u64 = 7;
/// Whether the adapter pack should be (re)synced: true on first use (nothing
/// installed) or when the last sync is older than the TTL. Disabled by
/// `AGENT_BROWSER_SITES_NO_AUTO_UPDATE=1`; TTL overridable via
/// `AGENT_BROWSER_SITES_TTL_DAYS` (0 = always).
pub fn needs_refresh() -> bool {
if std::env::var_os("AGENT_BROWSER_SITES_NO_AUTO_UPDATE").is_some() {
return false;
}
let Some(dir) = sites_dir() else {
return false;
};
// First use: no adapters installed yet.
if list_adapters().map(|l| l.is_empty()).unwrap_or(true) {
let _ = &dir;
return true;
}
let ttl_days = std::env::var("AGENT_BROWSER_SITES_TTL_DAYS")
.ok()
.and_then(|s| s.parse::<u64>().ok())
.unwrap_or(DEFAULT_TTL_DAYS);
let ttl = ttl_days.saturating_mul(86_400);
match last_update_path().and_then(|p| std::fs::read_to_string(p).ok()) {
Some(s) => match s.trim().parse::<u64>() {
Ok(ts) => now_secs().saturating_sub(ts) >= ttl,
Err(_) => true,
},
None => true, // no marker → treat as stale
}
}
/// Adapters whose `@meta.domain` matches `host` (exact, or `host` is a subdomain
/// of it) — for auto-suggesting `site` commands when you land on a known site.
/// Reads the prebuilt `.index.json`; empty if the pack isn't synced yet.
pub fn adapters_for_domain(host: &str) -> Vec<String> {
let host = host.trim_start_matches("www.");
let Some(raw) = index_path().and_then(|p| std::fs::read_to_string(p).ok()) else {
return Vec::new();
};
let Ok(idx) = serde_json::from_str::<std::collections::BTreeMap<String, Vec<String>>>(&raw)
else {
return Vec::new();
};
// Preserve the index's per-domain ordering (read-only adapters first); just
// dedup if a host somehow matches multiple domain keys.
let mut out: Vec<String> = Vec::new();
for (domain, specs) in idx {
let d = domain.trim_start_matches("www.");
if host == d || host.ends_with(&format!(".{d}")) {
for s in specs {
if !out.contains(&s) {
out.push(s);
}
}
}
}
out
}
/// Map CLI args to the adapter's `args` object. Positional args fill the adapter's
/// declared `args` keys in order; `--key value` overrides by name. The adapter
/// validates required args itself.
pub fn map_args(adapter: &Adapter, positional: &[String], named: &[(String, String)]) -> Value {
let mut obj = serde_json::Map::new();
// Positional args fill the adapter's declared args in DECLARATION order
// (`arg_order`), not serde's alphabetized key order — otherwise a 2-arg
// adapter like `{projectId, path}` would map positionals to `{path, projectId}`.
for (i, val) in positional.iter().enumerate() {
if let Some(k) = adapter.arg_order.get(i) {
obj.insert(k.clone(), Value::String(val.clone()));
}
}
for (k, v) in named {
obj.insert(k.clone(), Value::String(v.clone()));
}
Value::Object(obj)
}
#[cfg(test)]
mod tests {
use super::*;
const SAMPLE: &str = r#"/* @meta
{
"name": "github/issues",
"domain": "github.com",
"args": { "repo": {"required": true}, "state": {"required": false} }
}
*/
async function(args) { return { repo: args.repo }; }"#;
#[test]
fn parses_meta_and_function() {
let a = parse_adapter(SAMPLE, "github/issues").unwrap();
assert_eq!(a.domain(), Some("github.com"));
assert!(a.func_src.starts_with("async function(args)"));
}
#[test]
fn build_eval_wraps_and_passes_args() {
let a = parse_adapter(SAMPLE, "github/issues").unwrap();
let args = map_args(
&a,
&["owner/repo".into()],
&[("state".into(), "closed".into())],
);
let js = build_eval(&a, &args);
assert!(js.contains("async function(args)"));
assert!(js.contains("\"repo\":\"owner/repo\""));
assert!(js.contains("\"state\":\"closed\""));
}
#[test]
fn rejects_bad_spec() {
assert!(load_adapter("noslash").is_err());
assert!(load_adapter("../etc/passwd").is_err());
}
// Regression: positional args must follow DECLARATION order, not serde's
// alphabetical key order. With `{projectId, path}` (not alphabetical),
// `<uuid> <file>` must map projectId←uuid, path←file — not swapped.
#[test]
fn positional_args_follow_declaration_order_not_alphabetical() {
let raw = r#"/* @meta
{
"name": "claude-design/get-file",
"domain": "claude.ai",
"args": { "projectId": {"required": true}, "path": {"required": true} }
}
*/
async function(args){ return args; }"#;
let a = parse_adapter(raw, "claude-design/get-file").unwrap();
assert_eq!(a.arg_order, vec!["projectId", "path"]);
let args = map_args(&a, &["the-uuid".into(), "misonote.dc.html".into()], &[]);
assert_eq!(args["projectId"], "the-uuid");
assert_eq!(args["path"], "misonote.dc.html");
}
}
Binary file not shown.
Binary file not shown.
+123 -16
View File
@@ -30,6 +30,16 @@ const tabs = new Map()
const sessionToTab = new Map()
/** child (OOPIF/worker) sessionId -> tabId */
const childSessionToTab = new Map()
/** sessionId -> CDP targetId, kept ACROSS detach so a dead `cb-tab-<oldTabId>`
* session can be recovered by its stable targetId when the cross-process nav
* gave the tab a new Chrome tabId (issue #24). Capped to bound memory. */
const sessionTargets = new Map()
function rememberSessionTarget(sessionId, targetId) {
if (!sessionId || !targetId) return
sessionTargets.delete(sessionId)
sessionTargets.set(sessionId, targetId)
if (sessionTargets.size > 256) sessionTargets.delete(sessionTargets.keys().next().value)
}
/** tab-group name -> chrome tabGroups id (best-effort cache) */
const groupIdByName = new Map()
@@ -71,6 +81,35 @@ async function groupTabInto(tabId, name) {
groupIdByName.set(name, gid)
}
// Group-scoped relay isolation hints (issue #40). The relay scopes
// Target.getTargets per agent by tab group; report two things in the synthesized
// targetInfo so it can attribute each tab:
// - abGroup: the tab's Chrome tab-group TITLE (= the owning session name), so
// the relay can re-attribute existing tabs after a restart (createTarget
// tagging won't re-run for already-open tabs).
// - openerTargetId: the targetId of the tab that opened this one, so a pop-up
// (window.open / target=_blank / OAuth result) inherits its opener's group
// and the agent that opened it can follow it — without foreign tabs leaking.
// Best-effort: any failure yields empty strings, which the relay ignores.
async function tabScopeHints(tabId) {
let openerTargetId = ''
let abGroup = ''
try {
const t = await chrome.tabs.get(tabId)
if (t) {
if (typeof t.openerTabId === 'number') {
const op = tabs.get(t.openerTabId)
if (op) openerTargetId = op.targetId
}
if (t.groupId != null && t.groupId >= 0 && chrome.tabGroups) {
const g = await chrome.tabGroups.get(t.groupId).catch(() => null)
if (g && g.title) abGroup = g.title
}
}
} catch {}
return { openerTargetId, abGroup }
}
function postToHost(msg) {
try {
if (port) port.postMessage(msg)
@@ -116,7 +155,7 @@ function connectHost() {
} catch {}
// Tell the daemon about everything we already have attached, then attach
// anything new.
reannounceAttachedTabs()
void reannounceAttachedTabs()
void attachAllTabs()
}
@@ -130,7 +169,7 @@ async function onHostMessage(msg) {
// Daemon (re)connected — (re)attach and announce every tab so it discovers
// the user's existing tabs rather than racing an empty target list.
if (msg.method === 'attachAll') {
reannounceAttachedTabs()
void reannounceAttachedTabs()
await attachAllTabs()
return
}
@@ -172,17 +211,49 @@ function tabIdFromSession(sessionId) {
// (closed / restricted). (issues #20.1, #23)
async function recoverSessionTab(sessionId) {
const tabId = tabIdFromSession(sessionId)
if (tabId == null) return null
for (let i = 0; i < 3; i++) {
const tab = await chrome.tabs.get(tabId).catch(() => null)
if (!eligible(tab)) return null
try {
await attachTab(tabId)
if (tabs.has(tabId)) return tabId
} catch {
// mid-swap: the tab exists but isn't attachable yet — back off and retry.
// 1) Fast path: the encoded Chrome tabId still exists — re-attach it (covers
// the common renderer-process swap where the tabId is preserved, #23).
if (tabId != null) {
for (let i = 0; i < 3; i++) {
const tab = await chrome.tabs.get(tabId).catch(() => null)
if (!eligible(tab)) break // tabId is gone — fall through to targetId recovery
try {
await attachTab(tabId)
if (tabs.has(tabId)) return tabId
} catch {
// mid-swap: tab exists but isn't attachable yet — back off and retry.
}
await new Promise((r) => setTimeout(r, 120 + i * 150))
}
}
// 2) The Chrome tabId is gone, but the CDP targetId is STABLE across the nav.
// Some cross-process hops (Mercari's signin token exchange) give the tab a
// NEW tabId while keeping the same target, so `cb-tab-<oldTabId>` can't be
// recovered by tabId. Find the tab now hosting our remembered targetId via
// chrome.debugger.getTargets(), attach it, and ALIAS the dead session to it
// so the daemon's session id keeps resolving. Longer window: this hop can
// take several seconds to settle (issue #24).
const targetId = sessionTargets.get(sessionId)
if (targetId) {
for (let i = 0; i < 6; i++) {
const targets = await chrome.debugger.getTargets().catch(() => null)
const t = targets && targets.find((x) => x.id === targetId && x.tabId != null)
if (t && t.tabId != null) {
const tab = await chrome.tabs.get(t.tabId).catch(() => null)
if (eligible(tab)) {
try {
await attachTab(t.tabId)
if (tabs.has(t.tabId)) {
sessionToTab.set(sessionId, t.tabId) // alias dead session -> live tab
return t.tabId
}
} catch {
// not attachable yet — keep waiting for the swap to settle.
}
}
}
await new Promise((r) => setTimeout(r, 300 + i * 300))
}
await new Promise((r) => setTimeout(r, 120 + i * 150))
}
return null
}
@@ -217,6 +288,19 @@ async function handleForwardCdpCommand(msg) {
const params = msg?.params?.params || undefined
const sessionId = typeof msg?.params?.sessionId === 'string' ? msg.params.sessionId : undefined
// Non-CDP extension commands (ABExt.*) the daemon sends. `ungroupTab` removes a
// tab from its per-session tab group so a `keep`-marked tab is left for the user
// as a normal, ungrouped tab (the group can then be cleaned up). Best-effort.
if (method === 'ABExt.ungroupTab') {
const tabId = tabIdFromSession(sessionId) ?? tabForSession(sessionId)
if (tabId != null && chrome.tabs.ungroup) {
try {
await chrome.tabs.ungroup(tabId)
} catch {}
}
return { ungrouped: tabId ?? null }
}
// Browser-level Target methods that map onto chrome.tabs.
if (method === 'Target.createTarget') {
const url = typeof params?.url === 'string' && params.url ? params.url : 'about:blank'
@@ -343,13 +427,18 @@ async function attachTab(tabId) {
const entry = { sessionId, targetId }
tabs.set(tabId, entry)
sessionToTab.set(sessionId, tabId)
rememberSessionTarget(sessionId, targetId)
setBadge(tabId, port ? 'on' : 'connecting')
const { openerTargetId, abGroup } = await tabScopeHints(tabId)
postToHost({
method: 'forwardCDPEvent',
params: {
sessionId,
method: 'Target.attachedToTarget',
params: { sessionId, targetInfo: { ...targetInfo, attached: true } },
params: {
sessionId,
targetInfo: { ...targetInfo, attached: true, openerTargetId, abGroup },
},
},
})
return entry
@@ -391,14 +480,32 @@ async function attachAllTabs() {
}
}
function reannounceAttachedTabs() {
for (const [, entry] of tabs.entries()) {
async function reannounceAttachedTabs() {
for (const [tabId, entry] of tabs.entries()) {
// Re-send the group hint too (issue #40) so the relay can rebuild its
// targetId→group map after its own restart (createTarget tagging won't
// re-run for tabs that are already open). Include the live url/title so the
// relay's target list stays matchable by URL after a reconnect (otherwise a
// reannounced tab shows a blank url and `adopt <url>` can't find it).
const { openerTargetId, abGroup } = await tabScopeHints(tabId)
let url = ''
let title = ''
try {
const t = await chrome.tabs.get(tabId)
if (t) {
url = t.url || t.pendingUrl || ''
title = t.title || ''
}
} catch {}
postToHost({
method: 'forwardCDPEvent',
params: {
sessionId: entry.sessionId,
method: 'Target.attachedToTarget',
params: { sessionId: entry.sessionId, targetInfo: { targetId: entry.targetId, type: 'page', attached: true } },
params: {
sessionId: entry.sessionId,
targetInfo: { targetId: entry.targetId, type: 'page', url, title, attached: true, openerTargetId, abGroup },
},
},
})
}
+2 -2
View File
@@ -1,8 +1,8 @@
{
"manifest_version": 3,
"name": "chrome-use",
"version": "0.4.8",
"description": "Let chrome-use drive your logged-in Chrome \u2014 install once, no token, no per-use confirmation.",
"version": "0.4.12",
"description": "Let chrome-use drive your logged-in Chrome install once, no token, no per-use confirmation.",
"key": "MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA6vQIyscGIPYPZdSpPwPL0+0gxUROyRgCpmvCSDoc8XUm4qm97VbKnD9Ijc1lV22lNWZtE78gaRjt6BeSfuMgnBymnhLKjN1gU6AI5QUU0mrJyeHdWKvrKQR5FmsM2A7Xr1ykE2SiiS8zNUS3Y/6O5l+Nva7wrVy6E4a2dkBVQkOsu+DV+nEZvhIyuDY5D5SPXqNwUTWTaglwj5mjvHz36xSwCWlPmrtJ+ED0AUyrb2z4GIOmvk4kqtBVrh/UD058klLo4CkYOnIybB5aV6WYuwarfPY4bF/dLggPem+ewLNTUNBuwrxj/A4nUv0LJTuRO8rR7f8WR9qnRCY0Ic5saQIDAQAB",
"icons": {
"16": "icons/icon16.png",
+2 -2
View File
@@ -1,7 +1,7 @@
{
"name": "chrome-use",
"version": "1.5.0",
"description": "chrome-use \u2014 drive your real, logged-in Chrome from any AI agent, stealth by default",
"version": "1.5.27",
"description": "chrome-use drive your real, logged-in Chrome from any AI agent, stealth by default",
"type": "module",
"packageManager": "pnpm@11.1.3",
"files": [
+185 -7
View File
@@ -36,6 +36,41 @@ Refs (`@e1`, `@e2`, ...) are assigned fresh on every snapshot. They become
submits, dynamic re-renders, dialog opens. Always re-snapshot before your
next ref interaction.
> **Hard rule: snapshot-first, never screenshot-to-locate.** For form fields and
> buttons, ALWAYS `snapshot -i` and act on refs/selectors. Do **not** reach for
> `screenshot` + coordinate clicks to find or hit an element — `snapshot -i` now
> pierces **cross-origin iframes** (embedded Google Payments / Stripe / checkout /
> KYC forms) and lists their elements by `@ref`, including input values. Use
> coordinates only for canvas/WebGL, or when `snapshot` genuinely returns nothing
> for your target. Screenshots are for *visual verification you report*, never the
> agent's own input — and a full-page `screenshot` of a real retina browser is
> often too large for an image reader anyway. (If you ever feel you *need* a
> screenshot to read state or locate something, that's a bug — please file it.)
> **Snapshot-first, always. Never default to `screenshot` + coordinate clicking
> for form fields or buttons.** Run `snapshot -i` and act on `@refs`. Use
> coordinates only for canvas/WebGL, or when `snapshot` genuinely returns nothing
> for your target. This holds **even inside cross-origin embedded iframes**
> since v1.5.12 `snapshot -i` pierces out-of-process iframes (Google Payments,
> Stripe, embedded checkout/KYC) and lists their elements with refs, so
> `click @e` / `type @e` / `fill @e` work directly. A screenshot is for a genuine
> *visual* check you report to the user — not your own input. (Full-page
> screenshots of a real retina Chrome are often too large for the image reader
> anyway.) Driving off pixels on the relay also risks a coordinate event drifting
> onto the user's foreground tab — refs never do. See issue #37.
> **Two different intents — only one is discouraged.** The rule above is about
> *screenshot-to-locate* (using a picture to find/hit an element) — that's the bug.
> *screenshot-to-capture* — saving a region or element to a file as a **reusable
> image asset** (maps, charts, og-images, visual-diff baselines, report figures) —
> is fully supported and encouraged: `screenshot [selector] [--clip x,y,w,h] <file>`.
> Capturing a rendered map region to a PNG for a blog post is the right tool, not a
> smell. Screenshots are auto-downscaled to ≤2000px (longest edge) so they fit an
> image reader and their pixels line up with `click x y`; override with
> `--max-width`/`--max-height`/`--scale`. To click something you couldn't hit by
> ref, `box @ref` gives the element's CSS-px box + `centerX/centerY` to feed
> straight into `click <centerX> <centerY>` — no screenshot needed.
## Before you automate: pick the cheapest tool
Driving a browser is the heavy option. chrome-use earns its keep when you
@@ -45,6 +80,7 @@ need a **real, logged-in browser** — not for reading text off a public page.
|---|---|
| Discover what exists / find sources | `WebSearch` |
| Specific facts from a static or public page | `WebFetch` or `curl` (no browser) |
| **Structured data from a known site** (GitHub issues, Reddit/HN search, Bilibili/Twitter feed, …) — esp. behind login | `chrome-use site <name>/<cmd>` (see below) — skip snapshot+click entirely |
| Login state, interaction, JS-rendered or anti-bot pages | **chrome-use** (this skill) |
| A page the user saved before / an internal system | `chrome-use find-url <keywords>` (their bookmarks), then open it |
| The user's **own already-open, logged-in** Chrome window | the **extension connect** flow (below) |
@@ -108,7 +144,27 @@ Each `--session` that connects gets its **own colored Chrome tab group** (named
after the session) and drives only its own tabs — multiple agents share the one
real browser without cross-talk, and the user's own tabs are never grouped. CDP
drives the page without moving the user's mouse/keyboard, so it doesn't fight
them for control. **Anti-detection ranking: this real logged-in Chrome (extension
them for control.
**Strict multi-agent isolation.** A session over the relay tracks and drives
**only the tabs it created** (its own group). It does **not** adopt the user's
existing tabs, other agents' tabs, or pop-ups (e.g. an OAuth/login window — that's
the user's), so several agents (and other tools opening tabs) can work in the same
real Chrome concurrently without ever dropping or stealing each other's tabs —
another agent's tab churn can't make your bound tab vanish or drift your commands
onto the wrong page. Consequence: `tab list` shows only *your* session's tabs; to
drive a specific page, navigate to it in your own tab instead of expecting a
pre-existing or popped-up tab to appear in the list.
> **Need to read a tab the user already has open?** Use `chrome-use adopt
> <url-substring|targetId>` — it finds that pre-existing tab (the user's own, or
> another session's) across groups and drives it **without opening a new tab**.
> e.g. `adopt "claude.ai/design"` then `snapshot`/`eval`/`get text` on it. On no
> match it errors and lists the tabs it can see. This is the explicit, opt-in way
> through the isolation above (it tags the adopted tab into your group). Great for
> "read/extract from the page I'm looking at" without disturbing it.
**Anti-detection ranking: this real logged-in Chrome (extension
connect) > a headed launched browser > headless (forbidden).** A genuine human
browser has no headless/automation tells at all, so prefer it for anything
anti-bot-sensitive.
@@ -127,6 +183,19 @@ cadence, and scroll/drag ease. Default `off`; a per-navigation detector
auto-escalates pages guarded by Akamai/PerimeterX/DataDome to `human`. Leave it
on auto; force `human` only when you already know the target scores behaviour.
**Cloudflare clearance — solve once, reuse.** Passing a Cloudflare challenge
mints a `cf_clearance` cookie (HttpOnly — invisible to `eval`/`document.cookie`;
read it via `chrome-use cookies`). It's bound to your **IP + User-Agent**: reuse
the same exit IP and UA and you skip the challenge until it expires. Driving the
user's real Chrome (relay) persists it natively; for isolated sessions,
`--session-name <name>` save/restores it. Before spending effort solving, run
`chrome-use cf-status` (aliases `cf`, `clearance`): it reports whether the page
is *currently* a Cloudflare challenge and whether a still-valid `cf_clearance`
exists, with a recommendation — `proceed` (already cleared, don't re-solve),
`solve` (challenge up, no clearance), or `reissue` (clearance present but page
still blocks → IP/UA drifted, re-solve). Use it as a preflight to avoid
re-solving what you already cleared.
## Two ways to drive a page — and when to drop to `eval`
You have a **real Chrome with the user's DOM**. Two layers, mix them freely:
@@ -150,6 +219,34 @@ chrome-use eval "[...document.forms[0].elements].filter(e=>!e.validity.valid).ma
chrome-use eval "document.querySelector('#stubborn').click()" # direct DOM click, bypasses overlays
```
## Site adapters — the cheapest path for "read structured data from site X"
Before you `open` + `snapshot` + click your way through GitHub/Reddit/Bilibili/etc.,
check whether a **site adapter** already exists. An adapter is a community-written JS
function that hits the site's own JSON API *from inside your logged-in tab* and returns
clean structured data — no clicking, no scraping, no screenshots. It's the same idea as
`eval`, packaged per-site.
```bash
chrome-use site update # one-time: fetch the adapter pack (~145 cmds)
chrome-use site list # what's installed (github/issues, reddit/search, …)
chrome-use site info github/issues # an adapter's args + which domain it runs on
chrome-use site github/issues owner/repo --json # run it → JSON (navigates there for you)
```
- Positional args fill the adapter's declared args **in order**; `--key value` overrides by name.
- It navigates to the adapter's domain (reusing the current tab if you're already on it), so
login-gated feeds (`bilibili/feed`, `twitter/...`) work because they run as *you*.
- If no adapter fits, fall back to the normal `snapshot`/`eval` loop. Adapters come from the
[bb-sites](https://github.com/epiral/bb-sites) community pack; chrome-use fetches & runs them.
> **Auto-trigger — act on it.** chrome-use keeps the pack synced automatically (first use +
> weekly), and when you `open`/`navigate`/`snapshot` a page whose domain has adapters it tells
> you: a `💡 site adapters for <domain>` line on stderr, and a `siteAdapters: {domain, commands}`
> field in `--json`. **When you see that, prefer the listed `site <name>/<cmd>` over snapshot+click
> for reading data** — it's the cheaper, more reliable path and it's already installed. You don't
> need to run `site update` yourself; just use the command it names.
## Quickstart
```bash
@@ -207,7 +304,11 @@ assigned fresh on every snapshot.
For unstructured reading (no refs needed):
```bash
chrome-use get text @e1 # visible text of an element
chrome-use get text # WHOLE PAGE — all frames by default (see below)
chrome-use get text @e1 # visible text of one element (or a CSS selector)
chrome-use get text --main # main content only — skip nav/header/sidebar
chrome-use get text --pierce # read through CLOSED shadow DOM (injected panels)
chrome-use frames # list every frame + where the text lives
chrome-use get html @e1 # innerHTML
chrome-use get attr @e1 href # any attribute
chrome-use get value @e1 # input value
@@ -216,6 +317,27 @@ chrome-use get url # current URL
chrome-use get count ".item" # count matching elements
```
**Whole-page text is cross-frame by default.** `chrome-use get text` with no
selector aggregates visible text across **every** frame — top document plus
same-process child frames plus cross-origin iframes — so you never silently miss
content that lives in an iframe (Yahoo Auctions / Rakuten / Mercari shop
descriptions, embedded checkout/spec frames). Each child frame is delimited with
a `----- frame [kind] url -----` marker. You do **not** need to remember a flag —
the default already reads all frames. (`--all-frames` is still accepted as an
explicit alias.)
So: when text looks missing or wrong, you don't have to guess — just
`chrome-use get text` reads everything. To **see** the structure (which frame
holds what), run `chrome-use frames`. To **cut boilerplate** (global nav/header/
footer, "related items" sidebars), use `chrome-use get text --main`. If content
is lazy-loaded, `scroll` it into view first, then read.
**Closed shadow DOM.** Some injected UI (browser-extension debug panels, web
components) renders into a *closed* shadow root that `eval`/`innerText` cannot
read. `chrome-use get text --pierce` reads through closed shadow roots and child
documents via the CDP DOM tree — use it when content is clearly on screen (you
see it in a screenshot) but `get text`/`eval` come back empty.
## Interacting
```bash
@@ -226,6 +348,10 @@ chrome-use hover @e1 # hover
chrome-use focus @e1 # focus (useful before keyboard input)
chrome-use fill @e2 "hello" # clear then type
chrome-use type @e2 " world" # type without clearing
chrome-use type @e5 "201-0001" --key-events # real keystrokes (not insertText) —
# use for autocomplete/combobox fields that
# only react to key events (e.g. a postal box
# that auto-fills city/prefecture, Google Places)
chrome-use press Enter # press a key at current focus (down+up)
chrome-use press Control+a # key combination
chrome-use keydown d # HOLD a key down (no auto-release)
@@ -243,16 +369,38 @@ chrome-use pick @e4 --option "Europe" # ANY combobox (react-select / ARIA /
# (no silent no-op). Use this for custom
# dropdowns where `select` returns ✓ but
# changes nothing.
chrome-use upload @e5 file1.pdf # upload file(s) — NOTE: needs a --launch/direct-CDP
# session. Over the extension relay it CANNOT work
# (Chrome's chrome.debugger forbids it); chrome-use
# errors with a hint. Carry your login into a launched
# session via `cookies export` | `cookies set --curl`.
chrome-use upload @e5 file1.pdf # upload file(s) — works over the extension relay too:
# chrome.debugger forbids setFileInputFiles, so the
# file's bytes are streamed into the page and rebuilt as
# a File there (chunked under native-messaging's 1 MiB cap).
# Works on file <input>s and drop/paste composers (e.g. X).
chrome-use scroll down 500 # scroll page (up/down/left/right)
chrome-use scroll down 700 --at 640,400 # wheel at a pixel — scrolls a cross-origin
# iframe (Payments/Stripe/checkout/KYC) that
# plain page scroll can't reach
chrome-use scroll down 700 --frame 2 # scroll frame 2 from `chrome-use frames`
chrome-use scrollintoview @e1 # scroll element into view
chrome-use drag @e1 @e2 # drag and drop
```
**Cross-origin iframes (embedded payment / checkout / KYC widgets — Google
Payments, Stripe, etc.) — drive them by ref, never by screenshot.** `snapshot -i`
pierces these out-of-process iframes and lists their elements by `@ref`
(including input values); `get text --all-frames` reads their text. Then just act
on the refs: `click @e`, `type @e`, `hover @e`, `dblclick @e`, `drag @a @b` all
work into the iframe. Over the extension relay these are dispatched through the
DOM (in the element's own frame), so they hit the right element in the right tab
— a coordinate click/scroll there can drift onto whatever tab is in the
foreground, so prefer refs. For below-the-fold content in such a frame, scroll it
with `scroll down N --at x,y` (a pixel over the frame) or `--frame n`. For a
postal/autocomplete box inside the frame, `type @e "…" --key-events`.
> **Caveat: `find text "…"` can't reach into a cross-origin iframe** — it errors
> "Element not found" even though `snapshot -i` lists those nodes and
> `get text` reads them. Inside cross-origin iframes, target elements by their
> **snapshot `@ref`**, not by `find`. (`box @ref` also works on iframe refs when
> you need a coordinate fallback.)
### When refs don't work or you don't want to snapshot
Use semantic locators:
@@ -306,6 +454,13 @@ tree**, so `snapshot` comes back near-empty and refs are a dead end. `snapshot`
detects this and prints a one-line hint. Drive them the screenshot way:
```bash
chrome-use canvas list # enumerate <canvas> elements (size, type)
chrome-use canvas capture out.png # save the canvas's RENDERED pixels to PNG —
# toDataURL (full backing-store res, e.g.
# Figma 2522x1904), screenshot fallback for
# WebGL w/o preserveDrawingBuffer / tainted.
# Gets the RENDER, not hidden source data
# (those live in the app's binary store/API).
chrome-use screenshot /tmp/s.png # SEE the state (your only read path —
# eval/get text return nothing useful)
chrome-use click 640 360 # interact by viewport coordinate
@@ -636,6 +791,29 @@ chrome-use snapshot -i
chrome-use frame main # back to main frame
```
### Viewport / window size (responsive & overflow debugging)
To reproduce width-dependent bugs (responsive breakpoints, horizontal-overflow
hunts, mobile layouts) set the viewport. This is a **CDP virtual viewport**
(`Emulation.setDeviceMetricsOverride`) — it changes the layout viewport *for the
tab* without physically resizing the OS window, so it works headless **and** over
the extension relay without yanking the user's real Chrome window around.
```bash
chrome-use viewport 1280 800 # set width x height (alias: resize)
chrome-use viewport 375x812 # WxH shorthand
chrome-use viewport 375 812 --dpr 3 --mobile # retina + mobile emulation
chrome-use viewport reset # clear the override, restore real size
```
```bash
# Find what's overflowing at a narrow width:
chrome-use viewport 375 812
chrome-use eval 'document.documentElement.scrollWidth + " vs " + innerWidth'
```
`set viewport <w> <h> [scale]` is an equivalent alias.
### Dialogs
`alert` and `beforeunload` are auto-accepted so agents never block. For