Compare commits

...
13 Commits
Author SHA1 Message Date
leeguooooo d8f484eded fix(tabs): gate the about:blank cleanup to the relay only
CI / Version Sync Check (push) Has been cancelled
CI / Rust (push) Has been cancelled
CI / Rust (macos-latest - aarch64-apple-darwin) (push) Has been cancelled
CI / Rust (macos-latest - x86_64-apple-darwin) (push) Has been cancelled
CI / Rust (windows-latest - x86_64-pc-windows-msvc) (push) Has been cancelled
Release binaries / Build macOS ARM64 (push) Has been cancelled
Release binaries / Build macOS x64 (push) Has been cancelled
Release binaries / Build Linux ARM64 (push) Has been cancelled
Release binaries / Build Linux musl ARM64 (push) Has been cancelled
Release binaries / Build Linux musl x64 (push) Has been cancelled
Release binaries / Build Linux x64 (push) Has been cancelled
Release binaries / Build Windows x64 (push) Has been cancelled
CI / Native E2E Tests (push) Has been cancelled
CI / Windows Integration Test (push) Has been cancelled
CI / Global Install (macos-latest) (push) Has been cancelled
CI / Global Install (ubuntu-latest) (push) Has been cancelled
CI / Global Install (windows-latest) (push) Has been cancelled
Release binaries / Attach binaries to GitHub Release (push) Has been cancelled
The previous commit closed the leftover about:blank on ANY connection — but on a
launched browser the initial about:blank is the browser's own first tab, not
daemon scratch, so it must stay. Broke e2e_tab_ids_not_reused (launched). Gate
the cleanup on agent_group().is_some() (relay only), where the about:blank is a
tab WE created. e2e_tab_ids_not_reused passes; relay scratch-blank close intact.
2026-06-19 14:32:49 +09:00
leeguooooo 1eb40eabd5 fix(tabs): close the leftover initial about:blank when a real tab opens
A fresh session's daemon creates an about:blank scratch tab on connect; a
subsequent `tab new <url>` then opened the work tab beside it, so every session's
tab group showed a stray 'about:blank' next to the real page (e.g. about:blank +
ChatGPT). tab_new now closes any OWNED, still-blank tab once a real (non-blank)
tab exists, and re-pins the new tab. Verified live: `tab new <url>` on a fresh
session leaves only the work tab. 870 tests pass.
2026-06-19 14:06:42 +09:00
leeguooooo 601404ba72 feat(session): session stop <name> + session prune + lifecycle docs (#48)
Release binaries / Build macOS ARM64 (push) Has been cancelled
Release binaries / Build macOS x64 (push) Has been cancelled
Release binaries / Build Linux ARM64 (push) Has been cancelled
Release binaries / Build Linux musl ARM64 (push) Has been cancelled
Release binaries / Build Linux musl x64 (push) Has been cancelled
Release binaries / Build Linux x64 (push) Has been cancelled
Release binaries / Build Windows x64 (push) Has been cancelled
Release binaries / Attach binaries to GitHub Release (push) Has been cancelled
Explicit daemon reclamation to go with the v1.5.25 idle auto-shutdown:
- `session stop [name]` — stop one session daemon (default: current), graceful
  (SIGTERM → the daemon's shutdown runs close(), tidying the tabs it created).
- `session prune` — stop ALL session daemons now (clears the pile of idle
  daemons left after an automation/debug round; they respawn clean on next use).
  The __nm-host relay isn't a tracked session daemon, so the live-Chrome
  connection survives.
- --help Sessions section now documents the daemon lifecycle: spawn → 10-min idle
  auto-shutdown (AGENT_BROWSER_IDLE_TIMEOUT_MS / 0 to disable) → keep / stop / prune.

Closes #48. Verified live: session stop reclaimed a test daemon. 870 tests pass.
2026-06-18 14:44:35 +09:00
leeguooooo fd10766762 chore(ext): pack ab-connect 0.4.12 zip + crx (ABExt.ungroupTab for keep) 2026-06-18 12:16:14 +09:00
leeguooooo ba9b167ede feat(cleanup): default idle-shutdown + keep — stop leaving scratch tabs/groups behind
Release binaries / Build macOS ARM64 (push) Has been cancelled
Release binaries / Build macOS x64 (push) Has been cancelled
Release binaries / Build Linux ARM64 (push) Has been cancelled
Release binaries / Build Linux musl ARM64 (push) Has been cancelled
Release binaries / Build Linux musl x64 (push) Has been cancelled
Release binaries / Build Linux x64 (push) Has been cancelled
Release binaries / Build Windows x64 (push) Has been cancelled
Release binaries / Attach binaries to GitHub Release (push) Has been cancelled
Agents finish a task and just stop (never calling `close`), so daemons used to
run forever, leaving their per-session scratch tabs + tab group in the user's
Chrome. Two cases now handled:

- Default idle timeout (10 min; AGENT_BROWSER_IDLE_TIMEOUT_MS overrides, 0
  disables). On idle the daemon close()s the tabs IT created → the empty tab
  group is auto-removed by Chrome. Timer resets on every command, so active
  sessions are untouched. Adopted user tabs are never owned, so never closed.
- `keep` — leave the ACTIVE tab for the user: unown it (exempt from
  close/idle) + ask the extension to ungroup it (ABExt.ungroupTab → 0.4.12) so
  it becomes a normal tab. Scratch gets cleaned, deliverable tabs stay.

Also fix two clippy violations in the concurrently-landed #47 viewport code
(manual char comparison + iter().any→contains) that were failing main's CI.

ext 0.4.12: handle ABExt.ungroupTab (chrome.tabs.ungroup). 870 tests pass.
2026-06-18 12:11:59 +09:00
leeguooooo c077593e99 docs(viewport): document viewport/resize command for responsive debugging (#47) 2026-06-18 11:52:09 +09:00
leeguooooo 02e23ebe11 fix(build): include browser.rs clear_viewport/via_relay (#47) + cargo fmt
Release binaries / Build macOS ARM64 (push) Has been cancelled
Release binaries / Build macOS x64 (push) Has been cancelled
Release binaries / Build Linux ARM64 (push) Has been cancelled
Release binaries / Build Linux musl ARM64 (push) Has been cancelled
Release binaries / Build Linux musl x64 (push) Has been cancelled
Release binaries / Build Linux x64 (push) Has been cancelled
Release binaries / Build Windows x64 (push) Has been cancelled
Release binaries / Attach binaries to GitHub Release (push) Has been cancelled
v1.5.24 (d5cd9cd) shipped a commands.rs caller of `clear_viewport` but not the
browser.rs method it lives in (a concurrent in-progress #47 viewport/resize edit
was only partly staged), so main didn't compile and the format check failed.
Commit the matching browser.rs method + via_relay() helper and run cargo fmt.
Full tree builds; 863 tests pass.
2026-06-18 11:39:11 +09:00
leeguooooo d5cd9cd621 feat(canvas): extract WebGL/canvas-app content + fix site arg-order + adopt skill doc
canvas — chrome-use can now read canvas/WebGL apps (Figma, games, maps, charts,
drawing tools) that expose no DOM/refs:
  - canvas list: enumerate <canvas> (backing+CSS size, visibility, toDataUrl/tainted)
  - canvas capture [selector] [path]: save rendered pixels to PNG — toDataURL
    (full backing-store resolution), with a CDP screenshot fallback for WebGL
    without preserveDrawingBuffer or cross-origin-tainted canvases. --screenshot
    forces the screenshot path. Gets the RENDER, not hidden source data.
  Verified live: captured Figma's canvas at full 2522x1904 via toDataURL.

site — fix map_args losing the adapter's declared arg order: serde sorts @meta
keys alphabetically, so a 2-arg adapter like {projectId, path} mapped positionals
to {path, projectId} (swapped). Now parses declaration order from the raw @meta
text (Adapter.arg_order) + regression test. Affects any multi-arg adapter.

skill — core skill now documents `adopt <url|targetId>` (read a pre-existing tab,
the explicit way through strict isolation) and `canvas list`/`canvas capture` in
the canvas/WebGL section.

863 tests pass.
2026-06-18 11:31:24 +09:00
leeguooooo 284a60a54c feat(adopt): read a pre-existing tab without opening a new one
Release binaries / Build macOS ARM64 (push) Has been cancelled
Release binaries / Build macOS x64 (push) Has been cancelled
Release binaries / Build Linux ARM64 (push) Has been cancelled
Release binaries / Build Linux musl ARM64 (push) Has been cancelled
Release binaries / Build Linux musl x64 (push) Has been cancelled
Release binaries / Build Linux x64 (push) Has been cancelled
Release binaries / Build Windows x64 (push) Has been cancelled
Release binaries / Attach binaries to GitHub Release (push) Has been cancelled
New `chrome-use adopt <url-substring|targetId>`: drive a tab the user (or
another session) already has open, with ZERO new tabs. After group-scoped
isolation (#40) a session can't see foreign tabs, so adopt adds an explicit,
opt-in path:

- Relay (relay.rs): `ABRelay.getAllTargets` returns every attached target
  UNSCOPED (ignores group scoping), so the agent can find a specific tab by URL
  or targetId. +1 unit test.
- Daemon (browser.rs): `collect_all_targets` (unscoped, falls back to scoped on
  older relays) + `adopt_existing_target` — matches by exact targetId or
  case-insensitive URL substring, attaches it (the relay re-tags it into the
  adopter's group, so isolation holds), pins it; never creates a tab. On no
  match it errors AND lists the open tabs it can see, rather than launching.
  discover_and_attach_targets honors AGENT_BROWSER_ADOPT at first connect, so no
  about:blank is ever created.
- CLI (main.rs): `adopt` sets the env, forces a fresh daemon, and rewrites into
  `connect <relay-url>` (like `extension connect`) so the daemon attaches to the
  user's real Chrome before parse_command.

Extension (ab-connect 0.4.11): `reannounceAttachedTabs` now re-sends each tab's
url/title (it previously sent neither) so the relay's target list stays matchable
by URL after the MV3 service worker reconnects — otherwise reannounced tabs show
a blank url and `adopt <url>` can't find them. Repacked upload zip + crx.

Mechanism verified live (enumerated all 11 of the user's open tabs incl. the
target). 862 tests pass.
2026-06-17 21:18:01 +09:00
leeguooooo 10d196b6eb chore(ext): pack ab-connect 0.4.10 upload zip + crx (#40 group-scoped relay)
Rebuilt extensions/ab-connect.zip (key stripped for the Web Store) and the
reference .crx from the 0.4.10 source (openerTargetId + abGroup in the
synthesized Target.attachedToTarget).
2026-06-17 18:17:20 +09:00
leeguooooo 5be01e292d feat(relay): group-scoped Target.getTargets — restore follow-popup + cross-session adopt under isolation (#40)
Release binaries / Build macOS ARM64 (push) Has been cancelled
Release binaries / Build macOS x64 (push) Has been cancelled
Release binaries / Build Linux ARM64 (push) Has been cancelled
Release binaries / Build Linux musl ARM64 (push) Has been cancelled
Release binaries / Build Linux musl x64 (push) Has been cancelled
Release binaries / Build Linux x64 (push) Has been cancelled
Release binaries / Build Windows x64 (push) Has been cancelled
Release binaries / Attach binaries to GitHub Release (push) Has been cancelled
Move multi-agent isolation from blunt daemon-side filtering to relay-side
group scoping, so a session can adopt new tabs again (follow-popup, OAuth
results, cross-session adopt-by-targetId) without ever seeing the user's or
another agent's tabs.

Relay (relay.rs): track client->group (announced via new local ABRelay.setGroup,
or the first createTarget's agentGroup) and target->group (created tabs tagged
from the createTarget reply; an explicit attachToTarget tags the target into the
adopter's group = #21; a pop-up inherits its opener's group via openerTargetId).
Target.getTargets returns ONLY the requesting client's group; a client that never
announced a group (older daemon) gets the full list — fully backward-compatible.
+5 unit tests.

Daemon (browser.rs): announce_group() on connect sets relay_scoped. Adoption in
discover/resync/adopt_newly_opened is re-enabled ONLY when relay_scoped; without
it (launch / real CDP / older relay that didn't answer the announce) the daemon
keeps strict daemon-side isolation. So this can't regress the 125/125 isolation.

Extension (ab-connect 0.4.10): synthesized Target.attachedToTarget targetInfo now
carries openerTargetId (pop-ups inherit opener's group) and abGroup (the tab-group
title, so the relay re-attributes existing tabs after ITS own restart, since
createTarget tagging won't re-run). tabScopeHints().

Back-compat verified live: new daemon + OLD relay -> announce fails ->
relay_scoped=false -> strict fallback, open/eval/url all work. The new extension
(publish to CWS, strip manifest key) activates follow-popup; relay+daemon ship now.
861 tests pass.
2026-06-17 18:14:13 +09:00
leeguooooo a83d1b1df9 feat: rich-editor fill, box centers, screenshot downscale, disabled+docs (#41-#45)
Release binaries / Build macOS ARM64 (push) Has been cancelled
Release binaries / Build macOS x64 (push) Has been cancelled
Release binaries / Build Linux ARM64 (push) Has been cancelled
Release binaries / Build Linux musl ARM64 (push) Has been cancelled
Release binaries / Build Linux musl x64 (push) Has been cancelled
Release binaries / Build Linux x64 (push) Has been cancelled
Release binaries / Build Windows x64 (push) Has been cancelled
Release binaries / Attach binaries to GitHub Release (push) Has been cancelled
Dogfooding backlog from this session's embedded-form/editor work.

#41 fill on rich editors: detect CodeMirror 5 / Monaco / ProseMirror /
contenteditable and set via their own API or execCommand('insertText') so
beforeinput/input fire (a raw .value/textContent write no-op'd juejin's
CodeMirror and skipped React composers). Response echoes the `engine` used.
`fill <sel> --file <path>` / `--stdin` set large multiline text without
shell-escaping. `get value` now reads CodeMirror/Monaco/contenteditable too.

#42 screenshot --max-width/--max-height/--scale, plus a default 2000px
longest-edge cap (AGENT_BROWSER_SCREENSHOT_MAX_EDGE; 0 disables) so retina
full-page shots fit an agent's image reader and --scale 0.5 makes screenshot
px line up with click px. Annotated shots are never downscaled.

#43 `box @ref` (already a top-level alias of `get box`) now also returns
centerX/centerY/inViewport in CSS px — feed straight into `click x y` when a
ref-click no-ops (e.g. a button in a cross-origin iframe).

#44 no code change needed — disabled elements already list as
`button "Save" [disabled, ref=eN]`; the reporter's missing button was
DOM-gated on validity. Added a skill note: `find text` can't reach into a
cross-origin iframe — target those by snapshot @ref.

#45 core skill now distinguishes screenshot-to-locate (discouraged) from
screenshot-to-capture a reusable image asset via `screenshot [--clip] <file>`
(encouraged), so agents stop over-reading the prohibition.

#40 (group-scoped relay) stays deferred — needs an ab-connect extension change.

Verified live: fill --file round-trips multiline+CJK+backticks; contenteditable
engine=contenteditable + get value reads it back; box gives centerX/centerY/
inViewport; screenshot of retina example.com → 2000px; disabled button shows
[disabled]. 856 tests pass.
2026-06-17 17:49:38 +09:00
leeguooooo 50b27ac0e0 feat(site): auto-sync + auto-suggest adapters (auto-trigger)
Release binaries / Build macOS ARM64 (push) Has been cancelled
Release binaries / Build macOS x64 (push) Has been cancelled
Release binaries / Build Linux ARM64 (push) Has been cancelled
Release binaries / Build Linux musl ARM64 (push) Has been cancelled
Release binaries / Build Linux musl x64 (push) Has been cancelled
Release binaries / Build Linux x64 (push) Has been cancelled
Release binaries / Build Windows x64 (push) Has been cancelled
Release binaries / Attach binaries to GitHub Release (push) Has been cancelled
Make `site` trigger itself so an agent doesn't have to know adapters exist.

Auto-sync: the pack refreshes on first use and on a TTL (default 7d), both in
the `site` command path (blocking, fast) and as a non-blocking background task
on daemon startup — so ~/.chrome-use/sites/.index.json is always populated with
zero added latency. Tune via AGENT_BROWSER_SITES_TTL_DAYS; disable with
AGENT_BROWSER_SITES_NO_AUTO_UPDATE=1. `update` now writes .last_update + a
domain→adapters .index.json (read-only adapters ordered first).

Auto-suggest: `open`/`navigate`/`snapshot` onto a domain with adapters attaches
`siteAdapters: {domain, commands}` to the response; the CLI prints a
`💡 site adapters for <domain>` hint (stderr) and the field rides along in --json.
SKILL.md tells the agent to prefer the listed `site <name>/<cmd>` over scraping.
This keeps the 'never auto-disrupt user tabs' guarantee — it suggests, the agent
decides; nothing auto-runs on navigation.

site.rs: needs_refresh/adapters_for_domain/write_domain_index + timestamp/index
in update(). daemon.rs: background bootstrap. actions.rs: with_site_hint on
navigate + snapshot. output.rs: hint render. Verified live: open github.com →
hint leads with read-only github/issues; --json carries siteAdapters.
2026-06-17 17:15:34 +09:00
21 changed files with 1835 additions and 142 deletions
+16
View File
@@ -233,6 +233,22 @@ Positional args fill the adapter's declared args in order; `--key value` overrid
by name. Adapters are authored by the bb-sites community and remain their authors'
property — chrome-use just runs them.
**Auto-sync + auto-suggest.** You rarely type `site update` yourself: chrome-use
syncs the pack on first use and refreshes it weekly in the background (tune with
`AGENT_BROWSER_SITES_TTL_DAYS`, disable with `AGENT_BROWSER_SITES_NO_AUTO_UPDATE=1`).
And when you `open`/`snapshot` a page whose domain has adapters, chrome-use surfaces
them right in the output — a `💡 site adapters for <domain>` line, plus a
`siteAdapters` field under `--json` — so an agent reaches for the structured-data
adapter instead of scraping the DOM:
```text
$ chrome-use open https://github.com
💡 site adapters for github.com — prefer these for structured data:
github/issues, github/me, github/repo, …
e.g. chrome-use site github/issues --json
✓ GitHub
```
## Automated testing (`chrome-use test`)
Turn the repetitive "open it, click around, check it's right" work into a
+14
View File
@@ -188,6 +188,20 @@ chrome-use site bilibili/feed --json # 能用,因为走的是你的
位置参数按适配器声明的参数顺序填入;`--key value` 按名覆盖。适配器由 bb-sites 社区编写、
版权归各自作者所有 —— chrome-use 只负责运行它们。
**自动同步 + 自动提示。** 你基本不用手动 `site update`:chrome-use 首次使用时自动拉取,
之后每周后台刷新一次(`AGENT_BROWSER_SITES_TTL_DAYS` 调周期,`AGENT_BROWSER_SITES_NO_AUTO_UPDATE=1`
关闭)。而当你 `open`/`snapshot` 一个有适配器的域名时,chrome-use 会在输出里直接把可用命令
亮出来 —— 一行 `💡 site adapters for <域名>`,`--json` 下则是 `siteAdapters` 字段 —— 这样
agent 会直接改用结构化适配器,而不是去扒 DOM:
```text
$ chrome-use open https://github.com
💡 site adapters for github.com — prefer these for structured data:
github/issues, github/me, github/repo, …
e.g. chrome-use site github/issues --json
✓ GitHub
```
## 自动化测试(`chrome-use test`
把反复的「打开它、点一圈、看对不对」变成**可重跑的测试套件** —— 前端的单元测试。用 YAML 写用例;步骤复用 chrome-use 自己的命令,断言编译成一次检查:
+1 -1
View File
@@ -290,7 +290,7 @@ checksum = "613afe47fcd5fac7ccf1db93babcb082c5994d996f20b8b159f2ad1658eb5724"
[[package]]
name = "chrome-use"
version = "1.5.19"
version = "1.5.27"
dependencies = [
"aes",
"aes-gcm",
+1 -1
View File
@@ -1,6 +1,6 @@
[package]
name = "chrome-use"
version = "1.5.19"
version = "1.5.27"
edition = "2021"
description = "Fast browser automation CLI for AI agents"
license = "Apache-2.0"
+303 -35
View File
@@ -79,6 +79,12 @@ const KNOWN_COMMANDS: &[&str] = &[
"dialog",
"upload",
"site",
"box",
"adopt",
"canvas",
"viewport",
"resize",
"keep",
];
/// Levenshtein distance, capped — small inputs only (command names).
@@ -560,9 +566,37 @@ fn parse_command_inner(args: &[String], flags: &Flags) -> Result<Value, ParseErr
"fill" => {
let sel = rest.first().ok_or_else(|| ParseError::MissingArguments {
context: "fill".to_string(),
usage: "fill <selector> <text>",
usage: "fill <selector> <text> | fill <selector> --file <path> | fill <selector> --stdin",
})?;
Ok(json!({ "id": id, "action": "fill", "selector": sel, "value": rest[1..].join(" ") }))
// Large/multiline content without shell-escaping hell (issue #41):
// `fill <sel> --file <path>` reads the value from a UTF-8 file, and
// `fill <sel> --stdin` reads it from stdin — sent verbatim, so backticks,
// quotes, newlines and non-ASCII pass through untouched.
let value = match rest.get(1).copied() {
Some("--file") => {
let path = rest.get(2).ok_or(ParseError::InvalidValue {
message: "fill --file requires a path".to_string(),
usage: "fill <selector> --file <path>",
})?;
std::fs::read_to_string(path).map_err(|e| ParseError::InvalidValue {
message: format!("fill --file: cannot read {path}: {e}"),
usage: "fill <selector> --file <path>",
})?
}
Some("--stdin") => {
use std::io::Read;
let mut buf = String::new();
io::stdin()
.read_to_string(&mut buf)
.map_err(|e| ParseError::InvalidValue {
message: format!("fill --stdin: {e}"),
usage: "fill <selector> --stdin",
})?;
buf
}
_ => rest[1..].join(" "),
};
Ok(json!({ "id": id, "action": "fill", "selector": sel, "value": value }))
}
"type" => {
// `--key-events` (alias `--keys`): send real per-character keystrokes
@@ -1006,11 +1040,55 @@ fn parse_command_inner(args: &[String], flags: &Flags) -> Result<Value, ParseErr
// path: file path (contains / or . or ends with known extension)
let mut full_page = false;
let mut clip: Option<Value> = None;
let mut max_width: Option<u32> = None;
let mut max_height: Option<u32> = None;
let mut scale: Option<f64> = None;
let mut positional: Vec<&str> = Vec::new();
let mut i = 0;
// Parse a numeric value for a downscale flag (issue #42).
let parse_num = |i: &mut usize, flag: &str| -> Result<String, ParseError> {
let v = rest
.get(*i + 1)
.ok_or_else(|| ParseError::MissingArguments {
context: format!("screenshot {flag}"),
usage: "screenshot [--max-width <px>] [--max-height <px>] [--scale <0..1>]",
})?;
*i += 1;
Ok(v.to_string())
};
while i < rest.len() {
match rest[i] {
"--full" | "-f" => full_page = true,
// Downscale the saved image so retina/full-page shots fit an
// agent's image reader and screenshot px line up with click px (#42).
"--max-width" => {
let v = parse_num(&mut i, "--max-width")?;
max_width = Some(v.parse().map_err(|_| ParseError::InvalidValue {
message: format!("--max-width expects a number, got '{v}'"),
usage: "screenshot --max-width <px>",
})?);
}
"--max-height" => {
let v = parse_num(&mut i, "--max-height")?;
max_height = Some(v.parse().map_err(|_| ParseError::InvalidValue {
message: format!("--max-height expects a number, got '{v}'"),
usage: "screenshot --max-height <px>",
})?);
}
"--scale" => {
let v = parse_num(&mut i, "--scale")?;
let s: f64 = v.parse().map_err(|_| ParseError::InvalidValue {
message: format!("--scale expects a number like 0.5, got '{v}'"),
usage: "screenshot --scale <0..1>",
})?;
if s <= 0.0 || s > 1.0 {
return Err(ParseError::InvalidValue {
message: format!("--scale must be in (0, 1], got '{v}'"),
usage: "screenshot --scale <0..1>",
});
}
scale = Some(s);
}
// `--clip x,y,w,h` captures a pixel region (issue #34).
"--clip" => {
let raw = rest
@@ -1073,6 +1151,15 @@ fn parse_command_inner(args: &[String], flags: &Flags) -> Result<Value, ParseErr
if let Some(c) = clip {
cmd["clip"] = c;
}
if let Some(w) = max_width {
cmd["maxWidth"] = json!(w);
}
if let Some(h) = max_height {
cmd["maxHeight"] = json!(h);
}
if let Some(s) = scale {
cmd["scale"] = json!(s);
}
if let Some(ref fmt) = flags.screenshot_format {
cmd["format"] = json!(fmt);
}
@@ -1090,6 +1177,54 @@ fn parse_command_inner(args: &[String], flags: &Flags) -> Result<Value, ParseErr
}
Ok(cmd)
}
// Canvas/WebGL apps (Figma, games, maps, charts, drawing tools) render to a
// <canvas> with no DOM/refs to read. `canvas` extracts what's actually
// rendered: `list` enumerates canvases; `capture` saves a canvas to PNG via
// toDataURL (full backing-store resolution) with a CDP-screenshot fallback
// for WebGL contexts (no preserveDrawingBuffer) or cross-origin-tainted ones.
"canvas" => {
match rest.first().copied() {
Some("list") => Ok(json!({ "id": id, "action": "canvas_list" })),
Some("capture") => {
let force_screenshot = rest.contains(&"--screenshot");
let pos: Vec<&str> = rest[1..]
.iter()
.copied()
.filter(|a| !a.starts_with("--"))
.collect();
// `capture [selector] [path]`: a selector starts with . # @ or is
// a tag; a path contains / or ends in an image extension.
let is_path = |s: &str| {
s.contains('/')
|| s.ends_with(".png")
|| s.ends_with(".jpg")
|| s.ends_with(".jpeg")
|| s.ends_with(".webp")
};
let (selector, path) = match (pos.first(), pos.get(1)) {
(Some(a), Some(b)) => (Some(*a), Some(*b)),
(Some(a), None) if is_path(a) => (None, Some(*a)),
(Some(a), None) => (Some(*a), None),
_ => (None, None),
};
let mut cmd = json!({ "id": id, "action": "canvas_capture" });
if let Some(s) = selector {
cmd["selector"] = json!(s);
}
if let Some(p) = path {
cmd["path"] = json!(p);
}
if force_screenshot {
cmd["forceScreenshot"] = json!(true);
}
Ok(cmd)
}
_ => Err(ParseError::InvalidValue {
message: "canvas needs a subcommand".to_string(),
usage: "canvas list | canvas capture [selector] [path] [--screenshot]",
}),
}
}
"pdf" => {
let path = rest.first().ok_or_else(|| ParseError::MissingArguments {
context: "pdf".to_string(),
@@ -1235,6 +1370,16 @@ fn parse_command_inner(args: &[String], flags: &Flags) -> Result<Value, ParseErr
Ok(json!({ "id": id, "action": "site", "domain": domain, "script": script }))
}
// `adopt <url|targetId>`: the adoption happens at daemon connect (driven by
// the AGENT_BROWSER_ADOPT env main.rs set + a forced-fresh daemon), so by
// the time this command runs the tab is already attached. Resolve to a
// `url` read so the response confirms which tab got adopted.
"adopt" => Ok(json!({ "id": id, "action": "url" })),
// `keep`: leave the active tab for the user — exempt it from the daemon's
// auto-close/idle cleanup and remove it from the session's tab group.
"keep" => Ok(json!({ "id": id, "action": "keep" })),
// === Stealth self-check ===
"stealth" => {
// `stealth [status]` — local stealth self-check: mode, live probes
@@ -1558,6 +1703,13 @@ fn parse_command_inner(args: &[String], flags: &Flags) -> Result<Value, ParseErr
// === Mouse ===
"mouse" => parse_mouse(&rest, &id),
// === Viewport / window size ===
// Top-level shortcut for `set viewport` — agents (and the author, in
// issue #47) reach for `viewport`/`resize` first. Uses a CDP virtual
// viewport, so it also works on the extension relay without yanking the
// user's real window around.
"viewport" | "resize" => parse_viewport(&rest, &id),
// === Set (browser settings) ===
"set" => parse_set(&rest, &id),
@@ -3006,6 +3158,97 @@ fn parse_mouse(rest: &[&str], id: &str) -> Result<Value, ParseError> {
}
}
/// Parse a viewport / resize spec, shared by the top-level `viewport` / `resize`
/// commands and `set viewport`. Sets a CDP device-metrics override
/// (`Emulation.setDeviceMetricsOverride`) — a *virtual* viewport for the tab, so
/// it works headless AND on the extension relay without physically resizing the
/// user's real Chrome window (issue #47). Forms:
/// viewport <width> <height> [scale] [--dpr N] [--mobile]
/// viewport <width>x<height> (e.g. 1280x800)
/// viewport reset | clear (drop the override, restore real size)
fn parse_viewport(rest: &[&str], id: &str) -> Result<Value, ParseError> {
const USAGE: &str = "viewport <width> <height> [scale] [--dpr N] [--mobile] | viewport reset";
if matches!(
rest.first().copied(),
Some("reset") | Some("clear") | Some("off")
) {
return Ok(json!({ "id": id, "action": "viewport", "reset": true }));
}
// Positional (non-flag) tokens. A `WxH` token counts as one positional.
let positionals: Vec<&str> = rest
.iter()
.copied()
.filter(|a| !a.starts_with("--"))
.collect();
let (w, h, scale_tok): (i32, i32, Option<&str>) = match positionals.first() {
Some(first) if first.contains('x') || first.contains('X') => {
let mut parts = first.split(['x', 'X']);
let w = parts.next().and_then(|s| s.parse::<i32>().ok());
let h = parts.next().and_then(|s| s.parse::<i32>().ok());
match (w, h) {
(Some(w), Some(h)) => (w, h, positionals.get(1).copied()),
_ => {
return Err(ParseError::InvalidValue {
message: format!("Invalid viewport size: {}", first),
usage: USAGE,
})
}
}
}
Some(w_str) => {
let h_str = positionals.get(1).ok_or(ParseError::MissingArguments {
context: "viewport".to_string(),
usage: USAGE,
})?;
let w = w_str.parse::<i32>().map_err(|_| ParseError::InvalidValue {
message: format!("Invalid width: {}", w_str),
usage: USAGE,
})?;
let h = h_str.parse::<i32>().map_err(|_| ParseError::InvalidValue {
message: format!("Invalid height: {}", h_str),
usage: USAGE,
})?;
(w, h, positionals.get(2).copied())
}
None => {
return Err(ParseError::MissingArguments {
context: "viewport".to_string(),
usage: USAGE,
})
}
};
let mut cmd = json!({ "id": id, "action": "viewport", "width": w, "height": h });
// Device-scale-factor: positional, overridden by --dpr / --scale.
let mut scale: Option<f64> = match scale_tok {
Some(s) => Some(s.parse::<f64>().map_err(|_| ParseError::InvalidValue {
message: format!("Invalid scale: {}", s),
usage: USAGE,
})?),
None => None,
};
if let Some(i) = rest.iter().position(|a| *a == "--dpr" || *a == "--scale") {
let v = rest.get(i + 1).and_then(|s| s.parse::<f64>().ok()).ok_or(
ParseError::InvalidValue {
message: "--dpr/--scale needs a number".to_string(),
usage: USAGE,
},
)?;
scale = Some(v);
}
if let Some(s) = scale {
cmd["deviceScaleFactor"] = json!(s);
}
if rest.contains(&"--mobile") {
cmd["mobile"] = json!(true);
}
Ok(cmd)
}
fn parse_set(rest: &[&str], id: &str) -> Result<Value, ParseError> {
const VALID: &[&str] = &[
"viewport",
@@ -3020,39 +3263,8 @@ fn parse_set(rest: &[&str], id: &str) -> Result<Value, ParseError> {
];
match rest.first().copied() {
Some("viewport") => {
let w_str = rest.get(1).ok_or_else(|| ParseError::MissingArguments {
context: "set viewport".to_string(),
usage: "set viewport <width> <height> [scale]",
})?;
let h_str = rest.get(2).ok_or_else(|| ParseError::MissingArguments {
context: "set viewport".to_string(),
usage: "set viewport <width> <height> [scale]",
})?;
let w = w_str
.parse::<i32>()
.map_err(|_| ParseError::MissingArguments {
context: "set viewport".to_string(),
usage: "set viewport <width> <height> [scale]",
})?;
let h = h_str
.parse::<i32>()
.map_err(|_| ParseError::MissingArguments {
context: "set viewport".to_string(),
usage: "set viewport <width> <height> [scale]",
})?;
let mut cmd = json!({ "id": id, "action": "viewport", "width": w, "height": h });
if let Some(scale_str) = rest.get(3) {
let scale = scale_str
.parse::<f64>()
.map_err(|_| ParseError::MissingArguments {
context: "set viewport".to_string(),
usage: "set viewport <width> <height> [scale]",
})?;
cmd["deviceScaleFactor"] = json!(scale);
}
Ok(cmd)
}
// `set viewport ...` is an alias for the top-level `viewport` command.
Some("viewport") => parse_viewport(&rest[1..], id),
Some("device") => {
let dev = rest.get(1).ok_or_else(|| ParseError::MissingArguments {
context: "set device".to_string(),
@@ -5213,6 +5425,62 @@ mod tests {
assert!(result.is_err());
}
#[test]
fn test_viewport_toplevel() {
let cmd = parse_command(&args("viewport 1280 800"), &default_flags()).unwrap();
assert_eq!(cmd["action"], "viewport");
assert_eq!(cmd["width"], 1280);
assert_eq!(cmd["height"], 800);
assert!(cmd.get("deviceScaleFactor").is_none());
assert!(cmd.get("mobile").is_none());
}
#[test]
fn test_resize_alias() {
let cmd = parse_command(&args("resize 700 800"), &default_flags()).unwrap();
assert_eq!(cmd["action"], "viewport");
assert_eq!(cmd["width"], 700);
assert_eq!(cmd["height"], 800);
}
#[test]
fn test_viewport_wxh_form() {
let cmd = parse_command(&args("viewport 375x812"), &default_flags()).unwrap();
assert_eq!(cmd["action"], "viewport");
assert_eq!(cmd["width"], 375);
assert_eq!(cmd["height"], 812);
}
#[test]
fn test_viewport_dpr_and_mobile_flags() {
let cmd =
parse_command(&args("viewport 375 812 --dpr 3 --mobile"), &default_flags()).unwrap();
assert_eq!(cmd["action"], "viewport");
assert_eq!(cmd["width"], 375);
assert_eq!(cmd["height"], 812);
assert_eq!(cmd["deviceScaleFactor"], 3.0);
assert_eq!(cmd["mobile"], true);
}
#[test]
fn test_viewport_reset() {
for spec in ["viewport reset", "viewport clear", "resize reset"] {
let cmd = parse_command(&args(spec), &default_flags()).unwrap();
assert_eq!(cmd["action"], "viewport", "{spec}");
assert_eq!(cmd["reset"], true, "{spec}");
}
}
#[test]
fn test_viewport_missing_height() {
assert!(parse_command(&args("viewport 1280"), &default_flags()).is_err());
}
#[test]
fn test_viewport_invalid_width() {
assert!(parse_command(&args("viewport abc 800"), &default_flags()).is_err());
}
#[test]
fn test_find_first_no_value() {
let cmd = parse_command(&args("find first a click"), &default_flags()).unwrap();
+1 -1
View File
@@ -595,7 +595,7 @@ fn query_current_url(session: &str) -> Option<String> {
}
/// Kill a running daemon by reading its PID file and sending a kill signal.
fn kill_stale_daemon(session: &str) {
pub fn kill_stale_daemon(session: &str) {
// Remove the socket first so no new connections reach the old daemon
#[cfg(unix)]
{
+101
View File
@@ -305,6 +305,49 @@ fn run_session(args: &[String], session: &str, json_mode: bool) {
}
}
}
// Stop a specific session daemon (issue #48). Graceful: kill_stale_daemon
// sends SIGTERM first, so the daemon's shutdown handler runs `close()` and
// tidies the tabs IT created (its tab group) before exiting.
Some("stop") => {
let target = args.get(2).map(|s| s.as_str()).unwrap_or(session);
connection::kill_stale_daemon(target);
if json_mode {
print_json_value(json!({ "success": true, "data": { "stopped": target } }));
} else {
println!(
"{} stopped session daemon: {}",
color::success_indicator(),
target
);
}
}
// Reclaim ALL session daemons now (issue #48) — for clearing the pile of
// idle daemons left after a round of automation/debugging without waiting
// for the idle timeout. Each is stopped gracefully (closes its own tabs);
// they respawn clean on next use. The `__nm-host` relay is not a tracked
// session daemon, so the extension/live-Chrome connection survives.
Some("prune") => {
let sessions: Vec<String> = walk_daemons()
.sessions
.into_iter()
.map(|s| s.name)
.collect();
for s in &sessions {
connection::kill_stale_daemon(s);
}
if json_mode {
print_json_value(json!({ "success": true, "data": { "pruned": sessions } }));
} else if sessions.is_empty() {
println!("No session daemons to prune");
} else {
println!(
"{} pruned {} session daemon(s): {}",
color::success_indicator(),
sessions.len(),
sessions.join(", ")
);
}
}
None | Some(_) => {
// Just show current session
if json_mode {
@@ -840,6 +883,27 @@ fn main() {
// `info` are CLI-side (download/filesystem); `site <name>/<cmd> [args]` falls
// through to the daemon dispatch below (navigate to the adapter's domain + eval).
if clean.first().map(|s| s.as_str()) == Some("site") {
// Auto-sync the adapter pack on first use and periodically (TTL, default
// 7d) so adapters stay fresh without a manual `site update`. Skipped for an
// explicit `update` (full sync below). Best-effort: offline → cached pack.
// Disable with AGENT_BROWSER_SITES_NO_AUTO_UPDATE=1.
if clean.get(1).map(|s| s.as_str()) != Some("update") && site::needs_refresh() {
let rt = tokio::runtime::Runtime::new().expect("Failed to create tokio runtime");
match rt.block_on(site::update()) {
Ok(n) => {
eprintln!(
"{}",
color::dim(&format!("site: synced {n} adapters (auto)"))
)
}
Err(e) => eprintln!(
"{}",
color::dim(&format!(
"site: auto-sync skipped ({e}); using cached adapters"
))
),
}
}
match clean.get(1).map(|s| s.as_str()) {
Some("update") => {
let rt = tokio::runtime::Runtime::new().expect("Failed to create tokio runtime");
@@ -960,6 +1024,43 @@ fn main() {
}
}
// `adopt <url|targetId>`: read a PRE-EXISTING tab (the user's own, or another
// session's) WITHOUT opening a new one. Forces a fresh daemon and points it at
// the relay (like `extension connect`); the AGENT_BROWSER_ADOPT env makes the
// daemon's first connect ADOPT the matching tab instead of creating an
// about:blank. Rewrites into `connect <relay-url>` BEFORE parse_command so the
// daemon attaches to the user's real Chrome. Must run before parse_command.
if clean.first().map(|s| s.as_str()) == Some("adopt") {
match clean.get(1) {
Some(spec) if !spec.trim().is_empty() => {
std::env::set_var("AGENT_BROWSER_ADOPT", spec.trim());
connection::kill_stale_daemon(&flags.session);
match connect::relay_url() {
Some(url) => {
flags.cdp = Some(url.clone());
flags.auto_connect = false;
clean = vec!["connect".to_string(), url];
}
None => {
eprintln!(
"{} extension relay not connected — open Chrome with the ab-connect \
extension first (this command reads an EXISTING tab, it won't launch one).",
color::error_indicator()
);
exit(1);
}
}
}
_ => {
eprintln!(
"{} usage: chrome-use adopt <url-substring|targetId> (reads an existing tab, no new tab)",
color::error_indicator()
);
exit(2);
}
}
}
// Handle session separately (doesn't need daemon)
if clean.first().map(|s| s.as_str()) == Some("session") {
run_session(&clean, &flags.session, flags.json);
+319 -5
View File
@@ -1317,9 +1317,12 @@ pub async fn execute_command(cmd: &Value, state: &mut DaemonState) -> Value {
"evaluate" => handle_evaluate(cmd, state).await,
"site" => handle_site(cmd, state).await,
"close" => handle_close(state).await,
"keep" => handle_keep(state).await,
"stealth_status" => handle_stealth_status(state).await,
"snapshot" => handle_snapshot(cmd, state).await,
"screenshot" => handle_screenshot(cmd, state).await,
"canvas_list" => handle_canvas_list(state).await,
"canvas_capture" => handle_canvas_capture(cmd, state).await,
"click" => handle_click(cmd, state).await,
"dblclick" => handle_dblclick(cmd, state).await,
"fill" => handle_fill(cmd, state).await,
@@ -2478,7 +2481,10 @@ async fn handle_navigate(cmd: &Value, state: &mut DaemonState) -> Result<Value,
wb.navigate(url).await?;
let new_url = wb.get_url().await.unwrap_or_else(|_| url.to_string());
let title = wb.get_title().await.unwrap_or_default();
return Ok(json!({ "url": new_url, "title": title }));
return Ok(with_site_hint(
json!({ "url": new_url, "title": title }),
url,
));
}
}
@@ -2545,7 +2551,7 @@ async fn handle_navigate(cmd: &Value, state: &mut DaemonState) -> Result<Value,
.unwrap_or(false)
{
if let Ok(Some(switched)) = mgr.reuse_tab_for_url(url).await {
return Ok(switched);
return Ok(with_site_hint(switched, url));
}
}
@@ -2553,7 +2559,34 @@ async fn handle_navigate(cmd: &Value, state: &mut DaemonState) -> Result<Value,
// Adaptive humanize: sample the freshly loaded page for known behavioural
// anti-bot vendors and escalate this session to Human if any are present.
detect_and_set_humanize(mgr).await;
Ok(result)
Ok(with_site_hint(result, url))
}
/// Annotate a navigation/snapshot result with the `site` adapters available for
/// the page's domain (auto-trigger): when you land on e.g. github.com, the
/// response carries `siteAdapters: { domain, commands: ["github/issues", …] }` so
/// the agent reaches for a structured-data adapter instead of scraping. No-op
/// when nothing matches or the pack isn't synced yet.
fn with_site_hint(mut result: Value, fallback_url: &str) -> Value {
let url = result
.get("url")
.and_then(|v| v.as_str())
.unwrap_or(fallback_url);
let host = url::Url::parse(url)
.ok()
.and_then(|u| u.host_str().map(String::from));
if let Some(host) = host {
let adapters = crate::site::adapters_for_domain(&host);
if !adapters.is_empty() {
if let Some(obj) = result.as_object_mut() {
obj.insert(
"siteAdapters".to_string(),
json!({ "domain": host, "commands": adapters }),
);
}
}
}
result
}
/// After navigation, probe the page for known anti-bot vendor fingerprints
@@ -2823,6 +2856,34 @@ async fn handle_stealth_status(state: &DaemonState) -> Result<Value, String> {
}))
}
/// `keep` — leave the ACTIVE tab for the user: stop owning it (so the daemon's
/// `close()`/idle-shutdown won't close it) and best-effort remove it from this
/// session's tab group so it looks like a normal user tab. The "leave for the
/// user" half of the auto-close-on-idle cleanup: scratch tabs get closed, tabs
/// the agent explicitly `keep`s stay. (Adopted user tabs are never owned, so
/// they're already safe.)
async fn handle_keep(state: &mut DaemonState) -> Result<Value, String> {
let mgr = state.browser.as_mut().ok_or("Browser not launched")?;
let target_id = mgr.active_target_id()?.to_string();
let session_id = mgr.active_session_id()?.to_string();
let was_owned = mgr.unown_target(&target_id);
// Best-effort: ask the extension to ungroup the tab (relay only; no-ops on a
// launched browser or an older extension that doesn't know ABExt.ungroupTab).
let _ = mgr
.client
.send_command_typed::<_, Value>(
"ABExt.ungroupTab",
&json!({ "sessionId": session_id, "targetId": target_id }),
None,
)
.await;
Ok(json!({
"kept": target_id,
"wasOwned": was_owned,
"note": "tab left for the user — exempt from auto-close, removed from the session tab group",
}))
}
async fn handle_close(state: &mut DaemonState) -> Result<Value, String> {
if let Some(ref mgr) = state.browser {
if let Some(ref session_name) = state.session_name {
@@ -2936,6 +2997,13 @@ async fn handle_snapshot(cmd: &Value, state: &mut DaemonState) -> Result<Value,
let ref_count = refs.len();
let mut out = json!({ "snapshot": tree, "origin": url, "refs": refs });
// Auto-trigger: if this domain has site adapters, surface them so the agent
// pulls structured data instead of walking the tree. `with_site_hint` reads
// the `url` field, so pass it under that key.
if let Some(hint) = with_site_hint(json!({ "url": url }), &url).get("siteAdapters") {
out["siteAdapters"] = hint.clone();
}
// Canvas/WebGL apps (games, map/3D viewers, drawing tools) paint to a
// <canvas> and expose almost no accessibility tree, so `snapshot` comes back
// near-empty and agents get stuck looking for refs that will never exist
@@ -3077,7 +3145,40 @@ async fn handle_screenshot(cmd: &Value, state: &mut DaemonState) -> Result<Value
)
.await?;
// Downscale the saved image so retina/full-page shots fit an agent's image
// reader and screenshot pixels line up with `click x y` CSS px (issue #42).
// Explicit --scale / --max-width / --max-height win; otherwise a default cap
// (2000px longest edge, AGENT_BROWSER_SCREENSHOT_MAX_EDGE overrides, 0 = off)
// applies. Annotated shots are left untouched so ref overlays stay aligned.
let mut resized: Option<(u32, u32)> = None;
if !annotate {
let scale = cmd.get("scale").and_then(|v| v.as_f64());
let max_w = cmd
.get("maxWidth")
.and_then(|v| v.as_u64())
.map(|v| v as u32);
let max_h = cmd
.get("maxHeight")
.and_then(|v| v.as_u64())
.map(|v| v as u32);
let default_edge = if scale.is_none() && max_w.is_none() && max_h.is_none() {
std::env::var("AGENT_BROWSER_SCREENSHOT_MAX_EDGE")
.ok()
.and_then(|s| s.parse::<u32>().ok())
.or(Some(2000))
.filter(|&e| e > 0)
} else {
None
};
resized = downscale_screenshot(&result.path, scale, max_w, max_h, default_edge);
}
let mut response = json!({ "path": absolutize_saved_path(&result.path) });
if let Some((w, h)) = resized {
response["width"] = json!(w);
response["height"] = json!(h);
response["resized"] = json!(true);
}
if !result.annotations.is_empty() {
response["annotations"] = serde_json::to_value(&result.annotations)
.map_err(|e| format!("Failed to serialize annotations: {}", e))?;
@@ -3093,6 +3194,204 @@ async fn handle_screenshot(cmd: &Value, state: &mut DaemonState) -> Result<Value
Ok(response)
}
/// Downscale a saved screenshot in place (issue #42). Resolves the target longest
/// edge from `scale` (fraction of current), explicit `max_w`/`max_h` caps, or a
/// `default_edge` cap — whichever yields the smaller image. Only ever shrinks;
/// no-op (returns None) if the image is already within bounds or can't be read.
/// Returns the new (width, height) when it actually resized.
fn downscale_screenshot(
path: &str,
scale: Option<f64>,
max_w: Option<u32>,
max_h: Option<u32>,
default_edge: Option<u32>,
) -> Option<(u32, u32)> {
let img = image::open(path).ok()?;
let (w, h) = (img.width(), img.height());
if w == 0 || h == 0 {
return None;
}
// Collect candidate scale factors (≤ 1.0); the smallest wins.
let mut factor = 1.0f64;
if let Some(s) = scale {
factor = factor.min(s);
}
if let Some(mw) = max_w {
if w > mw {
factor = factor.min(mw as f64 / w as f64);
}
}
if let Some(mh) = max_h {
if h > mh {
factor = factor.min(mh as f64 / h as f64);
}
}
if let Some(edge) = default_edge {
let longest = w.max(h);
if longest > edge {
factor = factor.min(edge as f64 / longest as f64);
}
}
if factor >= 1.0 {
return None; // already within bounds — never upscale
}
let nw = ((w as f64 * factor).round() as u32).max(1);
let nh = ((h as f64 * factor).round() as u32).max(1);
let resized = img.resize(nw, nh, image::imageops::FilterType::Lanczos3);
resized.save(path).ok()?;
Some((resized.width(), resized.height()))
}
/// `canvas list` — enumerate <canvas> elements (size, visibility, whether
/// toDataURL is usable) so an agent can pick one to capture on a canvas/WebGL app
/// where snapshot/DOM reads see nothing.
async fn handle_canvas_list(state: &DaemonState) -> Result<Value, String> {
let mgr = state.browser.as_ref().ok_or("Browser not launched")?;
let js = r#"(() => [...document.querySelectorAll('canvas')].map((c, i) => {
const r = c.getBoundingClientRect();
let toDataUrl = true, tainted = false;
try { c.toDataURL('image/png'); } catch (e) { toDataUrl = false; tainted = true; }
return {
index: i,
backingWidth: c.width, backingHeight: c.height,
cssWidth: Math.round(r.width), cssHeight: Math.round(r.height),
visible: r.width > 0 && r.height > 0 && r.bottom > 0 && r.top < innerHeight,
id: c.id || null, className: c.className || null,
toDataUrl, tainted,
};
}))()"#;
let canvases = mgr.evaluate(js, None).await?;
Ok(json!({ "canvases": canvases }))
}
/// `canvas capture [selector] [path]` — save a canvas's rendered pixels to PNG.
/// Prefers `toDataURL` (full backing-store resolution); falls back to a CDP
/// screenshot of the canvas element when toDataURL is blank (WebGL without
/// preserveDrawingBuffer, e.g. Figma) or throws (cross-origin tainted). This is
/// how chrome-use "sees" canvas/WebGL apps that expose no DOM or refs.
async fn handle_canvas_capture(cmd: &Value, state: &mut DaemonState) -> Result<Value, String> {
let selector = cmd
.get("selector")
.and_then(|v| v.as_str())
.map(String::from);
let force_screenshot = cmd
.get("forceScreenshot")
.and_then(|v| v.as_bool())
.unwrap_or(false);
let out_path = cmd.get("path").and_then(|v| v.as_str()).map(String::from);
let id = cmd.get("id").and_then(|v| v.as_str()).unwrap_or("0");
// Probe: locate the canvas, get its bbox, and try toDataURL (unless forced).
let probe = {
let mgr = state.browser.as_ref().ok_or("Browser not launched")?;
let sel_lit = match &selector {
Some(s) => serde_json::to_string(s).unwrap_or_else(|_| "null".to_string()),
None => "null".to_string(),
};
let js = format!(
r#"(() => {{
const sel = {sel_lit};
const c = sel ? document.querySelector(sel)
: [...document.querySelectorAll('canvas')].sort((a,b)=>(b.width*b.height)-(a.width*a.height))[0];
if (!c) return {{ found: false, count: document.querySelectorAll('canvas').length }};
const r = c.getBoundingClientRect();
let dataUrl = null, err = null;
if ({try_data}) {{ try {{ dataUrl = c.toDataURL('image/png'); }} catch (e) {{ err = String(e && e.message || e); }} }}
return {{ found: true, w: c.width, h: c.height, x: r.x, y: r.y, cw: r.width, ch: r.height, dataUrl, err }};
}})()"#,
try_data = !force_screenshot
);
mgr.evaluate(&js, None).await?
};
if !probe
.get("found")
.and_then(|v| v.as_bool())
.unwrap_or(false)
{
let count = probe.get("count").and_then(|v| v.as_u64()).unwrap_or(0);
return Err(format!(
"canvas: no canvas matched{} ({count} canvas element(s) on the page — try `canvas list`)",
selector
.as_deref()
.map(|s| format!(" `{s}`"))
.unwrap_or_default()
));
}
let backing_w = probe.get("w").and_then(|v| v.as_u64()).unwrap_or(0);
let backing_h = probe.get("h").and_then(|v| v.as_u64()).unwrap_or(0);
// Decode toDataURL if present; a WebGL canvas without preserveDrawingBuffer
// returns a blank PNG (tiny when compressed), so reject suspiciously small
// results and fall back to the screenshot path.
let decoded: Option<Vec<u8>> = probe
.get("dataUrl")
.and_then(|v| v.as_str())
.and_then(|u| u.split_once(",").map(|(_, b)| b.to_string()))
.and_then(|b64| {
base64::Engine::decode(&base64::engine::general_purpose::STANDARD, b64.as_bytes()).ok()
});
let use_data = !force_screenshot && decoded.as_ref().map(|d| d.len() > 1024).unwrap_or(false);
if use_data {
let bytes = decoded.unwrap();
let path = out_path.unwrap_or_else(|| {
std::env::temp_dir()
.join(format!("canvas-{id}.png"))
.to_string_lossy()
.into_owned()
});
std::fs::write(&path, &bytes).map_err(|e| format!("canvas: write {path}: {e}"))?;
Ok(json!({
"path": absolutize_saved_path(&path),
"method": "toDataURL",
"width": backing_w,
"height": backing_h,
}))
} else {
// Fallback: screenshot the canvas element (or its bbox clip).
let clip = if selector.is_none() {
let x = probe.get("x").and_then(|v| v.as_f64()).unwrap_or(0.0);
let y = probe.get("y").and_then(|v| v.as_f64()).unwrap_or(0.0);
let cw = probe.get("cw").and_then(|v| v.as_f64()).unwrap_or(0.0);
let ch = probe.get("ch").and_then(|v| v.as_f64()).unwrap_or(0.0);
Some((x, y, cw, ch))
} else {
None
};
let options = ScreenshotOptions {
selector: selector.clone(),
path: out_path,
clip,
..ScreenshotOptions::default()
};
let mgr = state.browser.as_ref().ok_or("Browser not launched")?;
let session_id = mgr.active_session_id()?.to_string();
let result = screenshot::take_screenshot(
&mgr.client,
&session_id,
&state.ref_map,
&options,
&state.iframe_sessions,
)
.await?;
let why = probe
.get("err")
.and_then(|v| v.as_str())
.map(|e| format!("toDataURL failed ({e})"))
.unwrap_or_else(|| {
"toDataURL blank/unavailable (WebGL no preserveDrawingBuffer)".into()
});
Ok(json!({
"path": absolutize_saved_path(&result.path),
"method": "screenshot",
"note": format!("captured rendered pixels via screenshot — {why}"),
}))
}
}
async fn handle_click(cmd: &Value, state: &mut DaemonState) -> Result<Value, String> {
// First-class coordinate click (issue #8.4): click a raw viewport point with
// no element resolution. Parsed from `click <x> <y>` / `click --coords x,y`.
@@ -3251,7 +3550,7 @@ async fn handle_fill(cmd: &Value, state: &mut DaemonState) -> Result<Value, Stri
let mgr = state.browser.as_ref().ok_or("Browser not launched")?;
let session_id = mgr.active_session_id()?.to_string();
interaction::fill(
let engine = interaction::fill(
&mgr.client,
&session_id,
&state.ref_map,
@@ -3260,7 +3559,9 @@ async fn handle_fill(cmd: &Value, state: &mut DaemonState) -> Result<Value, Stri
&state.iframe_sessions,
)
.await?;
Ok(json!({ "filled": selector }))
// Echo the input path used (input/contenteditable/codemirror5/monaco/select)
// so the agent can confirm a rich editor was handled, not silently no-op'd (#41).
Ok(json!({ "filled": selector, "engine": engine }))
}
async fn handle_type(cmd: &Value, state: &mut DaemonState) -> Result<Value, String> {
@@ -4980,6 +5281,19 @@ async fn handle_tab_close(cmd: &Value, state: &mut DaemonState) -> Result<Value,
async fn handle_viewport(cmd: &Value, state: &mut DaemonState) -> Result<Value, String> {
let mgr = state.browser.as_ref().ok_or("Browser not launched")?;
// `viewport reset` clears the device-metrics override and restores the real
// layout viewport (the launched window's size, or — over the relay — the
// user's actual Chrome window).
if cmd.get("reset").and_then(|v| v.as_bool()).unwrap_or(false) {
mgr.clear_viewport().await?;
state.viewport = None;
if let Some(ref server) = state.stream_server {
server.set_viewport(1280, 720).await;
}
return Ok(json!({ "reset": true }));
}
let width = cmd.get("width").and_then(|v| v.as_i64()).unwrap_or(1280) as i32;
let height = cmd.get("height").and_then(|v| v.as_i64()).unwrap_or(720) as i32;
let scale = cmd
+272 -45
View File
@@ -509,6 +509,13 @@ pub struct BrowserManager {
/// for `RELAY_PRUNE_MISSES` consecutive snapshots; any snapshot that includes
/// it resets the counter. Keyed by stable target_id.
relay_target_misses: HashMap<String, u32>,
/// Whether the relay accepted this session's group announcement and is
/// therefore scoping `Target.getTargets` to our own tab group (issue #40).
/// When true the daemon can safely adopt new targets again (follow-popup,
/// cross-session adopt) — the relay has already filtered out foreign tabs.
/// When false (launch-on-real-CDP, or an older relay that didn't answer the
/// announce) the daemon keeps strict daemon-side isolation.
relay_scoped: bool,
next_tab_id: u32,
/// Whether to enable the CDP `Runtime` domain (console / error / exception capture).
/// OFF by default for stealth: a live `Runtime.enable` is a detectable CDP signal
@@ -646,6 +653,7 @@ impl BrowserManager {
created_targets: HashSet::new(),
active_target_id: None,
relay_target_misses: HashMap::new(),
relay_scoped: false,
next_tab_id: 1,
capture_console: console_capture_enabled(),
};
@@ -749,6 +757,7 @@ impl BrowserManager {
created_targets: HashSet::new(),
active_target_id: None,
relay_target_misses: HashMap::new(),
relay_scoped: false,
next_tab_id: 1,
capture_console: console_capture_enabled(),
};
@@ -815,6 +824,106 @@ impl BrowserManager {
Ok(by_id.into_values().collect())
}
/// Every tab the relay knows, UNSCOPED (ignores group scoping) — for explicit
/// cross-group adoption (`chrome-use adopt`). Falls back to the scoped
/// `collect_page_targets` on a relay/browser that doesn't support the
/// unscoped query. Retries a few times over the relay (discovery is eventual).
async fn collect_all_targets(&self) -> Result<Vec<TargetInfo>, String> {
let rounds = if crate::connect::relay_url().is_some() {
3
} else {
1
};
let mut by_id: HashMap<String, TargetInfo> = HashMap::new();
let mut any_ok = false;
for i in 0..rounds {
if i > 0 {
tokio::time::sleep(Duration::from_millis(150)).await;
}
if let Ok(result) = self
.client
.send_command_typed::<_, GetTargetsResult>(
"ABRelay.getAllTargets",
&json!({}),
None,
)
.await
{
any_ok = true;
for t in result.target_infos.into_iter().filter(should_track_target) {
by_id.entry(t.target_id.clone()).or_insert(t);
}
}
}
if any_ok {
Ok(by_id.into_values().collect())
} else {
// Older relay without ABRelay.getAllTargets → best-effort scoped list.
self.collect_page_targets().await
}
}
/// Adopt a specific pre-existing tab matched by `spec` (an exact CDP
/// `targetId`, or a case-insensitive substring of the tab URL) WITHOUT opening
/// a new tab — for `chrome-use adopt`. Attaches it (the relay tags it into our
/// group), tracks + pins it. Errors if nothing matches (never creates a tab).
async fn adopt_existing_target(&mut self, spec: &str) -> Result<(), String> {
let all = self.collect_all_targets().await?;
let spec_l = spec.to_lowercase();
let target = all
.iter()
.find(|t| t.target_id == spec)
.or_else(|| all.iter().find(|t| t.url.to_lowercase().contains(&spec_l)))
.ok_or_else(|| {
let mut open: Vec<String> = all
.iter()
.map(|t| {
let u = if t.url.len() > 80 {
&t.url[..80]
} else {
&t.url
};
u.to_string()
})
.collect();
open.sort();
open.dedup();
format!(
"adopt: no open tab matching `{spec}` (by targetId or URL substring).\n\
{} tab(s) the extension can see:\n {}",
open.len(),
open.join("\n ")
)
})?
.clone();
let attach: AttachToTargetResult = self
.client
.send_command_typed(
"Target.attachToTarget",
&AttachToTargetParams {
target_id: target.target_id.clone(),
flatten: true,
},
None,
)
.await?;
let tab_id = self.assign_tab_id();
self.pages.push(PageInfo {
tab_id,
label: None,
target_id: target.target_id.clone(),
session_id: attach.session_id.clone(),
url: target.url.clone(),
title: sanitize_title(&target.title),
target_type: target.target_type.clone(),
});
self.active_page_index = self.pages.len() - 1;
self.pin_active_target();
self.enable_domains(&attach.session_id).await?;
Ok(())
}
async fn discover_and_attach_targets(&mut self) -> Result<(), String> {
self.client
.send_command_typed::<_, Value>(
@@ -824,6 +933,21 @@ impl BrowserManager {
)
.await?;
// Announce our group FIRST so the relay scopes the getTargets below to our
// own tab group (issue #40). On a launched browser this is a no-op.
let scoped = self.announce_group().await;
// `chrome-use adopt <spec>`: adopt a specific PRE-EXISTING tab instead of
// creating one — true zero-new-tab reading of the user's own tab. The
// directive rides in via env so it takes effect at first connect (before
// any about:blank would be made). If nothing matches, error out rather
// than fall back to creating a tab.
if let Ok(spec) = std::env::var("AGENT_BROWSER_ADOPT") {
if !spec.trim().is_empty() {
return self.adopt_existing_target(spec.trim()).await;
}
}
let page_targets: Vec<TargetInfo> = self.collect_page_targets().await?;
if page_targets.is_empty() {
@@ -870,19 +994,19 @@ impl BrowserManager {
self.active_page_index = 0;
self.pin_active_target();
self.enable_domains(&attach_result.session_id).await?;
} else if self.agent_group().is_some() {
// STRICT MULTI-AGENT ISOLATION (relay / the user's real Chrome).
// `page_targets` here are the USER's and OTHER agents' tabs. A tab
// group belongs to exactly ONE agent, so this session must NOT adopt
// any of them — it tracks ONLY tabs it creates (its own colored group)
// plus popups it opens. Adopting foreign tabs is precisely what let
// another concurrent agent's tab churn drop the tab we were driving and
// drift eval/click onto the wrong page (multi-agent failure). Open our
// own dedicated background tab in the session's group and pin it; the
// user's / other agents' tabs stay invisible to us.
} else if self.agent_group().is_some() && !scoped {
// STRICT MULTI-AGENT ISOLATION fallback (relay, but the group announce
// didn't take — e.g. an older relay). Without relay-side scoping,
// `page_targets` could be the USER's and OTHER agents' tabs, so this
// session must NOT adopt any of them — adopting foreign tabs is what let
// another agent's tab churn drop the tab we were driving (multi-agent
// failure). Open our own dedicated background tab and pin it instead.
self.tab_new(None, None).await?;
} else {
// A browser WE launched: every tab is ours, so adopt them all.
// Either a browser WE launched (every tab is ours) or the relay has
// scoped getTargets to our own tab group (#40) — so `page_targets` are
// all ours: adopt them (this restores follow-popup + cross-session
// adopt under isolation, since foreign tabs were already filtered out).
for target in &page_targets {
let attach_result: AttachToTargetResult = self
.client
@@ -1422,6 +1546,13 @@ impl BrowserManager {
.ok_or_else(|| "No active page".to_string())
}
/// Stop owning a tab — drop it from `created_targets` so it survives `close()`
/// and idle-shutdown (the agent is leaving it for the user). Returns true if it
/// was owned. Used by `keep`.
pub fn unown_target(&mut self, target_id: &str) -> bool {
self.created_targets.remove(target_id)
}
/// Returns true if this manager was connected via CDP (as opposed to local launch).
pub fn is_cdp_connection(&self) -> bool {
self.browser_process.is_none()
@@ -1580,7 +1711,11 @@ impl BrowserManager {
// in the synthesized targetInfo), so don't adopt anything: the agent drives
// only tabs it explicitly created, and pop-ups (e.g. an OAuth/login window)
// are the user's. A launched browser (every tab ours) still follows pop-ups.
if self.agent_group().is_some() {
// Strict isolation only when on the relay WITHOUT group scoping: there a
// pop-up can't be told apart from a foreign tab, so adopt nothing. When the
// relay IS scoping (#40), getTargets returns only our group, so a tab that
// appeared after our own action is genuinely ours (a pop-up) — adopt it.
if self.agent_group().is_some() && !self.relay_scoped {
return None;
}
let result: GetTargetsResult = self
@@ -1658,16 +1793,17 @@ impl BrowserManager {
.collect();
let live_ids: HashSet<String> = live.iter().map(|t| t.target_id.clone()).collect();
let on_relay = self.agent_group().is_some();
// When the relay scopes getTargets to our group (#40), `live` is already
// only our own tabs, so adopting unknown ones is safe (a freshly-opened
// pop-up). Without scoping, keep strict isolation: never adopt a tab we
// didn't create — it belongs to the user or another agent.
let strict_isolation = on_relay && !self.relay_scoped;
for target in &live {
if self.update_page_target_info(target) {
continue;
}
// STRICT MULTI-AGENT ISOLATION: on the relay (the user's real Chrome,
// shared with other agents), NEVER adopt a tab this session didn't
// create — it belongs to the user or another agent's group. Only a
// browser we launched (every tab ours) adopts unknown targets.
if on_relay {
if strict_isolation {
continue;
}
let attach_result: AttachToTargetResult = match self
@@ -1821,9 +1957,17 @@ impl BrowserManager {
/// CDP browser the endpoint is strict, so we must NOT send the custom param —
/// hence `None` there. We detect the relay by matching our `ws_url` against
/// the live relay URL the native-messaging host published.
/// Whether this manager is driving the user's real Chrome through the
/// `ab-connect` extension relay (vs. a browser we launched or a direct CDP
/// endpoint). Detected by matching our `ws_url` against the live relay URL
/// the native-messaging host published. Used to avoid relay-unsafe CDP that
/// would disturb the user's window (e.g. Browser.setContentsSize, issue #47).
fn via_relay(&self) -> bool {
crate::connect::relay_url().as_deref() == Some(self.ws_url.as_str())
}
fn agent_group(&self) -> Option<String> {
let via_relay = crate::connect::relay_url().as_deref() == Some(self.ws_url.as_str());
if !via_relay {
if !self.via_relay() {
return None;
}
let name = DAEMON_SESSION
@@ -1837,6 +1981,25 @@ impl BrowserManager {
}
}
/// Tell the relay which tab group this session owns so it can scope
/// `Target.getTargets` to us (issue #40). Only meaningful on the relay; a
/// no-op (returns false) on a launched/real-CDP connection. Sets and returns
/// `relay_scoped`: when true, the daemon can trust getTargets to contain only
/// our group and re-enable adopting new tabs (pop-ups, cross-session adopt).
async fn announce_group(&mut self) -> bool {
let Some(group) = self.agent_group() else {
self.relay_scoped = false;
return false;
};
let ok = self
.client
.send_command_typed::<_, Value>("ABRelay.setGroup", &json!({ "group": group }), None)
.await
.is_ok();
self.relay_scoped = ok;
ok
}
pub async fn tab_new(
&mut self,
url: Option<&str>,
@@ -1907,6 +2070,46 @@ impl BrowserManager {
self.active_page_index = index;
self.pin_active_target();
// Close the daemon's leftover initial `about:blank` scratch tab once this
// real tab exists, so the session's tab group isn't left showing a stray
// blank page beside the work tab (every group otherwise carried one). Only
// on the RELAY — there the about:blank is a tab WE created as scratch; on a
// launched browser the initial about:blank is the browser's own first tab,
// which we must not close. Only when opening a real url, OWNED, still-blank.
if target_url != "about:blank" && self.agent_group().is_some() {
if let Some(new_tid) = self.pages.get(index).map(|p| p.target_id.clone()) {
let blanks: Vec<String> = self
.pages
.iter()
.filter(|p| {
p.target_id != new_tid
&& self.created_targets.contains(&p.target_id)
&& (p.url == "about:blank" || p.url.is_empty())
})
.map(|p| p.target_id.clone())
.collect();
for tid in blanks {
let _ = self
.client
.send_command_typed::<_, Value>(
"Target.closeTarget",
&CloseTargetParams {
target_id: tid.clone(),
},
None,
)
.await;
self.created_targets.remove(&tid);
self.remove_page_by_target_id(&tid);
}
// Removing earlier pages shifts indices — re-pin the new tab.
if let Some(i) = self.pages.iter().position(|p| p.target_id == new_tid) {
self.active_page_index = i;
self.pin_active_target();
}
}
}
Ok(json!({
"tabId": format_tab_id(tab_id),
"label": label,
@@ -2013,32 +2216,39 @@ impl BrowserManager {
.await?;
// Screencast captures the actual content area, not the emulated CSS
// viewport, so resize the content area to match.
if let Ok(target_id) = self.active_target_id() {
if let Ok(window_info) = self
.client
.send_command(
"Browser.getWindowForTarget",
Some(json!({ "targetId": target_id })),
None,
)
.await
{
if let Some(window_id) = window_info.get("windowId").and_then(|v| v.as_i64()) {
if let Err(e) = self
.client
.send_command(
"Browser.setContentsSize",
Some(json!({
"windowId": window_id,
"width": width,
"height": height,
})),
None,
)
.await
{
eprintln!("Browser.setContentsSize failed (experimental CDP): {e}");
// viewport, so resize the content area to match — but ONLY for a browser
// we launched. Over the ab-connect relay the "window" is the user's real
// Chrome window, and Browser.setContentsSize would physically resize it
// (issue #47) — the exact thing the CDP device-metrics override exists to
// avoid. The Emulation override above already gives the tab the requested
// CSS viewport without touching the OS window, so skip the resize there.
if !self.via_relay() {
if let Ok(target_id) = self.active_target_id() {
if let Ok(window_info) = self
.client
.send_command(
"Browser.getWindowForTarget",
Some(json!({ "targetId": target_id })),
None,
)
.await
{
if let Some(window_id) = window_info.get("windowId").and_then(|v| v.as_i64()) {
if let Err(e) = self
.client
.send_command(
"Browser.setContentsSize",
Some(json!({
"windowId": window_id,
"width": width,
"height": height,
})),
None,
)
.await
{
eprintln!("Browser.setContentsSize failed (experimental CDP): {e}");
}
}
}
}
@@ -2047,6 +2257,22 @@ impl BrowserManager {
Ok(())
}
/// Clear the CDP device-metrics override (`viewport reset`), restoring the
/// tab's real layout viewport. Never touches the OS window, so it is safe on
/// the relay (we never physically resized the user's window — see
/// `set_viewport`).
pub async fn clear_viewport(&self) -> Result<(), String> {
let session_id = self.active_session_id()?;
self.client
.send_command(
"Emulation.clearDeviceMetricsOverride",
Some(json!({})),
Some(session_id),
)
.await?;
Ok(())
}
pub async fn set_user_agent(&self, user_agent: &str) -> Result<(), String> {
let session_id = self.active_session_id()?;
self.client
@@ -2630,6 +2856,7 @@ async fn initialize_lightpanda_manager(
created_targets: HashSet::new(),
active_target_id: None,
relay_target_misses: HashMap::new(),
relay_scoped: false,
next_tab_id: 1,
capture_console: console_capture_enabled(),
};
+25 -6
View File
@@ -21,6 +21,16 @@ pub async fn run_daemon(session: &str) {
// (via the ab-connect extension) land in a per-session Chrome tab group.
let _ = super::browser::DAEMON_SESSION.set(session.to_string());
// Bootstrap / refresh the site-adapter pack in the background (first-run +
// periodic TTL). This populates ~/.chrome-use/sites/.index.json so navigation
// can auto-suggest `site` commands for the page you land on, with zero added
// latency to any command. Best-effort; offline is a no-op.
if crate::site::needs_refresh() {
tokio::spawn(async {
let _ = crate::site::update().await;
});
}
let socket_dir = get_daemon_socket_dir();
if !socket_dir.exists() {
let _ = fs::create_dir_all(&socket_dir);
@@ -120,12 +130,21 @@ pub async fn run_daemon(session: &str) {
}
}
// Auto-shutdown the daemon after this many ms of inactivity (no commands received).
// Disabled when unset or 0.
let idle_timeout_ms = env::var("AGENT_BROWSER_IDLE_TIMEOUT_MS")
.ok()
.and_then(|s| s.parse::<u64>().ok())
.filter(|&ms| ms > 0);
// Auto-shutdown the daemon after this many ms of inactivity (no commands
// received). On shutdown the daemon closes the tabs IT created (its per-session
// tab group), so an agent that finishes a task and just stops — without ever
// calling `close` — no longer leaves a pile of scratch tabs and a lingering
// tab group in the user's Chrome. The timer resets on every command, so active
// sessions are never interrupted; only genuinely-idle ones clean up.
//
// Defaults to 10 minutes. Set AGENT_BROWSER_IDLE_TIMEOUT_MS to override, or 0
// to disable (keep the daemon alive forever — the old behaviour). Adopted
// tabs (the user's own, via `adopt`) are never closed: only `created_targets`.
const DEFAULT_IDLE_TIMEOUT_MS: u64 = 600_000;
let idle_timeout_ms = match env::var("AGENT_BROWSER_IDLE_TIMEOUT_MS") {
Ok(s) => s.trim().parse::<u64>().ok().filter(|&ms| ms > 0),
Err(_) => Some(DEFAULT_IDLE_TIMEOUT_MS),
};
let result = run_socket_server(
&socket_path,
+30 -4
View File
@@ -1529,9 +1529,27 @@ pub async fn get_element_input_value(
.send_command_typed(
"Runtime.callFunctionOn",
&CallFunctionOnParams {
function_declaration:
"function() { return typeof this.value === 'string' ? this.value : ''; }"
.to_string(),
// Read rich-editor content too (issue #41): CodeMirror 5 / Monaco
// keep their text in a model, not `.value`; contenteditable keeps
// it as innerText. Falls back to `.value` for plain inputs.
function_declaration: r#"function() {
const el = this;
const cm5 = el.closest && el.closest('.CodeMirror');
if (cm5 && cm5.CodeMirror) return cm5.CodeMirror.getValue();
if (window.monaco && monaco.editor) {
try {
const eds = monaco.editor.getEditors ? monaco.editor.getEditors() : [];
const ed = eds.find(e => e.getDomNode && e.getDomNode().contains(el)) || eds[0];
if (ed) return ed.getValue();
const m = monaco.editor.getModels ? monaco.editor.getModels() : [];
if (m[0]) return m[0].getValue();
} catch (e) {}
}
if (typeof el.value === 'string') return el.value;
if (el.isContentEditable) return el.innerText;
return '';
}"#
.to_string(),
object_id: Some(object_id),
arguments: None,
return_by_value: Some(true),
@@ -1609,7 +1627,15 @@ pub async fn get_element_bounding_box(
&CallFunctionOnParams {
function_declaration: r#"function() {
const r = this.getBoundingClientRect();
return { x: r.x, y: r.y, width: r.width, height: r.height };
const inViewport = r.bottom > 0 && r.right > 0
&& r.top < (innerHeight || document.documentElement.clientHeight)
&& r.left < (innerWidth || document.documentElement.clientWidth);
return {
x: r.x, y: r.y, width: r.width, height: r.height,
centerX: Math.round(r.x + r.width / 2),
centerY: Math.round(r.y + r.height / 2),
inViewport,
};
}"#
.to_string(),
object_id: Some(object_id),
+52 -17
View File
@@ -579,7 +579,7 @@ pub async fn fill(
selector_or_ref: &str,
value: &str,
iframe_sessions: &HashMap<String, String>,
) -> Result<(), String> {
) -> Result<String, String> {
let (object_id, effective_session_id) = resolve_element_object_id(
client,
session_id,
@@ -590,14 +590,15 @@ pub async fn fill(
.await?;
// Emulate a real edit so framework-controlled inputs (React/Vue) and
// site-side listeners actually see the change (issue #25): the old path set
// `this.value` directly and used Input.insertText, which left React's
// internal value-tracker out of sync and never fired change/blur — so
// dependent logic (e.g. Mercari's postal-code → 都道府県 autocomplete) never
// ran even though the value was visible. Set the value through the element's
// PROTOTYPE setter (which React's _valueTracker hooks), then dispatch
// input → change → blur/focusout. `type <sel> <text>` remains for sites that
// need per-keystroke events.
// site-side listeners actually see the change (issue #25): set the value
// through the element's PROTOTYPE setter (which React's _valueTracker hooks),
// then dispatch input → changeblur/focusout. Beyond plain inputs, detect
// rich editors and use their own API/events (issue #41): CodeMirror 5 and
// Monaco have a model that `.value`/`textContent` can't touch; ProseMirror /
// contenteditable need `execCommand('insertText')` so beforeinput/input fire
// (a raw `textContent =` corrupts PM's doc and skips React composers).
// Returns the engine used so the caller can report it. `type <sel> <text>`
// remains for sites that need per-keystroke events.
let fill_js = format!(
r#"function() {{
const el = this;
@@ -605,13 +606,43 @@ pub async fn fill(
try {{ el.focus(); }} catch (e) {{}}
const tag = el.tagName;
const fire = (type, ctor) => el.dispatchEvent(new (ctor || Event)(type, {{ bubbles: true }}));
if (tag === 'SELECT') {{
el.value = v; fire('input'); fire('change'); return true;
// CodeMirror 5: a hidden <textarea> inside .CodeMirror with a live instance.
const cm5 = el.closest && el.closest('.CodeMirror');
if (cm5 && cm5.CodeMirror) {{ cm5.CodeMirror.setValue(v); return 'codemirror5'; }}
// Monaco: global `monaco`; prefer the editor whose DOM contains el.
if (window.monaco && monaco.editor) {{
try {{
const eds = monaco.editor.getEditors ? monaco.editor.getEditors() : [];
const ed = eds.find(e => e.getDomNode && e.getDomNode().contains(el)) || eds[0];
if (ed) {{ ed.setValue(v); return 'monaco'; }}
const models = monaco.editor.getModels ? monaco.editor.getModels() : [];
if (models[0]) {{ models[0].setValue(v); return 'monaco'; }}
}} catch (e) {{}}
}}
if (tag === 'SELECT') {{ el.value = v; fire('input'); fire('change'); return 'select'; }}
if (el.isContentEditable) {{
el.textContent = v; fire('input', window.InputEvent || Event); fire('change');
try {{ el.blur(); }} catch (e) {{}} fire('focusout'); return true;
// ProseMirror / contenteditable: select-all then insertText fires
// beforeinput/input that PM and React composers listen for.
let ok = false;
try {{
const sel = window.getSelection();
const range = document.createRange();
range.selectNodeContents(el);
sel.removeAllRanges();
sel.addRange(range);
ok = document.execCommand('insertText', false, v);
}} catch (e) {{}}
if (!ok) {{ el.textContent = v; fire('input', window.InputEvent || Event); }}
fire('change');
try {{ el.blur(); }} catch (e) {{}}
fire('focusout');
return ok ? 'contenteditable' : 'contenteditable-fallback';
}}
const proto = tag === 'TEXTAREA' ? window.HTMLTextAreaElement.prototype
: window.HTMLInputElement.prototype;
const desc = Object.getOwnPropertyDescriptor(proto, 'value');
@@ -623,13 +654,13 @@ pub async fn fill(
fire('change');
try {{ el.blur(); }} catch (e) {{}}
fire('focusout'); // blur-triggered lookups/validation
return true;
return 'input';
}}"#,
val = serde_json::to_string(value).unwrap_or_default()
);
client
.send_command_typed::<_, Value>(
let result: EvaluateResult = client
.send_command_typed(
"Runtime.callFunctionOn",
&CallFunctionOnParams {
function_declaration: fill_js,
@@ -642,7 +673,11 @@ pub async fn fill(
)
.await?;
Ok(())
Ok(result
.result
.value
.and_then(|v| v.as_str().map(String::from))
.unwrap_or_else(|| "input".to_string()))
}
#[allow(clippy::too_many_arguments)]
+273 -4
View File
@@ -52,6 +52,22 @@ pub struct RelayState {
pending: HashMap<i64, (ClientId, Value)>,
/// monotonic source of relay-global command ids
next_global_id: i64,
/// Group-scoped isolation (issue #40). A tab group belongs to exactly one
/// agent/session; the relay scopes `Target.getTargets` per client to its own
/// group so the daemon can safely adopt new tabs (follow-popup, cross-session
/// adopt) without ever seeing the user's or another agent's tabs.
///
/// clientId -> group name. A client that never announced a group (older
/// daemon) is absent here and gets the full, UNSCOPED target list — so this
/// is fully backward-compatible.
client_groups: HashMap<ClientId, String>,
/// targetId -> group name. Created tabs are tagged from `Target.createTarget`'s
/// `agentGroup`; an explicitly adopted tab is tagged to the adopter; a pop-up
/// inherits its opener's group (needs the extension to report `openerTargetId`).
target_group: HashMap<String, String>,
/// relay-global id of an in-flight `Target.createTarget` -> the `agentGroup`
/// it carried, so the reply's `targetId` can be tagged with that group.
pending_create: HashMap<i64, String>,
}
/// What to do with a raw CDP command received from a `CdpClient`.
@@ -95,6 +111,7 @@ impl RelayState {
/// `pending` entries don't leak.
pub fn drop_client(&mut self, client_id: ClientId) {
self.pending.retain(|_, (cid, _)| *cid != client_id);
self.client_groups.remove(&client_id);
}
/// Route a raw CDP command `{id, method, params?, sessionId?}` from a
@@ -123,12 +140,28 @@ impl RelayState {
"jsVersion": ""
}
})),
// Non-CDP control message: a daemon announces which tab group
// (session) it owns, so getTargets can be scoped to it (issue #40).
"ABRelay.setGroup" => {
if let Some(g) = params.get("group").and_then(|g| g.as_str()) {
if !g.is_empty() {
self.client_groups.insert(client_id, g.to_string());
}
}
ClientRoute::Local(json!({ "id": id, "result": {} }))
}
// Discovery is best-effort and event-driven in real CDP; abs only
// reads the getTargets result, so an empty ack is enough here.
"Target.setDiscoverTargets" | "Target.setAutoAttach" => {
ClientRoute::Local(json!({ "id": id, "result": {} }))
}
"Target.getTargets" => {
// Unscoped discovery for EXPLICIT cross-group adoption (`chrome-use
// adopt`): returns every target the extension has attached, ignoring
// group scoping, so an agent can find a specific pre-existing tab (the
// user's, another session's) by URL/targetId and adopt it. Isolation
// is preserved because the daemon only acts on the one tab it then
// attaches (which the relay re-tags into the adopter's group).
"ABRelay.getAllTargets" => {
let infos: Vec<Value> = self
.targets
.values()
@@ -136,15 +169,44 @@ impl RelayState {
.collect();
ClientRoute::Local(json!({ "id": id, "result": { "targetInfos": infos } }))
}
"Target.getTargets" => {
// Scope to the client's own group when it announced one; an
// un-announced (legacy) client gets the full list (back-compat).
let scoped = self.client_groups.get(&client_id).cloned();
let infos: Vec<Value> = self
.targets
.iter()
.filter(|(tid, _)| match &scoped {
Some(g) => self
.target_group
.get(*tid)
.map(|tg| tg == g)
.unwrap_or(false),
None => true,
})
.map(|(_, t)| t.target_info.clone())
.collect();
ClientRoute::Local(json!({ "id": id, "result": { "targetInfos": infos } }))
}
"Target.attachToTarget" => {
let target_id = params
.get("targetId")
.and_then(|t| t.as_str())
.unwrap_or("");
match self.targets.get(target_id) {
Some(entry) => ClientRoute::Local(
json!({ "id": id, "result": { "sessionId": entry.session_id } }),
),
Some(entry) => {
let session_id = entry.session_id.clone();
// Explicitly adopting a target makes it this client's
// (cross-session adopt, #21) — tag it into the adopter's
// group so it stays in that client's scoped getTargets and
// isn't churn-pruned.
if let Some(g) = self.client_groups.get(&client_id).cloned() {
self.target_group.insert(target_id.to_string(), g);
}
ClientRoute::Local(
json!({ "id": id, "result": { "sessionId": session_id } }),
)
}
None => ClientRoute::Local(json!({
"id": id,
"error": { "code": -32602, "message": format!("No such target {target_id}") }
@@ -157,6 +219,18 @@ impl RelayState {
self.next_global_id += 1;
let gid = self.next_global_id;
self.pending.insert(gid, (client_id, id));
// Remember the group a createTarget carries so the reply's
// targetId can be tagged to the creating session (issue #40).
if method == "Target.createTarget" {
if let Some(g) = params.get("agentGroup").and_then(|g| g.as_str()) {
if !g.is_empty() {
self.pending_create.insert(gid, g.to_string());
self.client_groups
.entry(client_id)
.or_insert_with(|| g.to_string());
}
}
}
ClientRoute::Forward(json!({
"id": gid,
"method": "forwardCDPCommand",
@@ -201,6 +275,17 @@ impl RelayState {
&& msg.get("method").is_none()
{
let gid = msg.get("id").and_then(|i| i.as_i64());
// A createTarget reply: tag the new tab's targetId with the group the
// command carried, so it lands in the creating session's scope (#40).
if let Some(g) = gid.and_then(|g| self.pending_create.remove(&g)) {
if let Some(tid) = msg
.get("result")
.and_then(|r| r.get("targetId"))
.and_then(|t| t.as_str())
{
self.target_group.insert(tid.to_string(), g);
}
}
let (to, orig_id) = match gid.and_then(|g| self.pending.remove(&g)) {
Some((client_id, orig)) => (Some(client_id), orig),
// No mapping (stale/unknown id) — fall back to broadcasting with
@@ -241,6 +326,27 @@ impl RelayState {
.and_then(|s| s.as_str())
.unwrap_or("")
.to_string();
// Attribute the tab to a group for scoping (issue #40),
// unless we already know it (createTarget tag). An
// explicit `abGroup` from the extension wins; otherwise a
// pop-up inherits its opener's group via `openerTargetId`.
if !self.target_group.contains_key(tid) {
if let Some(g) = info
.get("abGroup")
.and_then(|g| g.as_str())
.filter(|g| !g.is_empty())
{
self.target_group.insert(tid.to_string(), g.to_string());
} else if let Some(opener) = info
.get("openerTargetId")
.and_then(|o| o.as_str())
.filter(|o| !o.is_empty())
{
if let Some(g) = self.target_group.get(opener).cloned() {
self.target_group.insert(tid.to_string(), g);
}
}
}
self.targets.insert(
tid.to_string(),
TargetEntry {
@@ -255,6 +361,15 @@ impl RelayState {
"Target.detachedFromTarget" => {
let gone = inner_params.get("sessionId").and_then(|s| s.as_str());
if let Some(gone) = gone {
let gone_tids: Vec<String> = self
.targets
.iter()
.filter(|(_, e)| e.session_id == gone)
.map(|(tid, _)| tid.clone())
.collect();
for tid in gone_tids {
self.target_group.remove(&tid);
}
self.targets.retain(|_, e| e.session_id != gone);
}
return vec![];
@@ -557,4 +672,158 @@ mod tests {
_ => panic!("expected ToExt"),
}
}
// === Group-scoped isolation (issue #40) ===
/// Drive the real create path: announce group, createTarget(agentGroup), feed
/// the ext reply (tags target→group) + the attachedToTarget event (creates the
/// entry). Returns nothing; mutates `s`.
fn create_in_group(s: &mut RelayState, client: ClientId, group: &str, tid: &str, sid: &str) {
s.route_client_command(
client,
&json!({ "id": 1, "method": "ABRelay.setGroup", "params": { "group": group } }),
);
let route = s.route_client_command(
client,
&json!({ "id": 2, "method": "Target.createTarget",
"params": { "url": "about:blank", "agentGroup": group } }),
);
let gid = match route {
ClientRoute::Forward(env) => env["id"].as_i64().unwrap(),
_ => panic!("createTarget must forward"),
};
s.handle_ext_message(&json!({ "id": gid, "result": { "targetId": tid } }), "");
s.handle_ext_message(
&json!({ "method": "forwardCDPEvent", "params": {
"method": "Target.attachedToTarget",
"params": { "sessionId": sid, "targetInfo": {
"targetId": tid, "type": "page", "url": "about:blank", "attached": true } } } }),
"",
);
}
fn get_target_ids(s: &mut RelayState, client: ClientId) -> Vec<String> {
match s.route_client_command(client, &json!({ "id": 9, "method": "Target.getTargets" })) {
ClientRoute::Local(v) => v["result"]["targetInfos"]
.as_array()
.unwrap()
.iter()
.map(|t| t["targetId"].as_str().unwrap().to_string())
.collect(),
_ => panic!("getTargets must be local"),
}
}
#[test]
fn get_targets_is_scoped_to_each_clients_group() {
let mut s = RelayState::new();
create_in_group(&mut s, 1, "agent-a", "ta", "sa");
create_in_group(&mut s, 2, "agent-b", "tb", "sb");
// Each client sees ONLY its own group's tab — never the other agent's.
assert_eq!(get_target_ids(&mut s, 1), vec!["ta"]);
assert_eq!(get_target_ids(&mut s, 2), vec!["tb"]);
}
#[test]
fn legacy_client_without_group_sees_all_targets() {
let mut s = RelayState::new();
create_in_group(&mut s, 1, "agent-a", "ta", "sa");
create_in_group(&mut s, 2, "agent-b", "tb", "sb");
// Client 3 never announced a group → full, unscoped list (back-compat).
let mut all = get_target_ids(&mut s, 3);
all.sort();
assert_eq!(all, vec!["ta", "tb"]);
}
#[test]
fn popup_inherits_opener_group_and_is_visible_to_that_client_only() {
let mut s = RelayState::new();
create_in_group(&mut s, 1, "agent-a", "ta", "sa");
create_in_group(&mut s, 2, "agent-b", "tb", "sb");
// A pop-up that agent-a's tab opened: extension reports openerTargetId=ta.
s.handle_ext_message(
&json!({ "method": "forwardCDPEvent", "params": {
"method": "Target.attachedToTarget",
"params": { "sessionId": "sp", "targetInfo": {
"targetId": "tp", "type": "page", "url": "https://oauth.example/",
"attached": true, "openerTargetId": "ta" } } } }),
"",
);
// Only agent-a sees the pop-up; agent-b never does.
let mut a = get_target_ids(&mut s, 1);
a.sort();
assert_eq!(a, vec!["ta", "tp"]);
assert_eq!(get_target_ids(&mut s, 2), vec!["tb"]);
}
#[test]
fn explicit_attach_tags_target_into_adopter_group() {
let mut s = RelayState::new();
// A pre-existing, ungrouped tab the extension reported (e.g. user's tab).
s.handle_ext_message(
&json!({ "method": "forwardCDPEvent", "params": {
"method": "Target.attachedToTarget",
"params": { "sessionId": "su", "targetInfo": {
"targetId": "tu", "type": "page", "url": "https://user.example/", "attached": true } } } }),
"",
);
// Client 1 (group agent-a) explicitly adopts it by targetId (#21).
s.route_client_command(
1,
&json!({ "id": 1, "method": "ABRelay.setGroup", "params": { "group": "agent-a" } }),
);
s.route_client_command(
1,
&json!({ "id": 2, "method": "Target.attachToTarget", "params": { "targetId": "tu" } }),
);
// Now it's in agent-a's scope and survives the scoped getTargets.
assert_eq!(get_target_ids(&mut s, 1), vec!["tu"]);
// A different agent still doesn't see it.
s.route_client_command(
2,
&json!({ "id": 1, "method": "ABRelay.setGroup", "params": { "group": "agent-b" } }),
);
assert!(get_target_ids(&mut s, 2).is_empty());
}
#[test]
fn get_all_targets_is_unscoped() {
let mut s = RelayState::new();
create_in_group(&mut s, 1, "agent-a", "ta", "sa");
create_in_group(&mut s, 2, "agent-b", "tb", "sb");
// Client 1's scoped getTargets sees only its own group...
assert_eq!(get_target_ids(&mut s, 1), vec!["ta"]);
// ...but ABRelay.getAllTargets returns EVERY target regardless of group
// (for explicit cross-group adoption).
let all = match s
.route_client_command(1, &json!({ "id": 1, "method": "ABRelay.getAllTargets" }))
{
ClientRoute::Local(v) => {
let mut ids: Vec<String> = v["result"]["targetInfos"]
.as_array()
.unwrap()
.iter()
.map(|t| t["targetId"].as_str().unwrap().to_string())
.collect();
ids.sort();
ids
}
_ => panic!("getAllTargets must be local"),
};
assert_eq!(all, vec!["ta", "tb"]);
}
#[test]
fn detach_clears_target_group() {
let mut s = RelayState::new();
create_in_group(&mut s, 1, "agent-a", "ta", "sa");
assert_eq!(get_target_ids(&mut s, 1), vec!["ta"]);
s.handle_ext_message(
&json!({ "method": "forwardCDPEvent", "params": {
"method": "Target.detachedFromTarget", "params": { "sessionId": "sa" } } }),
"",
);
assert!(get_target_ids(&mut s, 1).is_empty());
assert!(!s.target_group.contains_key("ta"));
}
}
+82 -5
View File
@@ -186,6 +186,26 @@ pub fn print_response_with_opts(resp: &Response, action: Option<&str>, opts: &Ou
}
if let Some(data) = &resp.data {
// Auto-trigger: when you land on / read a page whose domain has site
// adapters, surface them so the agent pulls structured data via
// `chrome-use site <name>/<cmd>` instead of scraping the DOM. (In --json
// mode this same info rides along in the `siteAdapters` field above.)
if let Some(hint) = data.get("siteAdapters") {
let domain = hint.get("domain").and_then(|v| v.as_str()).unwrap_or("");
let cmds: Vec<&str> = hint
.get("commands")
.and_then(|v| v.as_array())
.map(|a| a.iter().filter_map(|v| v.as_str()).collect())
.unwrap_or_default();
if !cmds.is_empty() {
eprintln!("💡 site adapters for {domain} — prefer these for structured data:");
eprintln!(" {}", color::dim(&cmds.join(", ")));
eprintln!(
" {}",
color::dim(&format!("e.g. chrome-use site {} --json", cmds[0]))
);
}
}
// A click that opened a new tab: surface it so the agent doesn't read the
// unchanged old page as a failed click (issue #24-A).
if let Some(opened) = data.get("openedTab") {
@@ -469,6 +489,17 @@ pub fn print_response_with_opts(resp: &Response, action: Option<&str>, opts: &Ou
println!("y: {}", y);
println!("width: {}", w);
println!("height: {}", h);
if let (Some(cx), Some(cy)) = (
obj.get("centerX").and_then(|v| v.as_i64()),
obj.get("centerY").and_then(|v| v.as_i64()),
) {
// Echoed in click-ready CSS px so the agent can paste straight
// into `click <centerX> <centerY>` (issue #43).
println!("center: {} {}", cx, cy);
}
if let Some(iv) = obj.get("inViewport").and_then(|v| v.as_bool()) {
println!("inViewport: {}", iv);
}
}
return;
}
@@ -1470,9 +1501,20 @@ Examples:
chrome-use fill - Clear and fill an input field
Usage: chrome-use fill <selector> <text>
chrome-use fill <selector> --file <path>
chrome-use fill <selector> --stdin
Clears the input field and fills it with the specified text.
This replaces any existing content in the field.
Clears the field and fills it with the text, replacing existing content.
Works on rich editors too (issue #41): CodeMirror 5, Monaco, ProseMirror and
plain contenteditable are detected and set via their own API / input events,
not a raw `.value` write and the response echoes which `engine` was used.
For framework inputs (React/Vue/Angular) the value goes through the native
setter so the form registers it (no more "pristine" Save no-ops).
Options:
--file <path> Read the value from a UTF-8 file (large/multiline text,
backticks/quotes/newlines/non-ASCII no shell escaping)
--stdin Read the value from stdin
Global Options:
--json Output as JSON
@@ -1481,7 +1523,8 @@ Global Options:
Examples:
chrome-use fill "#email" "user@example.com"
chrome-use fill @e3 "Hello World"
chrome-use fill "input[name='search']" "query"
chrome-use fill ".CodeMirror" --file ./article.md # set a CodeMirror editor
cat post.md | chrome-use fill @e7 --stdin
"##
}
"type" => {
@@ -1883,6 +1926,13 @@ Options:
--full, -f Capture full page (not just viewport)
[selector] Capture just an element (CSS or @ref), e.g. `screenshot ".header" h.png`
--clip <x,y,w,h> Capture a pixel region, e.g. `screenshot --clip 0,0,200,40 corner.png`
--max-width <px> Downscale so the image's width px (preserves aspect)
--max-height <px> Downscale so the image's height px
--scale <0..1> Downscale by a factor, e.g. 0.5 (DPR-1, so screenshot px
line up 1:1 with `click x y`)
Default: capped at 2000px longest edge unless overridden
(AGENT_BROWSER_SCREENSHOT_MAX_EDGE; 0 disables). Annotated
shots are never downscaled, so ref overlays stay aligned.
--annotate Overlay numbered labels on interactive elements.
Each label [N] corresponds to ref @eN from snapshot.
Prints a legend mapping labels to element roles/names.
@@ -1905,6 +1955,8 @@ Examples:
chrome-use screenshot --full ./full-page.png
chrome-use screenshot ".header .indicator" corner.png # just one element
chrome-use screenshot --clip 1600,0,200,40 corner.png # a pixel region
chrome-use screenshot --scale 0.5 ./half.png # DPR-1: screenshot px == click px
chrome-use screenshot --max-width 1400 ./shot.png # cap width for image readers
chrome-use screenshot --annotate # Labeled screenshot + legend
chrome-use screenshot --annotate ./page.png # Save annotated screenshot
chrome-use screenshot --annotate --json # JSON output with annotations
@@ -3243,7 +3295,8 @@ Core Commands:
click <sel|x y> Click element/@ref, or a viewport coordinate
dblclick <sel> Double-click element
type <sel> <text> Type into element
fill <sel> <text> Clear and fill
fill <sel> <text> Clear and fill (handles CodeMirror/Monaco/ProseMirror/
contenteditable; `--file <path>`/`--stdin` for large text)
press <key> [--hold <ms>] Press key (Enter, Tab, Control+a). --hold keeps it
down <ms> then releases precise (in-daemon), for
games/charge: `press d --hold 800`
@@ -3263,11 +3316,19 @@ Core Commands:
scroll <dir> [px] Scroll (up/down/left/right)
scrollintoview <sel> Scroll element into view
wait <sel|ms> Wait for element or time
screenshot [path] Take screenshot
screenshot [path] Take screenshot (auto-downscaled to 2000px long edge;
--max-width/--max-height/--scale to override)
pdf <path> Save as PDF
canvas list List <canvas> elements (size, type) on the page
canvas capture [sel] [path] Save a canvas's rendered pixels to PNG for
WebGL/canvas apps (Figma, games, maps, charts) that
expose no DOM. toDataURL, with a screenshot fallback.
snapshot Accessibility tree with refs (for AI)
eval <js> Run JavaScript
connect <port|url> Connect to browser via CDP
keep Leave the active tab for the user exempt it from
auto-close/idle cleanup + remove it from the session
tab group (so scratch tabs get cleaned, this one stays)
close [--all] Close browser (--all closes every session)
Navigation:
@@ -3277,6 +3338,8 @@ Navigation:
Get Info: chrome-use get <what> [selector]
text, html, value, attr <name>, title, url, count, box, styles, cdp-url
box <sel> x,y,width,height,centerX,centerY,inViewport in CSS px (feed
centerX/centerY into `click x y`); value reads CodeMirror/Monaco too
text (no selector = whole page, all frames), text --main, frames (list)
Check State: chrome-use is <what> <selector>
@@ -3319,6 +3382,10 @@ Tabs:
stable targetId, no reload preserves in-page state
open <url> --reuse-tab Reuse an existing tab on that URL instead of spawning
a duplicate (matches origin+path; preserves state)
adopt <url|targetId> Read a PRE-EXISTING tab (the user's own, or another
session's) WITHOUT opening a new one matches by URL
substring or stable targetId, then drives it. e.g.
`adopt "github.com/owner/repo"`
Diff:
diff snapshot Compare current vs last snapshot
@@ -3388,11 +3455,21 @@ Confirmation:
Sessions:
session Show current session name
session list List active sessions
session stop [name] Stop one session daemon (default: current) graceful,
closes the tabs it created
session prune Stop ALL session daemons now (closes their tabs; they
respawn clean on next use). For clearing idle daemons.
sessions List running session daemons (alias of daemon status)
daemon status List running session daemons (+ relay state)
daemon restart Kill all session daemons; keeps the extension relay
up. Clears stale/cross-leaked state after an upgrade.
Lifecycle: each --session <name> spawns a background daemon that drives that
session's tabs. A daemon auto-shuts-down after 10 min idle (no commands)
AGENT_BROWSER_IDLE_TIMEOUT_MS overrides, 0 disables and on shutdown closes
the scratch tabs IT created (its tab group). Use `keep` to leave a tab for the
user (exempt from auto-close), `session stop/prune` to reclaim now.
Chat (AI):
chat <message> Send a natural language instruction (single-shot)
chat Start interactive chat (REPL mode when stdin is a TTY)
+206 -8
View File
@@ -31,6 +31,10 @@ fn dirs_home() -> Option<PathBuf> {
pub struct Adapter {
pub meta: Value,
pub func_src: String,
/// The adapter's declared `args` keys in DECLARATION order. Parsed from the
/// raw @meta text because `serde_json` sorts object keys alphabetically, which
/// would otherwise scramble positional-arg mapping for multi-arg adapters.
pub arg_order: Vec<String>,
}
impl Adapter {
@@ -111,7 +115,66 @@ pub fn parse_adapter(raw: &str, spec: &str) -> Result<Adapter, String> {
if func_src.is_empty() {
return Err(format!("site: {spec} has no function body after @meta"));
}
Ok(Adapter { meta, func_src })
let arg_order = arg_order_from_meta(&raw[start..end]);
Ok(Adapter {
meta,
func_src,
arg_order,
})
}
/// Extract the `args` object's keys in DECLARATION order from the raw @meta JSON
/// text (serde sorts them, losing order). Brace/string-aware: finds the `"args"`
/// value object and collects only its top-level keys.
fn arg_order_from_meta(meta_json: &str) -> Vec<String> {
let bytes = meta_json.as_bytes();
// Locate the `"args"` key, then the `{` that opens its value object.
let Some(args_pos) = meta_json.find("\"args\"") else {
return Vec::new();
};
let Some(brace_off) = meta_json[args_pos..].find('{') else {
return Vec::new();
};
let open = args_pos + brace_off;
let mut keys = Vec::new();
let mut depth = 0i32;
let mut in_str = false;
let mut esc = false;
let mut cur = String::new();
let mut last_str: Option<String> = None;
for &b in bytes.iter().skip(open) {
if in_str {
if esc {
esc = false;
} else if b == b'\\' {
esc = true;
} else if b == b'"' {
in_str = false;
last_str = Some(std::mem::take(&mut cur));
} else {
cur.push(b as char);
}
continue;
}
match b {
b'"' => in_str = true,
b'{' => depth += 1,
b'}' => {
depth -= 1;
if depth == 0 {
break; // end of the args object
}
}
// A `:` at depth 1 means the preceding string was a key of `args`.
b':' if depth == 1 => {
if let Some(k) = last_str.take() {
keys.push(k);
}
}
_ => {}
}
}
keys
}
/// Build the JS to eval: `(<adapter function>)(<args JSON>)`. The adapter's
@@ -199,22 +262,137 @@ pub async fn update() -> Result<usize, String> {
count += 1;
}
}
// Build the domain→adapters index and stamp the sync time so navigation can
// suggest adapters (auto-trigger) and `needs_refresh` can pace re-syncs.
write_domain_index(&dir);
if let Some(p) = last_update_path() {
let _ = std::fs::write(p, now_secs().to_string());
}
Ok(count)
}
/// `~/.chrome-use/sites/.last_update` — unix-seconds marker of the last sync.
fn last_update_path() -> Option<PathBuf> {
sites_dir().map(|d| d.join(".last_update"))
}
/// `~/.chrome-use/sites/.index.json` — `{ "github.com": ["github/issues", …], … }`,
/// built on `update` so navigation can look up adapters by domain without parsing
/// all ~145 adapter files on every command.
fn index_path() -> Option<PathBuf> {
sites_dir().map(|d| d.join(".index.json"))
}
fn now_secs() -> u64 {
std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.map(|d| d.as_secs())
.unwrap_or(0)
}
/// Parse every installed adapter and write the domain→adapters index. Within a
/// domain, read-only adapters are listed first (then alphabetical) so the
/// auto-suggested example leads with a safe read, not a write action.
fn write_domain_index(dir: &std::path::Path) {
let mut by_domain: std::collections::BTreeMap<String, Vec<(bool, String)>> = Default::default();
for spec in list_adapters().unwrap_or_default() {
if let Ok(a) = load_adapter(&spec) {
if let Some(d) = a.domain() {
let read_only = a
.meta
.get("readOnly")
.and_then(|v| v.as_bool())
.unwrap_or(false);
by_domain
.entry(d.to_string())
.or_default()
.push((read_only, spec));
}
}
}
let ordered: std::collections::BTreeMap<String, Vec<String>> = by_domain
.into_iter()
.map(|(domain, mut v)| {
// read-only (true) first, then by spec name
v.sort_by(|a, b| b.0.cmp(&a.0).then_with(|| a.1.cmp(&b.1)));
(domain, v.into_iter().map(|(_, s)| s).collect())
})
.collect();
if let Ok(json) = serde_json::to_string(&ordered) {
let _ = std::fs::write(dir.join(".index.json"), json);
}
}
const DEFAULT_TTL_DAYS: u64 = 7;
/// Whether the adapter pack should be (re)synced: true on first use (nothing
/// installed) or when the last sync is older than the TTL. Disabled by
/// `AGENT_BROWSER_SITES_NO_AUTO_UPDATE=1`; TTL overridable via
/// `AGENT_BROWSER_SITES_TTL_DAYS` (0 = always).
pub fn needs_refresh() -> bool {
if std::env::var_os("AGENT_BROWSER_SITES_NO_AUTO_UPDATE").is_some() {
return false;
}
let Some(dir) = sites_dir() else {
return false;
};
// First use: no adapters installed yet.
if list_adapters().map(|l| l.is_empty()).unwrap_or(true) {
let _ = &dir;
return true;
}
let ttl_days = std::env::var("AGENT_BROWSER_SITES_TTL_DAYS")
.ok()
.and_then(|s| s.parse::<u64>().ok())
.unwrap_or(DEFAULT_TTL_DAYS);
let ttl = ttl_days.saturating_mul(86_400);
match last_update_path().and_then(|p| std::fs::read_to_string(p).ok()) {
Some(s) => match s.trim().parse::<u64>() {
Ok(ts) => now_secs().saturating_sub(ts) >= ttl,
Err(_) => true,
},
None => true, // no marker → treat as stale
}
}
/// Adapters whose `@meta.domain` matches `host` (exact, or `host` is a subdomain
/// of it) — for auto-suggesting `site` commands when you land on a known site.
/// Reads the prebuilt `.index.json`; empty if the pack isn't synced yet.
pub fn adapters_for_domain(host: &str) -> Vec<String> {
let host = host.trim_start_matches("www.");
let Some(raw) = index_path().and_then(|p| std::fs::read_to_string(p).ok()) else {
return Vec::new();
};
let Ok(idx) = serde_json::from_str::<std::collections::BTreeMap<String, Vec<String>>>(&raw)
else {
return Vec::new();
};
// Preserve the index's per-domain ordering (read-only adapters first); just
// dedup if a host somehow matches multiple domain keys.
let mut out: Vec<String> = Vec::new();
for (domain, specs) in idx {
let d = domain.trim_start_matches("www.");
if host == d || host.ends_with(&format!(".{d}")) {
for s in specs {
if !out.contains(&s) {
out.push(s);
}
}
}
}
out
}
/// Map CLI args to the adapter's `args` object. Positional args fill the adapter's
/// declared `args` keys in order; `--key value` overrides by name. The adapter
/// validates required args itself.
pub fn map_args(adapter: &Adapter, positional: &[String], named: &[(String, String)]) -> Value {
let mut obj = serde_json::Map::new();
let keys: Vec<String> = adapter
.meta
.get("args")
.and_then(|a| a.as_object())
.map(|m| m.keys().cloned().collect())
.unwrap_or_default();
// Positional args fill the adapter's declared args in DECLARATION order
// (`arg_order`), not serde's alphabetized key order — otherwise a 2-arg
// adapter like `{projectId, path}` would map positionals to `{path, projectId}`.
for (i, val) in positional.iter().enumerate() {
if let Some(k) = keys.get(i) {
if let Some(k) = adapter.arg_order.get(i) {
obj.insert(k.clone(), Value::String(val.clone()));
}
}
@@ -264,4 +442,24 @@ async function(args) { return { repo: args.repo }; }"#;
assert!(load_adapter("noslash").is_err());
assert!(load_adapter("../etc/passwd").is_err());
}
// Regression: positional args must follow DECLARATION order, not serde's
// alphabetical key order. With `{projectId, path}` (not alphabetical),
// `<uuid> <file>` must map projectId←uuid, path←file — not swapped.
#[test]
fn positional_args_follow_declaration_order_not_alphabetical() {
let raw = r#"/* @meta
{
"name": "claude-design/get-file",
"domain": "claude.ai",
"args": { "projectId": {"required": true}, "path": {"required": true} }
}
*/
async function(args){ return args; }"#;
let a = parse_adapter(raw, "claude-design/get-file").unwrap();
assert_eq!(a.arg_order, vec!["projectId", "path"]);
let args = map_args(&a, &["the-uuid".into(), "misonote.dc.html".into()], &[]);
assert_eq!(args["projectId"], "the-uuid");
assert_eq!(args["path"], "misonote.dc.html");
}
}
Binary file not shown.
Binary file not shown.
+70 -6
View File
@@ -81,6 +81,35 @@ async function groupTabInto(tabId, name) {
groupIdByName.set(name, gid)
}
// Group-scoped relay isolation hints (issue #40). The relay scopes
// Target.getTargets per agent by tab group; report two things in the synthesized
// targetInfo so it can attribute each tab:
// - abGroup: the tab's Chrome tab-group TITLE (= the owning session name), so
// the relay can re-attribute existing tabs after a restart (createTarget
// tagging won't re-run for already-open tabs).
// - openerTargetId: the targetId of the tab that opened this one, so a pop-up
// (window.open / target=_blank / OAuth result) inherits its opener's group
// and the agent that opened it can follow it — without foreign tabs leaking.
// Best-effort: any failure yields empty strings, which the relay ignores.
async function tabScopeHints(tabId) {
let openerTargetId = ''
let abGroup = ''
try {
const t = await chrome.tabs.get(tabId)
if (t) {
if (typeof t.openerTabId === 'number') {
const op = tabs.get(t.openerTabId)
if (op) openerTargetId = op.targetId
}
if (t.groupId != null && t.groupId >= 0 && chrome.tabGroups) {
const g = await chrome.tabGroups.get(t.groupId).catch(() => null)
if (g && g.title) abGroup = g.title
}
}
} catch {}
return { openerTargetId, abGroup }
}
function postToHost(msg) {
try {
if (port) port.postMessage(msg)
@@ -126,7 +155,7 @@ function connectHost() {
} catch {}
// Tell the daemon about everything we already have attached, then attach
// anything new.
reannounceAttachedTabs()
void reannounceAttachedTabs()
void attachAllTabs()
}
@@ -140,7 +169,7 @@ async function onHostMessage(msg) {
// Daemon (re)connected — (re)attach and announce every tab so it discovers
// the user's existing tabs rather than racing an empty target list.
if (msg.method === 'attachAll') {
reannounceAttachedTabs()
void reannounceAttachedTabs()
await attachAllTabs()
return
}
@@ -259,6 +288,19 @@ async function handleForwardCdpCommand(msg) {
const params = msg?.params?.params || undefined
const sessionId = typeof msg?.params?.sessionId === 'string' ? msg.params.sessionId : undefined
// Non-CDP extension commands (ABExt.*) the daemon sends. `ungroupTab` removes a
// tab from its per-session tab group so a `keep`-marked tab is left for the user
// as a normal, ungrouped tab (the group can then be cleaned up). Best-effort.
if (method === 'ABExt.ungroupTab') {
const tabId = tabIdFromSession(sessionId) ?? tabForSession(sessionId)
if (tabId != null && chrome.tabs.ungroup) {
try {
await chrome.tabs.ungroup(tabId)
} catch {}
}
return { ungrouped: tabId ?? null }
}
// Browser-level Target methods that map onto chrome.tabs.
if (method === 'Target.createTarget') {
const url = typeof params?.url === 'string' && params.url ? params.url : 'about:blank'
@@ -387,12 +429,16 @@ async function attachTab(tabId) {
sessionToTab.set(sessionId, tabId)
rememberSessionTarget(sessionId, targetId)
setBadge(tabId, port ? 'on' : 'connecting')
const { openerTargetId, abGroup } = await tabScopeHints(tabId)
postToHost({
method: 'forwardCDPEvent',
params: {
sessionId,
method: 'Target.attachedToTarget',
params: { sessionId, targetInfo: { ...targetInfo, attached: true } },
params: {
sessionId,
targetInfo: { ...targetInfo, attached: true, openerTargetId, abGroup },
},
},
})
return entry
@@ -434,14 +480,32 @@ async function attachAllTabs() {
}
}
function reannounceAttachedTabs() {
for (const [, entry] of tabs.entries()) {
async function reannounceAttachedTabs() {
for (const [tabId, entry] of tabs.entries()) {
// Re-send the group hint too (issue #40) so the relay can rebuild its
// targetId→group map after its own restart (createTarget tagging won't
// re-run for tabs that are already open). Include the live url/title so the
// relay's target list stays matchable by URL after a reconnect (otherwise a
// reannounced tab shows a blank url and `adopt <url>` can't find it).
const { openerTargetId, abGroup } = await tabScopeHints(tabId)
let url = ''
let title = ''
try {
const t = await chrome.tabs.get(tabId)
if (t) {
url = t.url || t.pendingUrl || ''
title = t.title || ''
}
} catch {}
postToHost({
method: 'forwardCDPEvent',
params: {
sessionId: entry.sessionId,
method: 'Target.attachedToTarget',
params: { sessionId: entry.sessionId, targetInfo: { targetId: entry.targetId, type: 'page', attached: true } },
params: {
sessionId: entry.sessionId,
targetInfo: { targetId: entry.targetId, type: 'page', url, title, attached: true, openerTargetId, abGroup },
},
},
})
}
+2 -2
View File
@@ -1,8 +1,8 @@
{
"manifest_version": 3,
"name": "chrome-use",
"version": "0.4.9",
"description": "Let chrome-use drive your logged-in Chrome \u2014 install once, no token, no per-use confirmation.",
"version": "0.4.12",
"description": "Let chrome-use drive your logged-in Chrome install once, no token, no per-use confirmation.",
"key": "MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA6vQIyscGIPYPZdSpPwPL0+0gxUROyRgCpmvCSDoc8XUm4qm97VbKnD9Ijc1lV22lNWZtE78gaRjt6BeSfuMgnBymnhLKjN1gU6AI5QUU0mrJyeHdWKvrKQR5FmsM2A7Xr1ykE2SiiS8zNUS3Y/6O5l+Nva7wrVy6E4a2dkBVQkOsu+DV+nEZvhIyuDY5D5SPXqNwUTWTaglwj5mjvHz36xSwCWlPmrtJ+ED0AUyrb2z4GIOmvk4kqtBVrh/UD058klLo4CkYOnIybB5aV6WYuwarfPY4bF/dLggPem+ewLNTUNBuwrxj/A4nUv0LJTuRO8rR7f8WR9qnRCY0Ic5saQIDAQAB",
"icons": {
"16": "icons/icon16.png",
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "chrome-use",
"version": "1.5.19",
"version": "1.5.27",
"description": "chrome-use — drive your real, logged-in Chrome from any AI agent, stealth by default",
"type": "module",
"packageManager": "pnpm@11.1.3",
+66 -1
View File
@@ -59,6 +59,18 @@ next ref interaction.
> anyway.) Driving off pixels on the relay also risks a coordinate event drifting
> onto the user's foreground tab — refs never do. See issue #37.
> **Two different intents — only one is discouraged.** The rule above is about
> *screenshot-to-locate* (using a picture to find/hit an element) — that's the bug.
> *screenshot-to-capture* — saving a region or element to a file as a **reusable
> image asset** (maps, charts, og-images, visual-diff baselines, report figures) —
> is fully supported and encouraged: `screenshot [selector] [--clip x,y,w,h] <file>`.
> Capturing a rendered map region to a PNG for a blog post is the right tool, not a
> smell. Screenshots are auto-downscaled to ≤2000px (longest edge) so they fit an
> image reader and their pixels line up with `click x y`; override with
> `--max-width`/`--max-height`/`--scale`. To click something you couldn't hit by
> ref, `box @ref` gives the element's CSS-px box + `centerX/centerY` to feed
> straight into `click <centerX> <centerY>` — no screenshot needed.
## Before you automate: pick the cheapest tool
Driving a browser is the heavy option. chrome-use earns its keep when you
@@ -142,7 +154,17 @@ real Chrome concurrently without ever dropping or stealing each other's tabs —
another agent's tab churn can't make your bound tab vanish or drift your commands
onto the wrong page. Consequence: `tab list` shows only *your* session's tabs; to
drive a specific page, navigate to it in your own tab instead of expecting a
pre-existing or popped-up tab to appear in the list. **Anti-detection ranking: this real logged-in Chrome (extension
pre-existing or popped-up tab to appear in the list.
> **Need to read a tab the user already has open?** Use `chrome-use adopt
> <url-substring|targetId>` — it finds that pre-existing tab (the user's own, or
> another session's) across groups and drives it **without opening a new tab**.
> e.g. `adopt "claude.ai/design"` then `snapshot`/`eval`/`get text` on it. On no
> match it errors and lists the tabs it can see. This is the explicit, opt-in way
> through the isolation above (it tags the adopted tab into your group). Great for
> "read/extract from the page I'm looking at" without disturbing it.
**Anti-detection ranking: this real logged-in Chrome (extension
connect) > a headed launched browser > headless (forbidden).** A genuine human
browser has no headless/automation tells at all, so prefer it for anything
anti-bot-sensitive.
@@ -218,6 +240,13 @@ chrome-use site github/issues owner/repo --json # run it → JSON (navigates t
- If no adapter fits, fall back to the normal `snapshot`/`eval` loop. Adapters come from the
[bb-sites](https://github.com/epiral/bb-sites) community pack; chrome-use fetches & runs them.
> **Auto-trigger — act on it.** chrome-use keeps the pack synced automatically (first use +
> weekly), and when you `open`/`navigate`/`snapshot` a page whose domain has adapters it tells
> you: a `💡 site adapters for <domain>` line on stderr, and a `siteAdapters: {domain, commands}`
> field in `--json`. **When you see that, prefer the listed `site <name>/<cmd>` over snapshot+click
> for reading data** — it's the cheaper, more reliable path and it's already installed. You don't
> need to run `site update` yourself; just use the command it names.
## Quickstart
```bash
@@ -366,6 +395,12 @@ foreground, so prefer refs. For below-the-fold content in such a frame, scroll i
with `scroll down N --at x,y` (a pixel over the frame) or `--frame n`. For a
postal/autocomplete box inside the frame, `type @e "…" --key-events`.
> **Caveat: `find text "…"` can't reach into a cross-origin iframe** — it errors
> "Element not found" even though `snapshot -i` lists those nodes and
> `get text` reads them. Inside cross-origin iframes, target elements by their
> **snapshot `@ref`**, not by `find`. (`box @ref` also works on iframe refs when
> you need a coordinate fallback.)
### When refs don't work or you don't want to snapshot
Use semantic locators:
@@ -419,6 +454,13 @@ tree**, so `snapshot` comes back near-empty and refs are a dead end. `snapshot`
detects this and prints a one-line hint. Drive them the screenshot way:
```bash
chrome-use canvas list # enumerate <canvas> elements (size, type)
chrome-use canvas capture out.png # save the canvas's RENDERED pixels to PNG —
# toDataURL (full backing-store res, e.g.
# Figma 2522x1904), screenshot fallback for
# WebGL w/o preserveDrawingBuffer / tainted.
# Gets the RENDER, not hidden source data
# (those live in the app's binary store/API).
chrome-use screenshot /tmp/s.png # SEE the state (your only read path —
# eval/get text return nothing useful)
chrome-use click 640 360 # interact by viewport coordinate
@@ -749,6 +791,29 @@ chrome-use snapshot -i
chrome-use frame main # back to main frame
```
### Viewport / window size (responsive & overflow debugging)
To reproduce width-dependent bugs (responsive breakpoints, horizontal-overflow
hunts, mobile layouts) set the viewport. This is a **CDP virtual viewport**
(`Emulation.setDeviceMetricsOverride`) — it changes the layout viewport *for the
tab* without physically resizing the OS window, so it works headless **and** over
the extension relay without yanking the user's real Chrome window around.
```bash
chrome-use viewport 1280 800 # set width x height (alias: resize)
chrome-use viewport 375x812 # WxH shorthand
chrome-use viewport 375 812 --dpr 3 --mobile # retina + mobile emulation
chrome-use viewport reset # clear the override, restore real size
```
```bash
# Find what's overflowing at a narrow width:
chrome-use viewport 375 812
chrome-use eval 'document.documentElement.scrollWidth + " vs " + innerWidth'
```
`set viewport <w> <h> [scale]` is an equivalent alias.
### Dialogs
`alert` and `beforeunload` are auto-accepted so agents never block. For