Compare commits

...
7 Commits
Author SHA1 Message Date
leeguooooo 02e23ebe11 fix(build): include browser.rs clear_viewport/via_relay (#47) + cargo fmt
Release binaries / Build macOS ARM64 (push) Has been cancelled
Release binaries / Build macOS x64 (push) Has been cancelled
Release binaries / Build Linux ARM64 (push) Has been cancelled
Release binaries / Build Linux musl ARM64 (push) Has been cancelled
Release binaries / Build Linux musl x64 (push) Has been cancelled
Release binaries / Build Linux x64 (push) Has been cancelled
Release binaries / Build Windows x64 (push) Has been cancelled
Release binaries / Attach binaries to GitHub Release (push) Has been cancelled
v1.5.24 (d5cd9cd) shipped a commands.rs caller of `clear_viewport` but not the
browser.rs method it lives in (a concurrent in-progress #47 viewport/resize edit
was only partly staged), so main didn't compile and the format check failed.
Commit the matching browser.rs method + via_relay() helper and run cargo fmt.
Full tree builds; 863 tests pass.
2026-06-18 11:39:11 +09:00
leeguooooo d5cd9cd621 feat(canvas): extract WebGL/canvas-app content + fix site arg-order + adopt skill doc
canvas — chrome-use can now read canvas/WebGL apps (Figma, games, maps, charts,
drawing tools) that expose no DOM/refs:
  - canvas list: enumerate <canvas> (backing+CSS size, visibility, toDataUrl/tainted)
  - canvas capture [selector] [path]: save rendered pixels to PNG — toDataURL
    (full backing-store resolution), with a CDP screenshot fallback for WebGL
    without preserveDrawingBuffer or cross-origin-tainted canvases. --screenshot
    forces the screenshot path. Gets the RENDER, not hidden source data.
  Verified live: captured Figma's canvas at full 2522x1904 via toDataURL.

site — fix map_args losing the adapter's declared arg order: serde sorts @meta
keys alphabetically, so a 2-arg adapter like {projectId, path} mapped positionals
to {path, projectId} (swapped). Now parses declaration order from the raw @meta
text (Adapter.arg_order) + regression test. Affects any multi-arg adapter.

skill — core skill now documents `adopt <url|targetId>` (read a pre-existing tab,
the explicit way through strict isolation) and `canvas list`/`canvas capture` in
the canvas/WebGL section.

863 tests pass.
2026-06-18 11:31:24 +09:00
leeguooooo 284a60a54c feat(adopt): read a pre-existing tab without opening a new one
Release binaries / Build macOS ARM64 (push) Has been cancelled
Release binaries / Build macOS x64 (push) Has been cancelled
Release binaries / Build Linux ARM64 (push) Has been cancelled
Release binaries / Build Linux musl ARM64 (push) Has been cancelled
Release binaries / Build Linux musl x64 (push) Has been cancelled
Release binaries / Build Linux x64 (push) Has been cancelled
Release binaries / Build Windows x64 (push) Has been cancelled
Release binaries / Attach binaries to GitHub Release (push) Has been cancelled
New `chrome-use adopt <url-substring|targetId>`: drive a tab the user (or
another session) already has open, with ZERO new tabs. After group-scoped
isolation (#40) a session can't see foreign tabs, so adopt adds an explicit,
opt-in path:

- Relay (relay.rs): `ABRelay.getAllTargets` returns every attached target
  UNSCOPED (ignores group scoping), so the agent can find a specific tab by URL
  or targetId. +1 unit test.
- Daemon (browser.rs): `collect_all_targets` (unscoped, falls back to scoped on
  older relays) + `adopt_existing_target` — matches by exact targetId or
  case-insensitive URL substring, attaches it (the relay re-tags it into the
  adopter's group, so isolation holds), pins it; never creates a tab. On no
  match it errors AND lists the open tabs it can see, rather than launching.
  discover_and_attach_targets honors AGENT_BROWSER_ADOPT at first connect, so no
  about:blank is ever created.
- CLI (main.rs): `adopt` sets the env, forces a fresh daemon, and rewrites into
  `connect <relay-url>` (like `extension connect`) so the daemon attaches to the
  user's real Chrome before parse_command.

Extension (ab-connect 0.4.11): `reannounceAttachedTabs` now re-sends each tab's
url/title (it previously sent neither) so the relay's target list stays matchable
by URL after the MV3 service worker reconnects — otherwise reannounced tabs show
a blank url and `adopt <url>` can't find them. Repacked upload zip + crx.

Mechanism verified live (enumerated all 11 of the user's open tabs incl. the
target). 862 tests pass.
2026-06-17 21:18:01 +09:00
leeguooooo 10d196b6eb chore(ext): pack ab-connect 0.4.10 upload zip + crx (#40 group-scoped relay)
Rebuilt extensions/ab-connect.zip (key stripped for the Web Store) and the
reference .crx from the 0.4.10 source (openerTargetId + abGroup in the
synthesized Target.attachedToTarget).
2026-06-17 18:17:20 +09:00
leeguooooo 5be01e292d feat(relay): group-scoped Target.getTargets — restore follow-popup + cross-session adopt under isolation (#40)
Release binaries / Build macOS ARM64 (push) Has been cancelled
Release binaries / Build macOS x64 (push) Has been cancelled
Release binaries / Build Linux ARM64 (push) Has been cancelled
Release binaries / Build Linux musl ARM64 (push) Has been cancelled
Release binaries / Build Linux musl x64 (push) Has been cancelled
Release binaries / Build Linux x64 (push) Has been cancelled
Release binaries / Build Windows x64 (push) Has been cancelled
Release binaries / Attach binaries to GitHub Release (push) Has been cancelled
Move multi-agent isolation from blunt daemon-side filtering to relay-side
group scoping, so a session can adopt new tabs again (follow-popup, OAuth
results, cross-session adopt-by-targetId) without ever seeing the user's or
another agent's tabs.

Relay (relay.rs): track client->group (announced via new local ABRelay.setGroup,
or the first createTarget's agentGroup) and target->group (created tabs tagged
from the createTarget reply; an explicit attachToTarget tags the target into the
adopter's group = #21; a pop-up inherits its opener's group via openerTargetId).
Target.getTargets returns ONLY the requesting client's group; a client that never
announced a group (older daemon) gets the full list — fully backward-compatible.
+5 unit tests.

Daemon (browser.rs): announce_group() on connect sets relay_scoped. Adoption in
discover/resync/adopt_newly_opened is re-enabled ONLY when relay_scoped; without
it (launch / real CDP / older relay that didn't answer the announce) the daemon
keeps strict daemon-side isolation. So this can't regress the 125/125 isolation.

Extension (ab-connect 0.4.10): synthesized Target.attachedToTarget targetInfo now
carries openerTargetId (pop-ups inherit opener's group) and abGroup (the tab-group
title, so the relay re-attributes existing tabs after ITS own restart, since
createTarget tagging won't re-run). tabScopeHints().

Back-compat verified live: new daemon + OLD relay -> announce fails ->
relay_scoped=false -> strict fallback, open/eval/url all work. The new extension
(publish to CWS, strip manifest key) activates follow-popup; relay+daemon ship now.
861 tests pass.
2026-06-17 18:14:13 +09:00
leeguooooo a83d1b1df9 feat: rich-editor fill, box centers, screenshot downscale, disabled+docs (#41-#45)
Release binaries / Build macOS ARM64 (push) Has been cancelled
Release binaries / Build macOS x64 (push) Has been cancelled
Release binaries / Build Linux ARM64 (push) Has been cancelled
Release binaries / Build Linux musl ARM64 (push) Has been cancelled
Release binaries / Build Linux musl x64 (push) Has been cancelled
Release binaries / Build Linux x64 (push) Has been cancelled
Release binaries / Build Windows x64 (push) Has been cancelled
Release binaries / Attach binaries to GitHub Release (push) Has been cancelled
Dogfooding backlog from this session's embedded-form/editor work.

#41 fill on rich editors: detect CodeMirror 5 / Monaco / ProseMirror /
contenteditable and set via their own API or execCommand('insertText') so
beforeinput/input fire (a raw .value/textContent write no-op'd juejin's
CodeMirror and skipped React composers). Response echoes the `engine` used.
`fill <sel> --file <path>` / `--stdin` set large multiline text without
shell-escaping. `get value` now reads CodeMirror/Monaco/contenteditable too.

#42 screenshot --max-width/--max-height/--scale, plus a default 2000px
longest-edge cap (AGENT_BROWSER_SCREENSHOT_MAX_EDGE; 0 disables) so retina
full-page shots fit an agent's image reader and --scale 0.5 makes screenshot
px line up with click px. Annotated shots are never downscaled.

#43 `box @ref` (already a top-level alias of `get box`) now also returns
centerX/centerY/inViewport in CSS px — feed straight into `click x y` when a
ref-click no-ops (e.g. a button in a cross-origin iframe).

#44 no code change needed — disabled elements already list as
`button "Save" [disabled, ref=eN]`; the reporter's missing button was
DOM-gated on validity. Added a skill note: `find text` can't reach into a
cross-origin iframe — target those by snapshot @ref.

#45 core skill now distinguishes screenshot-to-locate (discouraged) from
screenshot-to-capture a reusable image asset via `screenshot [--clip] <file>`
(encouraged), so agents stop over-reading the prohibition.

#40 (group-scoped relay) stays deferred — needs an ab-connect extension change.

Verified live: fill --file round-trips multiline+CJK+backticks; contenteditable
engine=contenteditable + get value reads it back; box gives centerX/centerY/
inViewport; screenshot of retina example.com → 2000px; disabled button shows
[disabled]. 856 tests pass.
2026-06-17 17:49:38 +09:00
leeguooooo 50b27ac0e0 feat(site): auto-sync + auto-suggest adapters (auto-trigger)
Release binaries / Build macOS ARM64 (push) Has been cancelled
Release binaries / Build macOS x64 (push) Has been cancelled
Release binaries / Build Linux ARM64 (push) Has been cancelled
Release binaries / Build Linux musl ARM64 (push) Has been cancelled
Release binaries / Build Linux musl x64 (push) Has been cancelled
Release binaries / Build Linux x64 (push) Has been cancelled
Release binaries / Build Windows x64 (push) Has been cancelled
Release binaries / Attach binaries to GitHub Release (push) Has been cancelled
Make `site` trigger itself so an agent doesn't have to know adapters exist.

Auto-sync: the pack refreshes on first use and on a TTL (default 7d), both in
the `site` command path (blocking, fast) and as a non-blocking background task
on daemon startup — so ~/.chrome-use/sites/.index.json is always populated with
zero added latency. Tune via AGENT_BROWSER_SITES_TTL_DAYS; disable with
AGENT_BROWSER_SITES_NO_AUTO_UPDATE=1. `update` now writes .last_update + a
domain→adapters .index.json (read-only adapters ordered first).

Auto-suggest: `open`/`navigate`/`snapshot` onto a domain with adapters attaches
`siteAdapters: {domain, commands}` to the response; the CLI prints a
`💡 site adapters for <domain>` hint (stderr) and the field rides along in --json.
SKILL.md tells the agent to prefer the listed `site <name>/<cmd>` over scraping.
This keeps the 'never auto-disrupt user tabs' guarantee — it suggests, the agent
decides; nothing auto-runs on navigation.

site.rs: needs_refresh/adapters_for_domain/write_domain_index + timestamp/index
in update(). daemon.rs: background bootstrap. actions.rs: with_site_hint on
navigate + snapshot. output.rs: hint render. Verified live: open github.com →
hint leads with read-only github/issues; --json carries siteAdapters.
2026-06-17 17:15:34 +09:00
21 changed files with 1646 additions and 135 deletions
+16
View File
@@ -233,6 +233,22 @@ Positional args fill the adapter's declared args in order; `--key value` overrid
by name. Adapters are authored by the bb-sites community and remain their authors'
property — chrome-use just runs them.
**Auto-sync + auto-suggest.** You rarely type `site update` yourself: chrome-use
syncs the pack on first use and refreshes it weekly in the background (tune with
`AGENT_BROWSER_SITES_TTL_DAYS`, disable with `AGENT_BROWSER_SITES_NO_AUTO_UPDATE=1`).
And when you `open`/`snapshot` a page whose domain has adapters, chrome-use surfaces
them right in the output — a `💡 site adapters for <domain>` line, plus a
`siteAdapters` field under `--json` — so an agent reaches for the structured-data
adapter instead of scraping the DOM:
```text
$ chrome-use open https://github.com
💡 site adapters for github.com — prefer these for structured data:
github/issues, github/me, github/repo, …
e.g. chrome-use site github/issues --json
✓ GitHub
```
## Automated testing (`chrome-use test`)
Turn the repetitive "open it, click around, check it's right" work into a
+14
View File
@@ -188,6 +188,20 @@ chrome-use site bilibili/feed --json # 能用,因为走的是你的
位置参数按适配器声明的参数顺序填入;`--key value` 按名覆盖。适配器由 bb-sites 社区编写、
版权归各自作者所有 —— chrome-use 只负责运行它们。
**自动同步 + 自动提示。** 你基本不用手动 `site update`:chrome-use 首次使用时自动拉取,
之后每周后台刷新一次(`AGENT_BROWSER_SITES_TTL_DAYS` 调周期,`AGENT_BROWSER_SITES_NO_AUTO_UPDATE=1`
关闭)。而当你 `open`/`snapshot` 一个有适配器的域名时,chrome-use 会在输出里直接把可用命令
亮出来 —— 一行 `💡 site adapters for <域名>`,`--json` 下则是 `siteAdapters` 字段 —— 这样
agent 会直接改用结构化适配器,而不是去扒 DOM:
```text
$ chrome-use open https://github.com
💡 site adapters for github.com — prefer these for structured data:
github/issues, github/me, github/repo, …
e.g. chrome-use site github/issues --json
✓ GitHub
```
## 自动化测试(`chrome-use test`
把反复的「打开它、点一圈、看对不对」变成**可重跑的测试套件** —— 前端的单元测试。用 YAML 写用例;步骤复用 chrome-use 自己的命令,断言编译成一次检查:
+1 -1
View File
@@ -290,7 +290,7 @@ checksum = "613afe47fcd5fac7ccf1db93babcb082c5994d996f20b8b159f2ad1658eb5724"
[[package]]
name = "chrome-use"
version = "1.5.19"
version = "1.5.24"
dependencies = [
"aes",
"aes-gcm",
+1 -1
View File
@@ -1,6 +1,6 @@
[package]
name = "chrome-use"
version = "1.5.19"
version = "1.5.24"
edition = "2021"
description = "Fast browser automation CLI for AI agents"
license = "Apache-2.0"
+298 -35
View File
@@ -79,6 +79,11 @@ const KNOWN_COMMANDS: &[&str] = &[
"dialog",
"upload",
"site",
"box",
"adopt",
"canvas",
"viewport",
"resize",
];
/// Levenshtein distance, capped — small inputs only (command names).
@@ -560,9 +565,37 @@ fn parse_command_inner(args: &[String], flags: &Flags) -> Result<Value, ParseErr
"fill" => {
let sel = rest.first().ok_or_else(|| ParseError::MissingArguments {
context: "fill".to_string(),
usage: "fill <selector> <text>",
usage: "fill <selector> <text> | fill <selector> --file <path> | fill <selector> --stdin",
})?;
Ok(json!({ "id": id, "action": "fill", "selector": sel, "value": rest[1..].join(" ") }))
// Large/multiline content without shell-escaping hell (issue #41):
// `fill <sel> --file <path>` reads the value from a UTF-8 file, and
// `fill <sel> --stdin` reads it from stdin — sent verbatim, so backticks,
// quotes, newlines and non-ASCII pass through untouched.
let value = match rest.get(1).copied() {
Some("--file") => {
let path = rest.get(2).ok_or(ParseError::InvalidValue {
message: "fill --file requires a path".to_string(),
usage: "fill <selector> --file <path>",
})?;
std::fs::read_to_string(path).map_err(|e| ParseError::InvalidValue {
message: format!("fill --file: cannot read {path}: {e}"),
usage: "fill <selector> --file <path>",
})?
}
Some("--stdin") => {
use std::io::Read;
let mut buf = String::new();
io::stdin()
.read_to_string(&mut buf)
.map_err(|e| ParseError::InvalidValue {
message: format!("fill --stdin: {e}"),
usage: "fill <selector> --stdin",
})?;
buf
}
_ => rest[1..].join(" "),
};
Ok(json!({ "id": id, "action": "fill", "selector": sel, "value": value }))
}
"type" => {
// `--key-events` (alias `--keys`): send real per-character keystrokes
@@ -1006,11 +1039,55 @@ fn parse_command_inner(args: &[String], flags: &Flags) -> Result<Value, ParseErr
// path: file path (contains / or . or ends with known extension)
let mut full_page = false;
let mut clip: Option<Value> = None;
let mut max_width: Option<u32> = None;
let mut max_height: Option<u32> = None;
let mut scale: Option<f64> = None;
let mut positional: Vec<&str> = Vec::new();
let mut i = 0;
// Parse a numeric value for a downscale flag (issue #42).
let parse_num = |i: &mut usize, flag: &str| -> Result<String, ParseError> {
let v = rest
.get(*i + 1)
.ok_or_else(|| ParseError::MissingArguments {
context: format!("screenshot {flag}"),
usage: "screenshot [--max-width <px>] [--max-height <px>] [--scale <0..1>]",
})?;
*i += 1;
Ok(v.to_string())
};
while i < rest.len() {
match rest[i] {
"--full" | "-f" => full_page = true,
// Downscale the saved image so retina/full-page shots fit an
// agent's image reader and screenshot px line up with click px (#42).
"--max-width" => {
let v = parse_num(&mut i, "--max-width")?;
max_width = Some(v.parse().map_err(|_| ParseError::InvalidValue {
message: format!("--max-width expects a number, got '{v}'"),
usage: "screenshot --max-width <px>",
})?);
}
"--max-height" => {
let v = parse_num(&mut i, "--max-height")?;
max_height = Some(v.parse().map_err(|_| ParseError::InvalidValue {
message: format!("--max-height expects a number, got '{v}'"),
usage: "screenshot --max-height <px>",
})?);
}
"--scale" => {
let v = parse_num(&mut i, "--scale")?;
let s: f64 = v.parse().map_err(|_| ParseError::InvalidValue {
message: format!("--scale expects a number like 0.5, got '{v}'"),
usage: "screenshot --scale <0..1>",
})?;
if s <= 0.0 || s > 1.0 {
return Err(ParseError::InvalidValue {
message: format!("--scale must be in (0, 1], got '{v}'"),
usage: "screenshot --scale <0..1>",
});
}
scale = Some(s);
}
// `--clip x,y,w,h` captures a pixel region (issue #34).
"--clip" => {
let raw = rest
@@ -1073,6 +1150,15 @@ fn parse_command_inner(args: &[String], flags: &Flags) -> Result<Value, ParseErr
if let Some(c) = clip {
cmd["clip"] = c;
}
if let Some(w) = max_width {
cmd["maxWidth"] = json!(w);
}
if let Some(h) = max_height {
cmd["maxHeight"] = json!(h);
}
if let Some(s) = scale {
cmd["scale"] = json!(s);
}
if let Some(ref fmt) = flags.screenshot_format {
cmd["format"] = json!(fmt);
}
@@ -1090,6 +1176,54 @@ fn parse_command_inner(args: &[String], flags: &Flags) -> Result<Value, ParseErr
}
Ok(cmd)
}
// Canvas/WebGL apps (Figma, games, maps, charts, drawing tools) render to a
// <canvas> with no DOM/refs to read. `canvas` extracts what's actually
// rendered: `list` enumerates canvases; `capture` saves a canvas to PNG via
// toDataURL (full backing-store resolution) with a CDP-screenshot fallback
// for WebGL contexts (no preserveDrawingBuffer) or cross-origin-tainted ones.
"canvas" => {
match rest.first().copied() {
Some("list") => Ok(json!({ "id": id, "action": "canvas_list" })),
Some("capture") => {
let force_screenshot = rest.contains(&"--screenshot");
let pos: Vec<&str> = rest[1..]
.iter()
.copied()
.filter(|a| !a.starts_with("--"))
.collect();
// `capture [selector] [path]`: a selector starts with . # @ or is
// a tag; a path contains / or ends in an image extension.
let is_path = |s: &str| {
s.contains('/')
|| s.ends_with(".png")
|| s.ends_with(".jpg")
|| s.ends_with(".jpeg")
|| s.ends_with(".webp")
};
let (selector, path) = match (pos.first(), pos.get(1)) {
(Some(a), Some(b)) => (Some(*a), Some(*b)),
(Some(a), None) if is_path(a) => (None, Some(*a)),
(Some(a), None) => (Some(*a), None),
_ => (None, None),
};
let mut cmd = json!({ "id": id, "action": "canvas_capture" });
if let Some(s) = selector {
cmd["selector"] = json!(s);
}
if let Some(p) = path {
cmd["path"] = json!(p);
}
if force_screenshot {
cmd["forceScreenshot"] = json!(true);
}
Ok(cmd)
}
_ => Err(ParseError::InvalidValue {
message: "canvas needs a subcommand".to_string(),
usage: "canvas list | canvas capture [selector] [path] [--screenshot]",
}),
}
}
"pdf" => {
let path = rest.first().ok_or_else(|| ParseError::MissingArguments {
context: "pdf".to_string(),
@@ -1235,6 +1369,12 @@ fn parse_command_inner(args: &[String], flags: &Flags) -> Result<Value, ParseErr
Ok(json!({ "id": id, "action": "site", "domain": domain, "script": script }))
}
// `adopt <url|targetId>`: the adoption happens at daemon connect (driven by
// the AGENT_BROWSER_ADOPT env main.rs set + a forced-fresh daemon), so by
// the time this command runs the tab is already attached. Resolve to a
// `url` read so the response confirms which tab got adopted.
"adopt" => Ok(json!({ "id": id, "action": "url" })),
// === Stealth self-check ===
"stealth" => {
// `stealth [status]` — local stealth self-check: mode, live probes
@@ -1558,6 +1698,13 @@ fn parse_command_inner(args: &[String], flags: &Flags) -> Result<Value, ParseErr
// === Mouse ===
"mouse" => parse_mouse(&rest, &id),
// === Viewport / window size ===
// Top-level shortcut for `set viewport` — agents (and the author, in
// issue #47) reach for `viewport`/`resize` first. Uses a CDP virtual
// viewport, so it also works on the extension relay without yanking the
// user's real window around.
"viewport" | "resize" => parse_viewport(&rest, &id),
// === Set (browser settings) ===
"set" => parse_set(&rest, &id),
@@ -3006,6 +3153,97 @@ fn parse_mouse(rest: &[&str], id: &str) -> Result<Value, ParseError> {
}
}
/// Parse a viewport / resize spec, shared by the top-level `viewport` / `resize`
/// commands and `set viewport`. Sets a CDP device-metrics override
/// (`Emulation.setDeviceMetricsOverride`) — a *virtual* viewport for the tab, so
/// it works headless AND on the extension relay without physically resizing the
/// user's real Chrome window (issue #47). Forms:
/// viewport <width> <height> [scale] [--dpr N] [--mobile]
/// viewport <width>x<height> (e.g. 1280x800)
/// viewport reset | clear (drop the override, restore real size)
fn parse_viewport(rest: &[&str], id: &str) -> Result<Value, ParseError> {
const USAGE: &str = "viewport <width> <height> [scale] [--dpr N] [--mobile] | viewport reset";
if matches!(
rest.first().copied(),
Some("reset") | Some("clear") | Some("off")
) {
return Ok(json!({ "id": id, "action": "viewport", "reset": true }));
}
// Positional (non-flag) tokens. A `WxH` token counts as one positional.
let positionals: Vec<&str> = rest
.iter()
.copied()
.filter(|a| !a.starts_with("--"))
.collect();
let (w, h, scale_tok): (i32, i32, Option<&str>) = match positionals.first() {
Some(first) if first.contains('x') || first.contains('X') => {
let mut parts = first.split(|c| c == 'x' || c == 'X');
let w = parts.next().and_then(|s| s.parse::<i32>().ok());
let h = parts.next().and_then(|s| s.parse::<i32>().ok());
match (w, h) {
(Some(w), Some(h)) => (w, h, positionals.get(1).copied()),
_ => {
return Err(ParseError::InvalidValue {
message: format!("Invalid viewport size: {}", first),
usage: USAGE,
})
}
}
}
Some(w_str) => {
let h_str = positionals.get(1).ok_or(ParseError::MissingArguments {
context: "viewport".to_string(),
usage: USAGE,
})?;
let w = w_str.parse::<i32>().map_err(|_| ParseError::InvalidValue {
message: format!("Invalid width: {}", w_str),
usage: USAGE,
})?;
let h = h_str.parse::<i32>().map_err(|_| ParseError::InvalidValue {
message: format!("Invalid height: {}", h_str),
usage: USAGE,
})?;
(w, h, positionals.get(2).copied())
}
None => {
return Err(ParseError::MissingArguments {
context: "viewport".to_string(),
usage: USAGE,
})
}
};
let mut cmd = json!({ "id": id, "action": "viewport", "width": w, "height": h });
// Device-scale-factor: positional, overridden by --dpr / --scale.
let mut scale: Option<f64> = match scale_tok {
Some(s) => Some(s.parse::<f64>().map_err(|_| ParseError::InvalidValue {
message: format!("Invalid scale: {}", s),
usage: USAGE,
})?),
None => None,
};
if let Some(i) = rest.iter().position(|a| *a == "--dpr" || *a == "--scale") {
let v = rest.get(i + 1).and_then(|s| s.parse::<f64>().ok()).ok_or(
ParseError::InvalidValue {
message: "--dpr/--scale needs a number".to_string(),
usage: USAGE,
},
)?;
scale = Some(v);
}
if let Some(s) = scale {
cmd["deviceScaleFactor"] = json!(s);
}
if rest.iter().any(|a| *a == "--mobile") {
cmd["mobile"] = json!(true);
}
Ok(cmd)
}
fn parse_set(rest: &[&str], id: &str) -> Result<Value, ParseError> {
const VALID: &[&str] = &[
"viewport",
@@ -3020,39 +3258,8 @@ fn parse_set(rest: &[&str], id: &str) -> Result<Value, ParseError> {
];
match rest.first().copied() {
Some("viewport") => {
let w_str = rest.get(1).ok_or_else(|| ParseError::MissingArguments {
context: "set viewport".to_string(),
usage: "set viewport <width> <height> [scale]",
})?;
let h_str = rest.get(2).ok_or_else(|| ParseError::MissingArguments {
context: "set viewport".to_string(),
usage: "set viewport <width> <height> [scale]",
})?;
let w = w_str
.parse::<i32>()
.map_err(|_| ParseError::MissingArguments {
context: "set viewport".to_string(),
usage: "set viewport <width> <height> [scale]",
})?;
let h = h_str
.parse::<i32>()
.map_err(|_| ParseError::MissingArguments {
context: "set viewport".to_string(),
usage: "set viewport <width> <height> [scale]",
})?;
let mut cmd = json!({ "id": id, "action": "viewport", "width": w, "height": h });
if let Some(scale_str) = rest.get(3) {
let scale = scale_str
.parse::<f64>()
.map_err(|_| ParseError::MissingArguments {
context: "set viewport".to_string(),
usage: "set viewport <width> <height> [scale]",
})?;
cmd["deviceScaleFactor"] = json!(scale);
}
Ok(cmd)
}
// `set viewport ...` is an alias for the top-level `viewport` command.
Some("viewport") => parse_viewport(&rest[1..], id),
Some("device") => {
let dev = rest.get(1).ok_or_else(|| ParseError::MissingArguments {
context: "set device".to_string(),
@@ -5213,6 +5420,62 @@ mod tests {
assert!(result.is_err());
}
#[test]
fn test_viewport_toplevel() {
let cmd = parse_command(&args("viewport 1280 800"), &default_flags()).unwrap();
assert_eq!(cmd["action"], "viewport");
assert_eq!(cmd["width"], 1280);
assert_eq!(cmd["height"], 800);
assert!(cmd.get("deviceScaleFactor").is_none());
assert!(cmd.get("mobile").is_none());
}
#[test]
fn test_resize_alias() {
let cmd = parse_command(&args("resize 700 800"), &default_flags()).unwrap();
assert_eq!(cmd["action"], "viewport");
assert_eq!(cmd["width"], 700);
assert_eq!(cmd["height"], 800);
}
#[test]
fn test_viewport_wxh_form() {
let cmd = parse_command(&args("viewport 375x812"), &default_flags()).unwrap();
assert_eq!(cmd["action"], "viewport");
assert_eq!(cmd["width"], 375);
assert_eq!(cmd["height"], 812);
}
#[test]
fn test_viewport_dpr_and_mobile_flags() {
let cmd =
parse_command(&args("viewport 375 812 --dpr 3 --mobile"), &default_flags()).unwrap();
assert_eq!(cmd["action"], "viewport");
assert_eq!(cmd["width"], 375);
assert_eq!(cmd["height"], 812);
assert_eq!(cmd["deviceScaleFactor"], 3.0);
assert_eq!(cmd["mobile"], true);
}
#[test]
fn test_viewport_reset() {
for spec in ["viewport reset", "viewport clear", "resize reset"] {
let cmd = parse_command(&args(spec), &default_flags()).unwrap();
assert_eq!(cmd["action"], "viewport", "{spec}");
assert_eq!(cmd["reset"], true, "{spec}");
}
}
#[test]
fn test_viewport_missing_height() {
assert!(parse_command(&args("viewport 1280"), &default_flags()).is_err());
}
#[test]
fn test_viewport_invalid_width() {
assert!(parse_command(&args("viewport abc 800"), &default_flags()).is_err());
}
#[test]
fn test_find_first_no_value() {
let cmd = parse_command(&args("find first a click"), &default_flags()).unwrap();
+1 -1
View File
@@ -595,7 +595,7 @@ fn query_current_url(session: &str) -> Option<String> {
}
/// Kill a running daemon by reading its PID file and sending a kill signal.
fn kill_stale_daemon(session: &str) {
pub fn kill_stale_daemon(session: &str) {
// Remove the socket first so no new connections reach the old daemon
#[cfg(unix)]
{
+58
View File
@@ -840,6 +840,27 @@ fn main() {
// `info` are CLI-side (download/filesystem); `site <name>/<cmd> [args]` falls
// through to the daemon dispatch below (navigate to the adapter's domain + eval).
if clean.first().map(|s| s.as_str()) == Some("site") {
// Auto-sync the adapter pack on first use and periodically (TTL, default
// 7d) so adapters stay fresh without a manual `site update`. Skipped for an
// explicit `update` (full sync below). Best-effort: offline → cached pack.
// Disable with AGENT_BROWSER_SITES_NO_AUTO_UPDATE=1.
if clean.get(1).map(|s| s.as_str()) != Some("update") && site::needs_refresh() {
let rt = tokio::runtime::Runtime::new().expect("Failed to create tokio runtime");
match rt.block_on(site::update()) {
Ok(n) => {
eprintln!(
"{}",
color::dim(&format!("site: synced {n} adapters (auto)"))
)
}
Err(e) => eprintln!(
"{}",
color::dim(&format!(
"site: auto-sync skipped ({e}); using cached adapters"
))
),
}
}
match clean.get(1).map(|s| s.as_str()) {
Some("update") => {
let rt = tokio::runtime::Runtime::new().expect("Failed to create tokio runtime");
@@ -960,6 +981,43 @@ fn main() {
}
}
// `adopt <url|targetId>`: read a PRE-EXISTING tab (the user's own, or another
// session's) WITHOUT opening a new one. Forces a fresh daemon and points it at
// the relay (like `extension connect`); the AGENT_BROWSER_ADOPT env makes the
// daemon's first connect ADOPT the matching tab instead of creating an
// about:blank. Rewrites into `connect <relay-url>` BEFORE parse_command so the
// daemon attaches to the user's real Chrome. Must run before parse_command.
if clean.first().map(|s| s.as_str()) == Some("adopt") {
match clean.get(1) {
Some(spec) if !spec.trim().is_empty() => {
std::env::set_var("AGENT_BROWSER_ADOPT", spec.trim());
connection::kill_stale_daemon(&flags.session);
match connect::relay_url() {
Some(url) => {
flags.cdp = Some(url.clone());
flags.auto_connect = false;
clean = vec!["connect".to_string(), url];
}
None => {
eprintln!(
"{} extension relay not connected — open Chrome with the ab-connect \
extension first (this command reads an EXISTING tab, it won't launch one).",
color::error_indicator()
);
exit(1);
}
}
}
_ => {
eprintln!(
"{} usage: chrome-use adopt <url-substring|targetId> (reads an existing tab, no new tab)",
color::error_indicator()
);
exit(2);
}
}
}
// Handle session separately (doesn't need daemon)
if clean.first().map(|s| s.as_str()) == Some("session") {
run_session(&clean, &flags.session, flags.json);
+290 -5
View File
@@ -1320,6 +1320,8 @@ pub async fn execute_command(cmd: &Value, state: &mut DaemonState) -> Value {
"stealth_status" => handle_stealth_status(state).await,
"snapshot" => handle_snapshot(cmd, state).await,
"screenshot" => handle_screenshot(cmd, state).await,
"canvas_list" => handle_canvas_list(state).await,
"canvas_capture" => handle_canvas_capture(cmd, state).await,
"click" => handle_click(cmd, state).await,
"dblclick" => handle_dblclick(cmd, state).await,
"fill" => handle_fill(cmd, state).await,
@@ -2478,7 +2480,10 @@ async fn handle_navigate(cmd: &Value, state: &mut DaemonState) -> Result<Value,
wb.navigate(url).await?;
let new_url = wb.get_url().await.unwrap_or_else(|_| url.to_string());
let title = wb.get_title().await.unwrap_or_default();
return Ok(json!({ "url": new_url, "title": title }));
return Ok(with_site_hint(
json!({ "url": new_url, "title": title }),
url,
));
}
}
@@ -2545,7 +2550,7 @@ async fn handle_navigate(cmd: &Value, state: &mut DaemonState) -> Result<Value,
.unwrap_or(false)
{
if let Ok(Some(switched)) = mgr.reuse_tab_for_url(url).await {
return Ok(switched);
return Ok(with_site_hint(switched, url));
}
}
@@ -2553,7 +2558,34 @@ async fn handle_navigate(cmd: &Value, state: &mut DaemonState) -> Result<Value,
// Adaptive humanize: sample the freshly loaded page for known behavioural
// anti-bot vendors and escalate this session to Human if any are present.
detect_and_set_humanize(mgr).await;
Ok(result)
Ok(with_site_hint(result, url))
}
/// Annotate a navigation/snapshot result with the `site` adapters available for
/// the page's domain (auto-trigger): when you land on e.g. github.com, the
/// response carries `siteAdapters: { domain, commands: ["github/issues", …] }` so
/// the agent reaches for a structured-data adapter instead of scraping. No-op
/// when nothing matches or the pack isn't synced yet.
fn with_site_hint(mut result: Value, fallback_url: &str) -> Value {
let url = result
.get("url")
.and_then(|v| v.as_str())
.unwrap_or(fallback_url);
let host = url::Url::parse(url)
.ok()
.and_then(|u| u.host_str().map(String::from));
if let Some(host) = host {
let adapters = crate::site::adapters_for_domain(&host);
if !adapters.is_empty() {
if let Some(obj) = result.as_object_mut() {
obj.insert(
"siteAdapters".to_string(),
json!({ "domain": host, "commands": adapters }),
);
}
}
}
result
}
/// After navigation, probe the page for known anti-bot vendor fingerprints
@@ -2936,6 +2968,13 @@ async fn handle_snapshot(cmd: &Value, state: &mut DaemonState) -> Result<Value,
let ref_count = refs.len();
let mut out = json!({ "snapshot": tree, "origin": url, "refs": refs });
// Auto-trigger: if this domain has site adapters, surface them so the agent
// pulls structured data instead of walking the tree. `with_site_hint` reads
// the `url` field, so pass it under that key.
if let Some(hint) = with_site_hint(json!({ "url": url }), &url).get("siteAdapters") {
out["siteAdapters"] = hint.clone();
}
// Canvas/WebGL apps (games, map/3D viewers, drawing tools) paint to a
// <canvas> and expose almost no accessibility tree, so `snapshot` comes back
// near-empty and agents get stuck looking for refs that will never exist
@@ -3077,7 +3116,40 @@ async fn handle_screenshot(cmd: &Value, state: &mut DaemonState) -> Result<Value
)
.await?;
// Downscale the saved image so retina/full-page shots fit an agent's image
// reader and screenshot pixels line up with `click x y` CSS px (issue #42).
// Explicit --scale / --max-width / --max-height win; otherwise a default cap
// (2000px longest edge, AGENT_BROWSER_SCREENSHOT_MAX_EDGE overrides, 0 = off)
// applies. Annotated shots are left untouched so ref overlays stay aligned.
let mut resized: Option<(u32, u32)> = None;
if !annotate {
let scale = cmd.get("scale").and_then(|v| v.as_f64());
let max_w = cmd
.get("maxWidth")
.and_then(|v| v.as_u64())
.map(|v| v as u32);
let max_h = cmd
.get("maxHeight")
.and_then(|v| v.as_u64())
.map(|v| v as u32);
let default_edge = if scale.is_none() && max_w.is_none() && max_h.is_none() {
std::env::var("AGENT_BROWSER_SCREENSHOT_MAX_EDGE")
.ok()
.and_then(|s| s.parse::<u32>().ok())
.or(Some(2000))
.filter(|&e| e > 0)
} else {
None
};
resized = downscale_screenshot(&result.path, scale, max_w, max_h, default_edge);
}
let mut response = json!({ "path": absolutize_saved_path(&result.path) });
if let Some((w, h)) = resized {
response["width"] = json!(w);
response["height"] = json!(h);
response["resized"] = json!(true);
}
if !result.annotations.is_empty() {
response["annotations"] = serde_json::to_value(&result.annotations)
.map_err(|e| format!("Failed to serialize annotations: {}", e))?;
@@ -3093,6 +3165,204 @@ async fn handle_screenshot(cmd: &Value, state: &mut DaemonState) -> Result<Value
Ok(response)
}
/// Downscale a saved screenshot in place (issue #42). Resolves the target longest
/// edge from `scale` (fraction of current), explicit `max_w`/`max_h` caps, or a
/// `default_edge` cap — whichever yields the smaller image. Only ever shrinks;
/// no-op (returns None) if the image is already within bounds or can't be read.
/// Returns the new (width, height) when it actually resized.
fn downscale_screenshot(
path: &str,
scale: Option<f64>,
max_w: Option<u32>,
max_h: Option<u32>,
default_edge: Option<u32>,
) -> Option<(u32, u32)> {
let img = image::open(path).ok()?;
let (w, h) = (img.width(), img.height());
if w == 0 || h == 0 {
return None;
}
// Collect candidate scale factors (≤ 1.0); the smallest wins.
let mut factor = 1.0f64;
if let Some(s) = scale {
factor = factor.min(s);
}
if let Some(mw) = max_w {
if w > mw {
factor = factor.min(mw as f64 / w as f64);
}
}
if let Some(mh) = max_h {
if h > mh {
factor = factor.min(mh as f64 / h as f64);
}
}
if let Some(edge) = default_edge {
let longest = w.max(h);
if longest > edge {
factor = factor.min(edge as f64 / longest as f64);
}
}
if factor >= 1.0 {
return None; // already within bounds — never upscale
}
let nw = ((w as f64 * factor).round() as u32).max(1);
let nh = ((h as f64 * factor).round() as u32).max(1);
let resized = img.resize(nw, nh, image::imageops::FilterType::Lanczos3);
resized.save(path).ok()?;
Some((resized.width(), resized.height()))
}
/// `canvas list` — enumerate <canvas> elements (size, visibility, whether
/// toDataURL is usable) so an agent can pick one to capture on a canvas/WebGL app
/// where snapshot/DOM reads see nothing.
async fn handle_canvas_list(state: &DaemonState) -> Result<Value, String> {
let mgr = state.browser.as_ref().ok_or("Browser not launched")?;
let js = r#"(() => [...document.querySelectorAll('canvas')].map((c, i) => {
const r = c.getBoundingClientRect();
let toDataUrl = true, tainted = false;
try { c.toDataURL('image/png'); } catch (e) { toDataUrl = false; tainted = true; }
return {
index: i,
backingWidth: c.width, backingHeight: c.height,
cssWidth: Math.round(r.width), cssHeight: Math.round(r.height),
visible: r.width > 0 && r.height > 0 && r.bottom > 0 && r.top < innerHeight,
id: c.id || null, className: c.className || null,
toDataUrl, tainted,
};
}))()"#;
let canvases = mgr.evaluate(js, None).await?;
Ok(json!({ "canvases": canvases }))
}
/// `canvas capture [selector] [path]` — save a canvas's rendered pixels to PNG.
/// Prefers `toDataURL` (full backing-store resolution); falls back to a CDP
/// screenshot of the canvas element when toDataURL is blank (WebGL without
/// preserveDrawingBuffer, e.g. Figma) or throws (cross-origin tainted). This is
/// how chrome-use "sees" canvas/WebGL apps that expose no DOM or refs.
async fn handle_canvas_capture(cmd: &Value, state: &mut DaemonState) -> Result<Value, String> {
let selector = cmd
.get("selector")
.and_then(|v| v.as_str())
.map(String::from);
let force_screenshot = cmd
.get("forceScreenshot")
.and_then(|v| v.as_bool())
.unwrap_or(false);
let out_path = cmd.get("path").and_then(|v| v.as_str()).map(String::from);
let id = cmd.get("id").and_then(|v| v.as_str()).unwrap_or("0");
// Probe: locate the canvas, get its bbox, and try toDataURL (unless forced).
let probe = {
let mgr = state.browser.as_ref().ok_or("Browser not launched")?;
let sel_lit = match &selector {
Some(s) => serde_json::to_string(s).unwrap_or_else(|_| "null".to_string()),
None => "null".to_string(),
};
let js = format!(
r#"(() => {{
const sel = {sel_lit};
const c = sel ? document.querySelector(sel)
: [...document.querySelectorAll('canvas')].sort((a,b)=>(b.width*b.height)-(a.width*a.height))[0];
if (!c) return {{ found: false, count: document.querySelectorAll('canvas').length }};
const r = c.getBoundingClientRect();
let dataUrl = null, err = null;
if ({try_data}) {{ try {{ dataUrl = c.toDataURL('image/png'); }} catch (e) {{ err = String(e && e.message || e); }} }}
return {{ found: true, w: c.width, h: c.height, x: r.x, y: r.y, cw: r.width, ch: r.height, dataUrl, err }};
}})()"#,
try_data = !force_screenshot
);
mgr.evaluate(&js, None).await?
};
if !probe
.get("found")
.and_then(|v| v.as_bool())
.unwrap_or(false)
{
let count = probe.get("count").and_then(|v| v.as_u64()).unwrap_or(0);
return Err(format!(
"canvas: no canvas matched{} ({count} canvas element(s) on the page — try `canvas list`)",
selector
.as_deref()
.map(|s| format!(" `{s}`"))
.unwrap_or_default()
));
}
let backing_w = probe.get("w").and_then(|v| v.as_u64()).unwrap_or(0);
let backing_h = probe.get("h").and_then(|v| v.as_u64()).unwrap_or(0);
// Decode toDataURL if present; a WebGL canvas without preserveDrawingBuffer
// returns a blank PNG (tiny when compressed), so reject suspiciously small
// results and fall back to the screenshot path.
let decoded: Option<Vec<u8>> = probe
.get("dataUrl")
.and_then(|v| v.as_str())
.and_then(|u| u.split_once(",").map(|(_, b)| b.to_string()))
.and_then(|b64| {
base64::Engine::decode(&base64::engine::general_purpose::STANDARD, b64.as_bytes()).ok()
});
let use_data = !force_screenshot && decoded.as_ref().map(|d| d.len() > 1024).unwrap_or(false);
if use_data {
let bytes = decoded.unwrap();
let path = out_path.unwrap_or_else(|| {
std::env::temp_dir()
.join(format!("canvas-{id}.png"))
.to_string_lossy()
.into_owned()
});
std::fs::write(&path, &bytes).map_err(|e| format!("canvas: write {path}: {e}"))?;
Ok(json!({
"path": absolutize_saved_path(&path),
"method": "toDataURL",
"width": backing_w,
"height": backing_h,
}))
} else {
// Fallback: screenshot the canvas element (or its bbox clip).
let clip = if selector.is_none() {
let x = probe.get("x").and_then(|v| v.as_f64()).unwrap_or(0.0);
let y = probe.get("y").and_then(|v| v.as_f64()).unwrap_or(0.0);
let cw = probe.get("cw").and_then(|v| v.as_f64()).unwrap_or(0.0);
let ch = probe.get("ch").and_then(|v| v.as_f64()).unwrap_or(0.0);
Some((x, y, cw, ch))
} else {
None
};
let options = ScreenshotOptions {
selector: selector.clone(),
path: out_path,
clip,
..ScreenshotOptions::default()
};
let mgr = state.browser.as_ref().ok_or("Browser not launched")?;
let session_id = mgr.active_session_id()?.to_string();
let result = screenshot::take_screenshot(
&mgr.client,
&session_id,
&state.ref_map,
&options,
&state.iframe_sessions,
)
.await?;
let why = probe
.get("err")
.and_then(|v| v.as_str())
.map(|e| format!("toDataURL failed ({e})"))
.unwrap_or_else(|| {
"toDataURL blank/unavailable (WebGL no preserveDrawingBuffer)".into()
});
Ok(json!({
"path": absolutize_saved_path(&result.path),
"method": "screenshot",
"note": format!("captured rendered pixels via screenshot — {why}"),
}))
}
}
async fn handle_click(cmd: &Value, state: &mut DaemonState) -> Result<Value, String> {
// First-class coordinate click (issue #8.4): click a raw viewport point with
// no element resolution. Parsed from `click <x> <y>` / `click --coords x,y`.
@@ -3251,7 +3521,7 @@ async fn handle_fill(cmd: &Value, state: &mut DaemonState) -> Result<Value, Stri
let mgr = state.browser.as_ref().ok_or("Browser not launched")?;
let session_id = mgr.active_session_id()?.to_string();
interaction::fill(
let engine = interaction::fill(
&mgr.client,
&session_id,
&state.ref_map,
@@ -3260,7 +3530,9 @@ async fn handle_fill(cmd: &Value, state: &mut DaemonState) -> Result<Value, Stri
&state.iframe_sessions,
)
.await?;
Ok(json!({ "filled": selector }))
// Echo the input path used (input/contenteditable/codemirror5/monaco/select)
// so the agent can confirm a rich editor was handled, not silently no-op'd (#41).
Ok(json!({ "filled": selector, "engine": engine }))
}
async fn handle_type(cmd: &Value, state: &mut DaemonState) -> Result<Value, String> {
@@ -4980,6 +5252,19 @@ async fn handle_tab_close(cmd: &Value, state: &mut DaemonState) -> Result<Value,
async fn handle_viewport(cmd: &Value, state: &mut DaemonState) -> Result<Value, String> {
let mgr = state.browser.as_ref().ok_or("Browser not launched")?;
// `viewport reset` clears the device-metrics override and restores the real
// layout viewport (the launched window's size, or — over the relay — the
// user's actual Chrome window).
if cmd.get("reset").and_then(|v| v.as_bool()).unwrap_or(false) {
mgr.clear_viewport().await?;
state.viewport = None;
if let Some(ref server) = state.stream_server {
server.set_viewport(1280, 720).await;
}
return Ok(json!({ "reset": true }));
}
let width = cmd.get("width").and_then(|v| v.as_i64()).unwrap_or(1280) as i32;
let height = cmd.get("height").and_then(|v| v.as_i64()).unwrap_or(720) as i32;
let scale = cmd
+225 -45
View File
@@ -509,6 +509,13 @@ pub struct BrowserManager {
/// for `RELAY_PRUNE_MISSES` consecutive snapshots; any snapshot that includes
/// it resets the counter. Keyed by stable target_id.
relay_target_misses: HashMap<String, u32>,
/// Whether the relay accepted this session's group announcement and is
/// therefore scoping `Target.getTargets` to our own tab group (issue #40).
/// When true the daemon can safely adopt new targets again (follow-popup,
/// cross-session adopt) — the relay has already filtered out foreign tabs.
/// When false (launch-on-real-CDP, or an older relay that didn't answer the
/// announce) the daemon keeps strict daemon-side isolation.
relay_scoped: bool,
next_tab_id: u32,
/// Whether to enable the CDP `Runtime` domain (console / error / exception capture).
/// OFF by default for stealth: a live `Runtime.enable` is a detectable CDP signal
@@ -646,6 +653,7 @@ impl BrowserManager {
created_targets: HashSet::new(),
active_target_id: None,
relay_target_misses: HashMap::new(),
relay_scoped: false,
next_tab_id: 1,
capture_console: console_capture_enabled(),
};
@@ -749,6 +757,7 @@ impl BrowserManager {
created_targets: HashSet::new(),
active_target_id: None,
relay_target_misses: HashMap::new(),
relay_scoped: false,
next_tab_id: 1,
capture_console: console_capture_enabled(),
};
@@ -815,6 +824,106 @@ impl BrowserManager {
Ok(by_id.into_values().collect())
}
/// Every tab the relay knows, UNSCOPED (ignores group scoping) — for explicit
/// cross-group adoption (`chrome-use adopt`). Falls back to the scoped
/// `collect_page_targets` on a relay/browser that doesn't support the
/// unscoped query. Retries a few times over the relay (discovery is eventual).
async fn collect_all_targets(&self) -> Result<Vec<TargetInfo>, String> {
let rounds = if crate::connect::relay_url().is_some() {
3
} else {
1
};
let mut by_id: HashMap<String, TargetInfo> = HashMap::new();
let mut any_ok = false;
for i in 0..rounds {
if i > 0 {
tokio::time::sleep(Duration::from_millis(150)).await;
}
if let Ok(result) = self
.client
.send_command_typed::<_, GetTargetsResult>(
"ABRelay.getAllTargets",
&json!({}),
None,
)
.await
{
any_ok = true;
for t in result.target_infos.into_iter().filter(should_track_target) {
by_id.entry(t.target_id.clone()).or_insert(t);
}
}
}
if any_ok {
Ok(by_id.into_values().collect())
} else {
// Older relay without ABRelay.getAllTargets → best-effort scoped list.
self.collect_page_targets().await
}
}
/// Adopt a specific pre-existing tab matched by `spec` (an exact CDP
/// `targetId`, or a case-insensitive substring of the tab URL) WITHOUT opening
/// a new tab — for `chrome-use adopt`. Attaches it (the relay tags it into our
/// group), tracks + pins it. Errors if nothing matches (never creates a tab).
async fn adopt_existing_target(&mut self, spec: &str) -> Result<(), String> {
let all = self.collect_all_targets().await?;
let spec_l = spec.to_lowercase();
let target = all
.iter()
.find(|t| t.target_id == spec)
.or_else(|| all.iter().find(|t| t.url.to_lowercase().contains(&spec_l)))
.ok_or_else(|| {
let mut open: Vec<String> = all
.iter()
.map(|t| {
let u = if t.url.len() > 80 {
&t.url[..80]
} else {
&t.url
};
u.to_string()
})
.collect();
open.sort();
open.dedup();
format!(
"adopt: no open tab matching `{spec}` (by targetId or URL substring).\n\
{} tab(s) the extension can see:\n {}",
open.len(),
open.join("\n ")
)
})?
.clone();
let attach: AttachToTargetResult = self
.client
.send_command_typed(
"Target.attachToTarget",
&AttachToTargetParams {
target_id: target.target_id.clone(),
flatten: true,
},
None,
)
.await?;
let tab_id = self.assign_tab_id();
self.pages.push(PageInfo {
tab_id,
label: None,
target_id: target.target_id.clone(),
session_id: attach.session_id.clone(),
url: target.url.clone(),
title: sanitize_title(&target.title),
target_type: target.target_type.clone(),
});
self.active_page_index = self.pages.len() - 1;
self.pin_active_target();
self.enable_domains(&attach.session_id).await?;
Ok(())
}
async fn discover_and_attach_targets(&mut self) -> Result<(), String> {
self.client
.send_command_typed::<_, Value>(
@@ -824,6 +933,21 @@ impl BrowserManager {
)
.await?;
// Announce our group FIRST so the relay scopes the getTargets below to our
// own tab group (issue #40). On a launched browser this is a no-op.
let scoped = self.announce_group().await;
// `chrome-use adopt <spec>`: adopt a specific PRE-EXISTING tab instead of
// creating one — true zero-new-tab reading of the user's own tab. The
// directive rides in via env so it takes effect at first connect (before
// any about:blank would be made). If nothing matches, error out rather
// than fall back to creating a tab.
if let Ok(spec) = std::env::var("AGENT_BROWSER_ADOPT") {
if !spec.trim().is_empty() {
return self.adopt_existing_target(spec.trim()).await;
}
}
let page_targets: Vec<TargetInfo> = self.collect_page_targets().await?;
if page_targets.is_empty() {
@@ -870,19 +994,19 @@ impl BrowserManager {
self.active_page_index = 0;
self.pin_active_target();
self.enable_domains(&attach_result.session_id).await?;
} else if self.agent_group().is_some() {
// STRICT MULTI-AGENT ISOLATION (relay / the user's real Chrome).
// `page_targets` here are the USER's and OTHER agents' tabs. A tab
// group belongs to exactly ONE agent, so this session must NOT adopt
// any of them — it tracks ONLY tabs it creates (its own colored group)
// plus popups it opens. Adopting foreign tabs is precisely what let
// another concurrent agent's tab churn drop the tab we were driving and
// drift eval/click onto the wrong page (multi-agent failure). Open our
// own dedicated background tab in the session's group and pin it; the
// user's / other agents' tabs stay invisible to us.
} else if self.agent_group().is_some() && !scoped {
// STRICT MULTI-AGENT ISOLATION fallback (relay, but the group announce
// didn't take — e.g. an older relay). Without relay-side scoping,
// `page_targets` could be the USER's and OTHER agents' tabs, so this
// session must NOT adopt any of them — adopting foreign tabs is what let
// another agent's tab churn drop the tab we were driving (multi-agent
// failure). Open our own dedicated background tab and pin it instead.
self.tab_new(None, None).await?;
} else {
// A browser WE launched: every tab is ours, so adopt them all.
// Either a browser WE launched (every tab is ours) or the relay has
// scoped getTargets to our own tab group (#40) — so `page_targets` are
// all ours: adopt them (this restores follow-popup + cross-session
// adopt under isolation, since foreign tabs were already filtered out).
for target in &page_targets {
let attach_result: AttachToTargetResult = self
.client
@@ -1580,7 +1704,11 @@ impl BrowserManager {
// in the synthesized targetInfo), so don't adopt anything: the agent drives
// only tabs it explicitly created, and pop-ups (e.g. an OAuth/login window)
// are the user's. A launched browser (every tab ours) still follows pop-ups.
if self.agent_group().is_some() {
// Strict isolation only when on the relay WITHOUT group scoping: there a
// pop-up can't be told apart from a foreign tab, so adopt nothing. When the
// relay IS scoping (#40), getTargets returns only our group, so a tab that
// appeared after our own action is genuinely ours (a pop-up) — adopt it.
if self.agent_group().is_some() && !self.relay_scoped {
return None;
}
let result: GetTargetsResult = self
@@ -1658,16 +1786,17 @@ impl BrowserManager {
.collect();
let live_ids: HashSet<String> = live.iter().map(|t| t.target_id.clone()).collect();
let on_relay = self.agent_group().is_some();
// When the relay scopes getTargets to our group (#40), `live` is already
// only our own tabs, so adopting unknown ones is safe (a freshly-opened
// pop-up). Without scoping, keep strict isolation: never adopt a tab we
// didn't create — it belongs to the user or another agent.
let strict_isolation = on_relay && !self.relay_scoped;
for target in &live {
if self.update_page_target_info(target) {
continue;
}
// STRICT MULTI-AGENT ISOLATION: on the relay (the user's real Chrome,
// shared with other agents), NEVER adopt a tab this session didn't
// create — it belongs to the user or another agent's group. Only a
// browser we launched (every tab ours) adopts unknown targets.
if on_relay {
if strict_isolation {
continue;
}
let attach_result: AttachToTargetResult = match self
@@ -1821,9 +1950,17 @@ impl BrowserManager {
/// CDP browser the endpoint is strict, so we must NOT send the custom param —
/// hence `None` there. We detect the relay by matching our `ws_url` against
/// the live relay URL the native-messaging host published.
/// Whether this manager is driving the user's real Chrome through the
/// `ab-connect` extension relay (vs. a browser we launched or a direct CDP
/// endpoint). Detected by matching our `ws_url` against the live relay URL
/// the native-messaging host published. Used to avoid relay-unsafe CDP that
/// would disturb the user's window (e.g. Browser.setContentsSize, issue #47).
fn via_relay(&self) -> bool {
crate::connect::relay_url().as_deref() == Some(self.ws_url.as_str())
}
fn agent_group(&self) -> Option<String> {
let via_relay = crate::connect::relay_url().as_deref() == Some(self.ws_url.as_str());
if !via_relay {
if !self.via_relay() {
return None;
}
let name = DAEMON_SESSION
@@ -1837,6 +1974,25 @@ impl BrowserManager {
}
}
/// Tell the relay which tab group this session owns so it can scope
/// `Target.getTargets` to us (issue #40). Only meaningful on the relay; a
/// no-op (returns false) on a launched/real-CDP connection. Sets and returns
/// `relay_scoped`: when true, the daemon can trust getTargets to contain only
/// our group and re-enable adopting new tabs (pop-ups, cross-session adopt).
async fn announce_group(&mut self) -> bool {
let Some(group) = self.agent_group() else {
self.relay_scoped = false;
return false;
};
let ok = self
.client
.send_command_typed::<_, Value>("ABRelay.setGroup", &json!({ "group": group }), None)
.await
.is_ok();
self.relay_scoped = ok;
ok
}
pub async fn tab_new(
&mut self,
url: Option<&str>,
@@ -2013,32 +2169,39 @@ impl BrowserManager {
.await?;
// Screencast captures the actual content area, not the emulated CSS
// viewport, so resize the content area to match.
if let Ok(target_id) = self.active_target_id() {
if let Ok(window_info) = self
.client
.send_command(
"Browser.getWindowForTarget",
Some(json!({ "targetId": target_id })),
None,
)
.await
{
if let Some(window_id) = window_info.get("windowId").and_then(|v| v.as_i64()) {
if let Err(e) = self
.client
.send_command(
"Browser.setContentsSize",
Some(json!({
"windowId": window_id,
"width": width,
"height": height,
})),
None,
)
.await
{
eprintln!("Browser.setContentsSize failed (experimental CDP): {e}");
// viewport, so resize the content area to match — but ONLY for a browser
// we launched. Over the ab-connect relay the "window" is the user's real
// Chrome window, and Browser.setContentsSize would physically resize it
// (issue #47) — the exact thing the CDP device-metrics override exists to
// avoid. The Emulation override above already gives the tab the requested
// CSS viewport without touching the OS window, so skip the resize there.
if !self.via_relay() {
if let Ok(target_id) = self.active_target_id() {
if let Ok(window_info) = self
.client
.send_command(
"Browser.getWindowForTarget",
Some(json!({ "targetId": target_id })),
None,
)
.await
{
if let Some(window_id) = window_info.get("windowId").and_then(|v| v.as_i64()) {
if let Err(e) = self
.client
.send_command(
"Browser.setContentsSize",
Some(json!({
"windowId": window_id,
"width": width,
"height": height,
})),
None,
)
.await
{
eprintln!("Browser.setContentsSize failed (experimental CDP): {e}");
}
}
}
}
@@ -2047,6 +2210,22 @@ impl BrowserManager {
Ok(())
}
/// Clear the CDP device-metrics override (`viewport reset`), restoring the
/// tab's real layout viewport. Never touches the OS window, so it is safe on
/// the relay (we never physically resized the user's window — see
/// `set_viewport`).
pub async fn clear_viewport(&self) -> Result<(), String> {
let session_id = self.active_session_id()?;
self.client
.send_command(
"Emulation.clearDeviceMetricsOverride",
Some(json!({})),
Some(session_id),
)
.await?;
Ok(())
}
pub async fn set_user_agent(&self, user_agent: &str) -> Result<(), String> {
let session_id = self.active_session_id()?;
self.client
@@ -2630,6 +2809,7 @@ async fn initialize_lightpanda_manager(
created_targets: HashSet::new(),
active_target_id: None,
relay_target_misses: HashMap::new(),
relay_scoped: false,
next_tab_id: 1,
capture_console: console_capture_enabled(),
};
+10
View File
@@ -21,6 +21,16 @@ pub async fn run_daemon(session: &str) {
// (via the ab-connect extension) land in a per-session Chrome tab group.
let _ = super::browser::DAEMON_SESSION.set(session.to_string());
// Bootstrap / refresh the site-adapter pack in the background (first-run +
// periodic TTL). This populates ~/.chrome-use/sites/.index.json so navigation
// can auto-suggest `site` commands for the page you land on, with zero added
// latency to any command. Best-effort; offline is a no-op.
if crate::site::needs_refresh() {
tokio::spawn(async {
let _ = crate::site::update().await;
});
}
let socket_dir = get_daemon_socket_dir();
if !socket_dir.exists() {
let _ = fs::create_dir_all(&socket_dir);
+30 -4
View File
@@ -1529,9 +1529,27 @@ pub async fn get_element_input_value(
.send_command_typed(
"Runtime.callFunctionOn",
&CallFunctionOnParams {
function_declaration:
"function() { return typeof this.value === 'string' ? this.value : ''; }"
.to_string(),
// Read rich-editor content too (issue #41): CodeMirror 5 / Monaco
// keep their text in a model, not `.value`; contenteditable keeps
// it as innerText. Falls back to `.value` for plain inputs.
function_declaration: r#"function() {
const el = this;
const cm5 = el.closest && el.closest('.CodeMirror');
if (cm5 && cm5.CodeMirror) return cm5.CodeMirror.getValue();
if (window.monaco && monaco.editor) {
try {
const eds = monaco.editor.getEditors ? monaco.editor.getEditors() : [];
const ed = eds.find(e => e.getDomNode && e.getDomNode().contains(el)) || eds[0];
if (ed) return ed.getValue();
const m = monaco.editor.getModels ? monaco.editor.getModels() : [];
if (m[0]) return m[0].getValue();
} catch (e) {}
}
if (typeof el.value === 'string') return el.value;
if (el.isContentEditable) return el.innerText;
return '';
}"#
.to_string(),
object_id: Some(object_id),
arguments: None,
return_by_value: Some(true),
@@ -1609,7 +1627,15 @@ pub async fn get_element_bounding_box(
&CallFunctionOnParams {
function_declaration: r#"function() {
const r = this.getBoundingClientRect();
return { x: r.x, y: r.y, width: r.width, height: r.height };
const inViewport = r.bottom > 0 && r.right > 0
&& r.top < (innerHeight || document.documentElement.clientHeight)
&& r.left < (innerWidth || document.documentElement.clientWidth);
return {
x: r.x, y: r.y, width: r.width, height: r.height,
centerX: Math.round(r.x + r.width / 2),
centerY: Math.round(r.y + r.height / 2),
inViewport,
};
}"#
.to_string(),
object_id: Some(object_id),
+52 -17
View File
@@ -579,7 +579,7 @@ pub async fn fill(
selector_or_ref: &str,
value: &str,
iframe_sessions: &HashMap<String, String>,
) -> Result<(), String> {
) -> Result<String, String> {
let (object_id, effective_session_id) = resolve_element_object_id(
client,
session_id,
@@ -590,14 +590,15 @@ pub async fn fill(
.await?;
// Emulate a real edit so framework-controlled inputs (React/Vue) and
// site-side listeners actually see the change (issue #25): the old path set
// `this.value` directly and used Input.insertText, which left React's
// internal value-tracker out of sync and never fired change/blur — so
// dependent logic (e.g. Mercari's postal-code → 都道府県 autocomplete) never
// ran even though the value was visible. Set the value through the element's
// PROTOTYPE setter (which React's _valueTracker hooks), then dispatch
// input → change → blur/focusout. `type <sel> <text>` remains for sites that
// need per-keystroke events.
// site-side listeners actually see the change (issue #25): set the value
// through the element's PROTOTYPE setter (which React's _valueTracker hooks),
// then dispatch input → changeblur/focusout. Beyond plain inputs, detect
// rich editors and use their own API/events (issue #41): CodeMirror 5 and
// Monaco have a model that `.value`/`textContent` can't touch; ProseMirror /
// contenteditable need `execCommand('insertText')` so beforeinput/input fire
// (a raw `textContent =` corrupts PM's doc and skips React composers).
// Returns the engine used so the caller can report it. `type <sel> <text>`
// remains for sites that need per-keystroke events.
let fill_js = format!(
r#"function() {{
const el = this;
@@ -605,13 +606,43 @@ pub async fn fill(
try {{ el.focus(); }} catch (e) {{}}
const tag = el.tagName;
const fire = (type, ctor) => el.dispatchEvent(new (ctor || Event)(type, {{ bubbles: true }}));
if (tag === 'SELECT') {{
el.value = v; fire('input'); fire('change'); return true;
// CodeMirror 5: a hidden <textarea> inside .CodeMirror with a live instance.
const cm5 = el.closest && el.closest('.CodeMirror');
if (cm5 && cm5.CodeMirror) {{ cm5.CodeMirror.setValue(v); return 'codemirror5'; }}
// Monaco: global `monaco`; prefer the editor whose DOM contains el.
if (window.monaco && monaco.editor) {{
try {{
const eds = monaco.editor.getEditors ? monaco.editor.getEditors() : [];
const ed = eds.find(e => e.getDomNode && e.getDomNode().contains(el)) || eds[0];
if (ed) {{ ed.setValue(v); return 'monaco'; }}
const models = monaco.editor.getModels ? monaco.editor.getModels() : [];
if (models[0]) {{ models[0].setValue(v); return 'monaco'; }}
}} catch (e) {{}}
}}
if (tag === 'SELECT') {{ el.value = v; fire('input'); fire('change'); return 'select'; }}
if (el.isContentEditable) {{
el.textContent = v; fire('input', window.InputEvent || Event); fire('change');
try {{ el.blur(); }} catch (e) {{}} fire('focusout'); return true;
// ProseMirror / contenteditable: select-all then insertText fires
// beforeinput/input that PM and React composers listen for.
let ok = false;
try {{
const sel = window.getSelection();
const range = document.createRange();
range.selectNodeContents(el);
sel.removeAllRanges();
sel.addRange(range);
ok = document.execCommand('insertText', false, v);
}} catch (e) {{}}
if (!ok) {{ el.textContent = v; fire('input', window.InputEvent || Event); }}
fire('change');
try {{ el.blur(); }} catch (e) {{}}
fire('focusout');
return ok ? 'contenteditable' : 'contenteditable-fallback';
}}
const proto = tag === 'TEXTAREA' ? window.HTMLTextAreaElement.prototype
: window.HTMLInputElement.prototype;
const desc = Object.getOwnPropertyDescriptor(proto, 'value');
@@ -623,13 +654,13 @@ pub async fn fill(
fire('change');
try {{ el.blur(); }} catch (e) {{}}
fire('focusout'); // blur-triggered lookups/validation
return true;
return 'input';
}}"#,
val = serde_json::to_string(value).unwrap_or_default()
);
client
.send_command_typed::<_, Value>(
let result: EvaluateResult = client
.send_command_typed(
"Runtime.callFunctionOn",
&CallFunctionOnParams {
function_declaration: fill_js,
@@ -642,7 +673,11 @@ pub async fn fill(
)
.await?;
Ok(())
Ok(result
.result
.value
.and_then(|v| v.as_str().map(String::from))
.unwrap_or_else(|| "input".to_string()))
}
#[allow(clippy::too_many_arguments)]
+273 -4
View File
@@ -52,6 +52,22 @@ pub struct RelayState {
pending: HashMap<i64, (ClientId, Value)>,
/// monotonic source of relay-global command ids
next_global_id: i64,
/// Group-scoped isolation (issue #40). A tab group belongs to exactly one
/// agent/session; the relay scopes `Target.getTargets` per client to its own
/// group so the daemon can safely adopt new tabs (follow-popup, cross-session
/// adopt) without ever seeing the user's or another agent's tabs.
///
/// clientId -> group name. A client that never announced a group (older
/// daemon) is absent here and gets the full, UNSCOPED target list — so this
/// is fully backward-compatible.
client_groups: HashMap<ClientId, String>,
/// targetId -> group name. Created tabs are tagged from `Target.createTarget`'s
/// `agentGroup`; an explicitly adopted tab is tagged to the adopter; a pop-up
/// inherits its opener's group (needs the extension to report `openerTargetId`).
target_group: HashMap<String, String>,
/// relay-global id of an in-flight `Target.createTarget` -> the `agentGroup`
/// it carried, so the reply's `targetId` can be tagged with that group.
pending_create: HashMap<i64, String>,
}
/// What to do with a raw CDP command received from a `CdpClient`.
@@ -95,6 +111,7 @@ impl RelayState {
/// `pending` entries don't leak.
pub fn drop_client(&mut self, client_id: ClientId) {
self.pending.retain(|_, (cid, _)| *cid != client_id);
self.client_groups.remove(&client_id);
}
/// Route a raw CDP command `{id, method, params?, sessionId?}` from a
@@ -123,12 +140,28 @@ impl RelayState {
"jsVersion": ""
}
})),
// Non-CDP control message: a daemon announces which tab group
// (session) it owns, so getTargets can be scoped to it (issue #40).
"ABRelay.setGroup" => {
if let Some(g) = params.get("group").and_then(|g| g.as_str()) {
if !g.is_empty() {
self.client_groups.insert(client_id, g.to_string());
}
}
ClientRoute::Local(json!({ "id": id, "result": {} }))
}
// Discovery is best-effort and event-driven in real CDP; abs only
// reads the getTargets result, so an empty ack is enough here.
"Target.setDiscoverTargets" | "Target.setAutoAttach" => {
ClientRoute::Local(json!({ "id": id, "result": {} }))
}
"Target.getTargets" => {
// Unscoped discovery for EXPLICIT cross-group adoption (`chrome-use
// adopt`): returns every target the extension has attached, ignoring
// group scoping, so an agent can find a specific pre-existing tab (the
// user's, another session's) by URL/targetId and adopt it. Isolation
// is preserved because the daemon only acts on the one tab it then
// attaches (which the relay re-tags into the adopter's group).
"ABRelay.getAllTargets" => {
let infos: Vec<Value> = self
.targets
.values()
@@ -136,15 +169,44 @@ impl RelayState {
.collect();
ClientRoute::Local(json!({ "id": id, "result": { "targetInfos": infos } }))
}
"Target.getTargets" => {
// Scope to the client's own group when it announced one; an
// un-announced (legacy) client gets the full list (back-compat).
let scoped = self.client_groups.get(&client_id).cloned();
let infos: Vec<Value> = self
.targets
.iter()
.filter(|(tid, _)| match &scoped {
Some(g) => self
.target_group
.get(*tid)
.map(|tg| tg == g)
.unwrap_or(false),
None => true,
})
.map(|(_, t)| t.target_info.clone())
.collect();
ClientRoute::Local(json!({ "id": id, "result": { "targetInfos": infos } }))
}
"Target.attachToTarget" => {
let target_id = params
.get("targetId")
.and_then(|t| t.as_str())
.unwrap_or("");
match self.targets.get(target_id) {
Some(entry) => ClientRoute::Local(
json!({ "id": id, "result": { "sessionId": entry.session_id } }),
),
Some(entry) => {
let session_id = entry.session_id.clone();
// Explicitly adopting a target makes it this client's
// (cross-session adopt, #21) — tag it into the adopter's
// group so it stays in that client's scoped getTargets and
// isn't churn-pruned.
if let Some(g) = self.client_groups.get(&client_id).cloned() {
self.target_group.insert(target_id.to_string(), g);
}
ClientRoute::Local(
json!({ "id": id, "result": { "sessionId": session_id } }),
)
}
None => ClientRoute::Local(json!({
"id": id,
"error": { "code": -32602, "message": format!("No such target {target_id}") }
@@ -157,6 +219,18 @@ impl RelayState {
self.next_global_id += 1;
let gid = self.next_global_id;
self.pending.insert(gid, (client_id, id));
// Remember the group a createTarget carries so the reply's
// targetId can be tagged to the creating session (issue #40).
if method == "Target.createTarget" {
if let Some(g) = params.get("agentGroup").and_then(|g| g.as_str()) {
if !g.is_empty() {
self.pending_create.insert(gid, g.to_string());
self.client_groups
.entry(client_id)
.or_insert_with(|| g.to_string());
}
}
}
ClientRoute::Forward(json!({
"id": gid,
"method": "forwardCDPCommand",
@@ -201,6 +275,17 @@ impl RelayState {
&& msg.get("method").is_none()
{
let gid = msg.get("id").and_then(|i| i.as_i64());
// A createTarget reply: tag the new tab's targetId with the group the
// command carried, so it lands in the creating session's scope (#40).
if let Some(g) = gid.and_then(|g| self.pending_create.remove(&g)) {
if let Some(tid) = msg
.get("result")
.and_then(|r| r.get("targetId"))
.and_then(|t| t.as_str())
{
self.target_group.insert(tid.to_string(), g);
}
}
let (to, orig_id) = match gid.and_then(|g| self.pending.remove(&g)) {
Some((client_id, orig)) => (Some(client_id), orig),
// No mapping (stale/unknown id) — fall back to broadcasting with
@@ -241,6 +326,27 @@ impl RelayState {
.and_then(|s| s.as_str())
.unwrap_or("")
.to_string();
// Attribute the tab to a group for scoping (issue #40),
// unless we already know it (createTarget tag). An
// explicit `abGroup` from the extension wins; otherwise a
// pop-up inherits its opener's group via `openerTargetId`.
if !self.target_group.contains_key(tid) {
if let Some(g) = info
.get("abGroup")
.and_then(|g| g.as_str())
.filter(|g| !g.is_empty())
{
self.target_group.insert(tid.to_string(), g.to_string());
} else if let Some(opener) = info
.get("openerTargetId")
.and_then(|o| o.as_str())
.filter(|o| !o.is_empty())
{
if let Some(g) = self.target_group.get(opener).cloned() {
self.target_group.insert(tid.to_string(), g);
}
}
}
self.targets.insert(
tid.to_string(),
TargetEntry {
@@ -255,6 +361,15 @@ impl RelayState {
"Target.detachedFromTarget" => {
let gone = inner_params.get("sessionId").and_then(|s| s.as_str());
if let Some(gone) = gone {
let gone_tids: Vec<String> = self
.targets
.iter()
.filter(|(_, e)| e.session_id == gone)
.map(|(tid, _)| tid.clone())
.collect();
for tid in gone_tids {
self.target_group.remove(&tid);
}
self.targets.retain(|_, e| e.session_id != gone);
}
return vec![];
@@ -557,4 +672,158 @@ mod tests {
_ => panic!("expected ToExt"),
}
}
// === Group-scoped isolation (issue #40) ===
/// Drive the real create path: announce group, createTarget(agentGroup), feed
/// the ext reply (tags target→group) + the attachedToTarget event (creates the
/// entry). Returns nothing; mutates `s`.
fn create_in_group(s: &mut RelayState, client: ClientId, group: &str, tid: &str, sid: &str) {
s.route_client_command(
client,
&json!({ "id": 1, "method": "ABRelay.setGroup", "params": { "group": group } }),
);
let route = s.route_client_command(
client,
&json!({ "id": 2, "method": "Target.createTarget",
"params": { "url": "about:blank", "agentGroup": group } }),
);
let gid = match route {
ClientRoute::Forward(env) => env["id"].as_i64().unwrap(),
_ => panic!("createTarget must forward"),
};
s.handle_ext_message(&json!({ "id": gid, "result": { "targetId": tid } }), "");
s.handle_ext_message(
&json!({ "method": "forwardCDPEvent", "params": {
"method": "Target.attachedToTarget",
"params": { "sessionId": sid, "targetInfo": {
"targetId": tid, "type": "page", "url": "about:blank", "attached": true } } } }),
"",
);
}
fn get_target_ids(s: &mut RelayState, client: ClientId) -> Vec<String> {
match s.route_client_command(client, &json!({ "id": 9, "method": "Target.getTargets" })) {
ClientRoute::Local(v) => v["result"]["targetInfos"]
.as_array()
.unwrap()
.iter()
.map(|t| t["targetId"].as_str().unwrap().to_string())
.collect(),
_ => panic!("getTargets must be local"),
}
}
#[test]
fn get_targets_is_scoped_to_each_clients_group() {
let mut s = RelayState::new();
create_in_group(&mut s, 1, "agent-a", "ta", "sa");
create_in_group(&mut s, 2, "agent-b", "tb", "sb");
// Each client sees ONLY its own group's tab — never the other agent's.
assert_eq!(get_target_ids(&mut s, 1), vec!["ta"]);
assert_eq!(get_target_ids(&mut s, 2), vec!["tb"]);
}
#[test]
fn legacy_client_without_group_sees_all_targets() {
let mut s = RelayState::new();
create_in_group(&mut s, 1, "agent-a", "ta", "sa");
create_in_group(&mut s, 2, "agent-b", "tb", "sb");
// Client 3 never announced a group → full, unscoped list (back-compat).
let mut all = get_target_ids(&mut s, 3);
all.sort();
assert_eq!(all, vec!["ta", "tb"]);
}
#[test]
fn popup_inherits_opener_group_and_is_visible_to_that_client_only() {
let mut s = RelayState::new();
create_in_group(&mut s, 1, "agent-a", "ta", "sa");
create_in_group(&mut s, 2, "agent-b", "tb", "sb");
// A pop-up that agent-a's tab opened: extension reports openerTargetId=ta.
s.handle_ext_message(
&json!({ "method": "forwardCDPEvent", "params": {
"method": "Target.attachedToTarget",
"params": { "sessionId": "sp", "targetInfo": {
"targetId": "tp", "type": "page", "url": "https://oauth.example/",
"attached": true, "openerTargetId": "ta" } } } }),
"",
);
// Only agent-a sees the pop-up; agent-b never does.
let mut a = get_target_ids(&mut s, 1);
a.sort();
assert_eq!(a, vec!["ta", "tp"]);
assert_eq!(get_target_ids(&mut s, 2), vec!["tb"]);
}
#[test]
fn explicit_attach_tags_target_into_adopter_group() {
let mut s = RelayState::new();
// A pre-existing, ungrouped tab the extension reported (e.g. user's tab).
s.handle_ext_message(
&json!({ "method": "forwardCDPEvent", "params": {
"method": "Target.attachedToTarget",
"params": { "sessionId": "su", "targetInfo": {
"targetId": "tu", "type": "page", "url": "https://user.example/", "attached": true } } } }),
"",
);
// Client 1 (group agent-a) explicitly adopts it by targetId (#21).
s.route_client_command(
1,
&json!({ "id": 1, "method": "ABRelay.setGroup", "params": { "group": "agent-a" } }),
);
s.route_client_command(
1,
&json!({ "id": 2, "method": "Target.attachToTarget", "params": { "targetId": "tu" } }),
);
// Now it's in agent-a's scope and survives the scoped getTargets.
assert_eq!(get_target_ids(&mut s, 1), vec!["tu"]);
// A different agent still doesn't see it.
s.route_client_command(
2,
&json!({ "id": 1, "method": "ABRelay.setGroup", "params": { "group": "agent-b" } }),
);
assert!(get_target_ids(&mut s, 2).is_empty());
}
#[test]
fn get_all_targets_is_unscoped() {
let mut s = RelayState::new();
create_in_group(&mut s, 1, "agent-a", "ta", "sa");
create_in_group(&mut s, 2, "agent-b", "tb", "sb");
// Client 1's scoped getTargets sees only its own group...
assert_eq!(get_target_ids(&mut s, 1), vec!["ta"]);
// ...but ABRelay.getAllTargets returns EVERY target regardless of group
// (for explicit cross-group adoption).
let all = match s
.route_client_command(1, &json!({ "id": 1, "method": "ABRelay.getAllTargets" }))
{
ClientRoute::Local(v) => {
let mut ids: Vec<String> = v["result"]["targetInfos"]
.as_array()
.unwrap()
.iter()
.map(|t| t["targetId"].as_str().unwrap().to_string())
.collect();
ids.sort();
ids
}
_ => panic!("getAllTargets must be local"),
};
assert_eq!(all, vec!["ta", "tb"]);
}
#[test]
fn detach_clears_target_group() {
let mut s = RelayState::new();
create_in_group(&mut s, 1, "agent-a", "ta", "sa");
assert_eq!(get_target_ids(&mut s, 1), vec!["ta"]);
s.handle_ext_message(
&json!({ "method": "forwardCDPEvent", "params": {
"method": "Target.detachedFromTarget", "params": { "sessionId": "sa" } } }),
"",
);
assert!(get_target_ids(&mut s, 1).is_empty());
assert!(!s.target_group.contains_key("ta"));
}
}
+69 -5
View File
@@ -186,6 +186,26 @@ pub fn print_response_with_opts(resp: &Response, action: Option<&str>, opts: &Ou
}
if let Some(data) = &resp.data {
// Auto-trigger: when you land on / read a page whose domain has site
// adapters, surface them so the agent pulls structured data via
// `chrome-use site <name>/<cmd>` instead of scraping the DOM. (In --json
// mode this same info rides along in the `siteAdapters` field above.)
if let Some(hint) = data.get("siteAdapters") {
let domain = hint.get("domain").and_then(|v| v.as_str()).unwrap_or("");
let cmds: Vec<&str> = hint
.get("commands")
.and_then(|v| v.as_array())
.map(|a| a.iter().filter_map(|v| v.as_str()).collect())
.unwrap_or_default();
if !cmds.is_empty() {
eprintln!("💡 site adapters for {domain} — prefer these for structured data:");
eprintln!(" {}", color::dim(&cmds.join(", ")));
eprintln!(
" {}",
color::dim(&format!("e.g. chrome-use site {} --json", cmds[0]))
);
}
}
// A click that opened a new tab: surface it so the agent doesn't read the
// unchanged old page as a failed click (issue #24-A).
if let Some(opened) = data.get("openedTab") {
@@ -469,6 +489,17 @@ pub fn print_response_with_opts(resp: &Response, action: Option<&str>, opts: &Ou
println!("y: {}", y);
println!("width: {}", w);
println!("height: {}", h);
if let (Some(cx), Some(cy)) = (
obj.get("centerX").and_then(|v| v.as_i64()),
obj.get("centerY").and_then(|v| v.as_i64()),
) {
// Echoed in click-ready CSS px so the agent can paste straight
// into `click <centerX> <centerY>` (issue #43).
println!("center: {} {}", cx, cy);
}
if let Some(iv) = obj.get("inViewport").and_then(|v| v.as_bool()) {
println!("inViewport: {}", iv);
}
}
return;
}
@@ -1470,9 +1501,20 @@ Examples:
chrome-use fill - Clear and fill an input field
Usage: chrome-use fill <selector> <text>
chrome-use fill <selector> --file <path>
chrome-use fill <selector> --stdin
Clears the input field and fills it with the specified text.
This replaces any existing content in the field.
Clears the field and fills it with the text, replacing existing content.
Works on rich editors too (issue #41): CodeMirror 5, Monaco, ProseMirror and
plain contenteditable are detected and set via their own API / input events,
not a raw `.value` write and the response echoes which `engine` was used.
For framework inputs (React/Vue/Angular) the value goes through the native
setter so the form registers it (no more "pristine" Save no-ops).
Options:
--file <path> Read the value from a UTF-8 file (large/multiline text,
backticks/quotes/newlines/non-ASCII no shell escaping)
--stdin Read the value from stdin
Global Options:
--json Output as JSON
@@ -1481,7 +1523,8 @@ Global Options:
Examples:
chrome-use fill "#email" "user@example.com"
chrome-use fill @e3 "Hello World"
chrome-use fill "input[name='search']" "query"
chrome-use fill ".CodeMirror" --file ./article.md # set a CodeMirror editor
cat post.md | chrome-use fill @e7 --stdin
"##
}
"type" => {
@@ -1883,6 +1926,13 @@ Options:
--full, -f Capture full page (not just viewport)
[selector] Capture just an element (CSS or @ref), e.g. `screenshot ".header" h.png`
--clip <x,y,w,h> Capture a pixel region, e.g. `screenshot --clip 0,0,200,40 corner.png`
--max-width <px> Downscale so the image's width px (preserves aspect)
--max-height <px> Downscale so the image's height px
--scale <0..1> Downscale by a factor, e.g. 0.5 (DPR-1, so screenshot px
line up 1:1 with `click x y`)
Default: capped at 2000px longest edge unless overridden
(AGENT_BROWSER_SCREENSHOT_MAX_EDGE; 0 disables). Annotated
shots are never downscaled, so ref overlays stay aligned.
--annotate Overlay numbered labels on interactive elements.
Each label [N] corresponds to ref @eN from snapshot.
Prints a legend mapping labels to element roles/names.
@@ -1905,6 +1955,8 @@ Examples:
chrome-use screenshot --full ./full-page.png
chrome-use screenshot ".header .indicator" corner.png # just one element
chrome-use screenshot --clip 1600,0,200,40 corner.png # a pixel region
chrome-use screenshot --scale 0.5 ./half.png # DPR-1: screenshot px == click px
chrome-use screenshot --max-width 1400 ./shot.png # cap width for image readers
chrome-use screenshot --annotate # Labeled screenshot + legend
chrome-use screenshot --annotate ./page.png # Save annotated screenshot
chrome-use screenshot --annotate --json # JSON output with annotations
@@ -3243,7 +3295,8 @@ Core Commands:
click <sel|x y> Click element/@ref, or a viewport coordinate
dblclick <sel> Double-click element
type <sel> <text> Type into element
fill <sel> <text> Clear and fill
fill <sel> <text> Clear and fill (handles CodeMirror/Monaco/ProseMirror/
contenteditable; `--file <path>`/`--stdin` for large text)
press <key> [--hold <ms>] Press key (Enter, Tab, Control+a). --hold keeps it
down <ms> then releases precise (in-daemon), for
games/charge: `press d --hold 800`
@@ -3263,8 +3316,13 @@ Core Commands:
scroll <dir> [px] Scroll (up/down/left/right)
scrollintoview <sel> Scroll element into view
wait <sel|ms> Wait for element or time
screenshot [path] Take screenshot
screenshot [path] Take screenshot (auto-downscaled to 2000px long edge;
--max-width/--max-height/--scale to override)
pdf <path> Save as PDF
canvas list List <canvas> elements (size, type) on the page
canvas capture [sel] [path] Save a canvas's rendered pixels to PNG for
WebGL/canvas apps (Figma, games, maps, charts) that
expose no DOM. toDataURL, with a screenshot fallback.
snapshot Accessibility tree with refs (for AI)
eval <js> Run JavaScript
connect <port|url> Connect to browser via CDP
@@ -3277,6 +3335,8 @@ Navigation:
Get Info: chrome-use get <what> [selector]
text, html, value, attr <name>, title, url, count, box, styles, cdp-url
box <sel> x,y,width,height,centerX,centerY,inViewport in CSS px (feed
centerX/centerY into `click x y`); value reads CodeMirror/Monaco too
text (no selector = whole page, all frames), text --main, frames (list)
Check State: chrome-use is <what> <selector>
@@ -3319,6 +3379,10 @@ Tabs:
stable targetId, no reload preserves in-page state
open <url> --reuse-tab Reuse an existing tab on that URL instead of spawning
a duplicate (matches origin+path; preserves state)
adopt <url|targetId> Read a PRE-EXISTING tab (the user's own, or another
session's) WITHOUT opening a new one matches by URL
substring or stable targetId, then drives it. e.g.
`adopt "github.com/owner/repo"`
Diff:
diff snapshot Compare current vs last snapshot
+206 -8
View File
@@ -31,6 +31,10 @@ fn dirs_home() -> Option<PathBuf> {
pub struct Adapter {
pub meta: Value,
pub func_src: String,
/// The adapter's declared `args` keys in DECLARATION order. Parsed from the
/// raw @meta text because `serde_json` sorts object keys alphabetically, which
/// would otherwise scramble positional-arg mapping for multi-arg adapters.
pub arg_order: Vec<String>,
}
impl Adapter {
@@ -111,7 +115,66 @@ pub fn parse_adapter(raw: &str, spec: &str) -> Result<Adapter, String> {
if func_src.is_empty() {
return Err(format!("site: {spec} has no function body after @meta"));
}
Ok(Adapter { meta, func_src })
let arg_order = arg_order_from_meta(&raw[start..end]);
Ok(Adapter {
meta,
func_src,
arg_order,
})
}
/// Extract the `args` object's keys in DECLARATION order from the raw @meta JSON
/// text (serde sorts them, losing order). Brace/string-aware: finds the `"args"`
/// value object and collects only its top-level keys.
fn arg_order_from_meta(meta_json: &str) -> Vec<String> {
let bytes = meta_json.as_bytes();
// Locate the `"args"` key, then the `{` that opens its value object.
let Some(args_pos) = meta_json.find("\"args\"") else {
return Vec::new();
};
let Some(brace_off) = meta_json[args_pos..].find('{') else {
return Vec::new();
};
let open = args_pos + brace_off;
let mut keys = Vec::new();
let mut depth = 0i32;
let mut in_str = false;
let mut esc = false;
let mut cur = String::new();
let mut last_str: Option<String> = None;
for &b in bytes.iter().skip(open) {
if in_str {
if esc {
esc = false;
} else if b == b'\\' {
esc = true;
} else if b == b'"' {
in_str = false;
last_str = Some(std::mem::take(&mut cur));
} else {
cur.push(b as char);
}
continue;
}
match b {
b'"' => in_str = true,
b'{' => depth += 1,
b'}' => {
depth -= 1;
if depth == 0 {
break; // end of the args object
}
}
// A `:` at depth 1 means the preceding string was a key of `args`.
b':' if depth == 1 => {
if let Some(k) = last_str.take() {
keys.push(k);
}
}
_ => {}
}
}
keys
}
/// Build the JS to eval: `(<adapter function>)(<args JSON>)`. The adapter's
@@ -199,22 +262,137 @@ pub async fn update() -> Result<usize, String> {
count += 1;
}
}
// Build the domain→adapters index and stamp the sync time so navigation can
// suggest adapters (auto-trigger) and `needs_refresh` can pace re-syncs.
write_domain_index(&dir);
if let Some(p) = last_update_path() {
let _ = std::fs::write(p, now_secs().to_string());
}
Ok(count)
}
/// `~/.chrome-use/sites/.last_update` — unix-seconds marker of the last sync.
fn last_update_path() -> Option<PathBuf> {
sites_dir().map(|d| d.join(".last_update"))
}
/// `~/.chrome-use/sites/.index.json` — `{ "github.com": ["github/issues", …], … }`,
/// built on `update` so navigation can look up adapters by domain without parsing
/// all ~145 adapter files on every command.
fn index_path() -> Option<PathBuf> {
sites_dir().map(|d| d.join(".index.json"))
}
fn now_secs() -> u64 {
std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.map(|d| d.as_secs())
.unwrap_or(0)
}
/// Parse every installed adapter and write the domain→adapters index. Within a
/// domain, read-only adapters are listed first (then alphabetical) so the
/// auto-suggested example leads with a safe read, not a write action.
fn write_domain_index(dir: &std::path::Path) {
let mut by_domain: std::collections::BTreeMap<String, Vec<(bool, String)>> = Default::default();
for spec in list_adapters().unwrap_or_default() {
if let Ok(a) = load_adapter(&spec) {
if let Some(d) = a.domain() {
let read_only = a
.meta
.get("readOnly")
.and_then(|v| v.as_bool())
.unwrap_or(false);
by_domain
.entry(d.to_string())
.or_default()
.push((read_only, spec));
}
}
}
let ordered: std::collections::BTreeMap<String, Vec<String>> = by_domain
.into_iter()
.map(|(domain, mut v)| {
// read-only (true) first, then by spec name
v.sort_by(|a, b| b.0.cmp(&a.0).then_with(|| a.1.cmp(&b.1)));
(domain, v.into_iter().map(|(_, s)| s).collect())
})
.collect();
if let Ok(json) = serde_json::to_string(&ordered) {
let _ = std::fs::write(dir.join(".index.json"), json);
}
}
const DEFAULT_TTL_DAYS: u64 = 7;
/// Whether the adapter pack should be (re)synced: true on first use (nothing
/// installed) or when the last sync is older than the TTL. Disabled by
/// `AGENT_BROWSER_SITES_NO_AUTO_UPDATE=1`; TTL overridable via
/// `AGENT_BROWSER_SITES_TTL_DAYS` (0 = always).
pub fn needs_refresh() -> bool {
if std::env::var_os("AGENT_BROWSER_SITES_NO_AUTO_UPDATE").is_some() {
return false;
}
let Some(dir) = sites_dir() else {
return false;
};
// First use: no adapters installed yet.
if list_adapters().map(|l| l.is_empty()).unwrap_or(true) {
let _ = &dir;
return true;
}
let ttl_days = std::env::var("AGENT_BROWSER_SITES_TTL_DAYS")
.ok()
.and_then(|s| s.parse::<u64>().ok())
.unwrap_or(DEFAULT_TTL_DAYS);
let ttl = ttl_days.saturating_mul(86_400);
match last_update_path().and_then(|p| std::fs::read_to_string(p).ok()) {
Some(s) => match s.trim().parse::<u64>() {
Ok(ts) => now_secs().saturating_sub(ts) >= ttl,
Err(_) => true,
},
None => true, // no marker → treat as stale
}
}
/// Adapters whose `@meta.domain` matches `host` (exact, or `host` is a subdomain
/// of it) — for auto-suggesting `site` commands when you land on a known site.
/// Reads the prebuilt `.index.json`; empty if the pack isn't synced yet.
pub fn adapters_for_domain(host: &str) -> Vec<String> {
let host = host.trim_start_matches("www.");
let Some(raw) = index_path().and_then(|p| std::fs::read_to_string(p).ok()) else {
return Vec::new();
};
let Ok(idx) = serde_json::from_str::<std::collections::BTreeMap<String, Vec<String>>>(&raw)
else {
return Vec::new();
};
// Preserve the index's per-domain ordering (read-only adapters first); just
// dedup if a host somehow matches multiple domain keys.
let mut out: Vec<String> = Vec::new();
for (domain, specs) in idx {
let d = domain.trim_start_matches("www.");
if host == d || host.ends_with(&format!(".{d}")) {
for s in specs {
if !out.contains(&s) {
out.push(s);
}
}
}
}
out
}
/// Map CLI args to the adapter's `args` object. Positional args fill the adapter's
/// declared `args` keys in order; `--key value` overrides by name. The adapter
/// validates required args itself.
pub fn map_args(adapter: &Adapter, positional: &[String], named: &[(String, String)]) -> Value {
let mut obj = serde_json::Map::new();
let keys: Vec<String> = adapter
.meta
.get("args")
.and_then(|a| a.as_object())
.map(|m| m.keys().cloned().collect())
.unwrap_or_default();
// Positional args fill the adapter's declared args in DECLARATION order
// (`arg_order`), not serde's alphabetized key order — otherwise a 2-arg
// adapter like `{projectId, path}` would map positionals to `{path, projectId}`.
for (i, val) in positional.iter().enumerate() {
if let Some(k) = keys.get(i) {
if let Some(k) = adapter.arg_order.get(i) {
obj.insert(k.clone(), Value::String(val.clone()));
}
}
@@ -264,4 +442,24 @@ async function(args) { return { repo: args.repo }; }"#;
assert!(load_adapter("noslash").is_err());
assert!(load_adapter("../etc/passwd").is_err());
}
// Regression: positional args must follow DECLARATION order, not serde's
// alphabetical key order. With `{projectId, path}` (not alphabetical),
// `<uuid> <file>` must map projectId←uuid, path←file — not swapped.
#[test]
fn positional_args_follow_declaration_order_not_alphabetical() {
let raw = r#"/* @meta
{
"name": "claude-design/get-file",
"domain": "claude.ai",
"args": { "projectId": {"required": true}, "path": {"required": true} }
}
*/
async function(args){ return args; }"#;
let a = parse_adapter(raw, "claude-design/get-file").unwrap();
assert_eq!(a.arg_order, vec!["projectId", "path"]);
let args = map_args(&a, &["the-uuid".into(), "misonote.dc.html".into()], &[]);
assert_eq!(args["projectId"], "the-uuid");
assert_eq!(args["path"], "misonote.dc.html");
}
}
Binary file not shown.
Binary file not shown.
+57 -6
View File
@@ -81,6 +81,35 @@ async function groupTabInto(tabId, name) {
groupIdByName.set(name, gid)
}
// Group-scoped relay isolation hints (issue #40). The relay scopes
// Target.getTargets per agent by tab group; report two things in the synthesized
// targetInfo so it can attribute each tab:
// - abGroup: the tab's Chrome tab-group TITLE (= the owning session name), so
// the relay can re-attribute existing tabs after a restart (createTarget
// tagging won't re-run for already-open tabs).
// - openerTargetId: the targetId of the tab that opened this one, so a pop-up
// (window.open / target=_blank / OAuth result) inherits its opener's group
// and the agent that opened it can follow it — without foreign tabs leaking.
// Best-effort: any failure yields empty strings, which the relay ignores.
async function tabScopeHints(tabId) {
let openerTargetId = ''
let abGroup = ''
try {
const t = await chrome.tabs.get(tabId)
if (t) {
if (typeof t.openerTabId === 'number') {
const op = tabs.get(t.openerTabId)
if (op) openerTargetId = op.targetId
}
if (t.groupId != null && t.groupId >= 0 && chrome.tabGroups) {
const g = await chrome.tabGroups.get(t.groupId).catch(() => null)
if (g && g.title) abGroup = g.title
}
}
} catch {}
return { openerTargetId, abGroup }
}
function postToHost(msg) {
try {
if (port) port.postMessage(msg)
@@ -126,7 +155,7 @@ function connectHost() {
} catch {}
// Tell the daemon about everything we already have attached, then attach
// anything new.
reannounceAttachedTabs()
void reannounceAttachedTabs()
void attachAllTabs()
}
@@ -140,7 +169,7 @@ async function onHostMessage(msg) {
// Daemon (re)connected — (re)attach and announce every tab so it discovers
// the user's existing tabs rather than racing an empty target list.
if (msg.method === 'attachAll') {
reannounceAttachedTabs()
void reannounceAttachedTabs()
await attachAllTabs()
return
}
@@ -387,12 +416,16 @@ async function attachTab(tabId) {
sessionToTab.set(sessionId, tabId)
rememberSessionTarget(sessionId, targetId)
setBadge(tabId, port ? 'on' : 'connecting')
const { openerTargetId, abGroup } = await tabScopeHints(tabId)
postToHost({
method: 'forwardCDPEvent',
params: {
sessionId,
method: 'Target.attachedToTarget',
params: { sessionId, targetInfo: { ...targetInfo, attached: true } },
params: {
sessionId,
targetInfo: { ...targetInfo, attached: true, openerTargetId, abGroup },
},
},
})
return entry
@@ -434,14 +467,32 @@ async function attachAllTabs() {
}
}
function reannounceAttachedTabs() {
for (const [, entry] of tabs.entries()) {
async function reannounceAttachedTabs() {
for (const [tabId, entry] of tabs.entries()) {
// Re-send the group hint too (issue #40) so the relay can rebuild its
// targetId→group map after its own restart (createTarget tagging won't
// re-run for tabs that are already open). Include the live url/title so the
// relay's target list stays matchable by URL after a reconnect (otherwise a
// reannounced tab shows a blank url and `adopt <url>` can't find it).
const { openerTargetId, abGroup } = await tabScopeHints(tabId)
let url = ''
let title = ''
try {
const t = await chrome.tabs.get(tabId)
if (t) {
url = t.url || t.pendingUrl || ''
title = t.title || ''
}
} catch {}
postToHost({
method: 'forwardCDPEvent',
params: {
sessionId: entry.sessionId,
method: 'Target.attachedToTarget',
params: { sessionId: entry.sessionId, targetInfo: { targetId: entry.targetId, type: 'page', attached: true } },
params: {
sessionId: entry.sessionId,
targetInfo: { targetId: entry.targetId, type: 'page', url, title, attached: true, openerTargetId, abGroup },
},
},
})
}
+1 -1
View File
@@ -1,7 +1,7 @@
{
"manifest_version": 3,
"name": "chrome-use",
"version": "0.4.9",
"version": "0.4.11",
"description": "Let chrome-use drive your logged-in Chrome \u2014 install once, no token, no per-use confirmation.",
"key": "MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA6vQIyscGIPYPZdSpPwPL0+0gxUROyRgCpmvCSDoc8XUm4qm97VbKnD9Ijc1lV22lNWZtE78gaRjt6BeSfuMgnBymnhLKjN1gU6AI5QUU0mrJyeHdWKvrKQR5FmsM2A7Xr1ykE2SiiS8zNUS3Y/6O5l+Nva7wrVy6E4a2dkBVQkOsu+DV+nEZvhIyuDY5D5SPXqNwUTWTaglwj5mjvHz36xSwCWlPmrtJ+ED0AUyrb2z4GIOmvk4kqtBVrh/UD058klLo4CkYOnIybB5aV6WYuwarfPY4bF/dLggPem+ewLNTUNBuwrxj/A4nUv0LJTuRO8rR7f8WR9qnRCY0Ic5saQIDAQAB",
"icons": {
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "chrome-use",
"version": "1.5.19",
"version": "1.5.24",
"description": "chrome-use — drive your real, logged-in Chrome from any AI agent, stealth by default",
"type": "module",
"packageManager": "pnpm@11.1.3",
+43 -1
View File
@@ -59,6 +59,18 @@ next ref interaction.
> anyway.) Driving off pixels on the relay also risks a coordinate event drifting
> onto the user's foreground tab — refs never do. See issue #37.
> **Two different intents — only one is discouraged.** The rule above is about
> *screenshot-to-locate* (using a picture to find/hit an element) — that's the bug.
> *screenshot-to-capture* — saving a region or element to a file as a **reusable
> image asset** (maps, charts, og-images, visual-diff baselines, report figures) —
> is fully supported and encouraged: `screenshot [selector] [--clip x,y,w,h] <file>`.
> Capturing a rendered map region to a PNG for a blog post is the right tool, not a
> smell. Screenshots are auto-downscaled to ≤2000px (longest edge) so they fit an
> image reader and their pixels line up with `click x y`; override with
> `--max-width`/`--max-height`/`--scale`. To click something you couldn't hit by
> ref, `box @ref` gives the element's CSS-px box + `centerX/centerY` to feed
> straight into `click <centerX> <centerY>` — no screenshot needed.
## Before you automate: pick the cheapest tool
Driving a browser is the heavy option. chrome-use earns its keep when you
@@ -142,7 +154,17 @@ real Chrome concurrently without ever dropping or stealing each other's tabs —
another agent's tab churn can't make your bound tab vanish or drift your commands
onto the wrong page. Consequence: `tab list` shows only *your* session's tabs; to
drive a specific page, navigate to it in your own tab instead of expecting a
pre-existing or popped-up tab to appear in the list. **Anti-detection ranking: this real logged-in Chrome (extension
pre-existing or popped-up tab to appear in the list.
> **Need to read a tab the user already has open?** Use `chrome-use adopt
> <url-substring|targetId>` — it finds that pre-existing tab (the user's own, or
> another session's) across groups and drives it **without opening a new tab**.
> e.g. `adopt "claude.ai/design"` then `snapshot`/`eval`/`get text` on it. On no
> match it errors and lists the tabs it can see. This is the explicit, opt-in way
> through the isolation above (it tags the adopted tab into your group). Great for
> "read/extract from the page I'm looking at" without disturbing it.
**Anti-detection ranking: this real logged-in Chrome (extension
connect) > a headed launched browser > headless (forbidden).** A genuine human
browser has no headless/automation tells at all, so prefer it for anything
anti-bot-sensitive.
@@ -218,6 +240,13 @@ chrome-use site github/issues owner/repo --json # run it → JSON (navigates t
- If no adapter fits, fall back to the normal `snapshot`/`eval` loop. Adapters come from the
[bb-sites](https://github.com/epiral/bb-sites) community pack; chrome-use fetches & runs them.
> **Auto-trigger — act on it.** chrome-use keeps the pack synced automatically (first use +
> weekly), and when you `open`/`navigate`/`snapshot` a page whose domain has adapters it tells
> you: a `💡 site adapters for <domain>` line on stderr, and a `siteAdapters: {domain, commands}`
> field in `--json`. **When you see that, prefer the listed `site <name>/<cmd>` over snapshot+click
> for reading data** — it's the cheaper, more reliable path and it's already installed. You don't
> need to run `site update` yourself; just use the command it names.
## Quickstart
```bash
@@ -366,6 +395,12 @@ foreground, so prefer refs. For below-the-fold content in such a frame, scroll i
with `scroll down N --at x,y` (a pixel over the frame) or `--frame n`. For a
postal/autocomplete box inside the frame, `type @e "…" --key-events`.
> **Caveat: `find text "…"` can't reach into a cross-origin iframe** — it errors
> "Element not found" even though `snapshot -i` lists those nodes and
> `get text` reads them. Inside cross-origin iframes, target elements by their
> **snapshot `@ref`**, not by `find`. (`box @ref` also works on iframe refs when
> you need a coordinate fallback.)
### When refs don't work or you don't want to snapshot
Use semantic locators:
@@ -419,6 +454,13 @@ tree**, so `snapshot` comes back near-empty and refs are a dead end. `snapshot`
detects this and prints a one-line hint. Drive them the screenshot way:
```bash
chrome-use canvas list # enumerate <canvas> elements (size, type)
chrome-use canvas capture out.png # save the canvas's RENDERED pixels to PNG —
# toDataURL (full backing-store res, e.g.
# Figma 2522x1904), screenshot fallback for
# WebGL w/o preserveDrawingBuffer / tainted.
# Gets the RENDER, not hidden source data
# (those live in the app's binary store/API).
chrome-use screenshot /tmp/s.png # SEE the state (your only read path —
# eval/get text return nothing useful)
chrome-use click 640 360 # interact by viewport coordinate