Compare commits

...
5 Commits
Author SHA1 Message Date
leeguooooo 284a60a54c feat(adopt): read a pre-existing tab without opening a new one
Release binaries / Build macOS ARM64 (push) Has been cancelled
Release binaries / Build macOS x64 (push) Has been cancelled
Release binaries / Build Linux ARM64 (push) Has been cancelled
Release binaries / Build Linux musl ARM64 (push) Has been cancelled
Release binaries / Build Linux musl x64 (push) Has been cancelled
Release binaries / Build Linux x64 (push) Has been cancelled
Release binaries / Build Windows x64 (push) Has been cancelled
Release binaries / Attach binaries to GitHub Release (push) Has been cancelled
New `chrome-use adopt <url-substring|targetId>`: drive a tab the user (or
another session) already has open, with ZERO new tabs. After group-scoped
isolation (#40) a session can't see foreign tabs, so adopt adds an explicit,
opt-in path:

- Relay (relay.rs): `ABRelay.getAllTargets` returns every attached target
  UNSCOPED (ignores group scoping), so the agent can find a specific tab by URL
  or targetId. +1 unit test.
- Daemon (browser.rs): `collect_all_targets` (unscoped, falls back to scoped on
  older relays) + `adopt_existing_target` — matches by exact targetId or
  case-insensitive URL substring, attaches it (the relay re-tags it into the
  adopter's group, so isolation holds), pins it; never creates a tab. On no
  match it errors AND lists the open tabs it can see, rather than launching.
  discover_and_attach_targets honors AGENT_BROWSER_ADOPT at first connect, so no
  about:blank is ever created.
- CLI (main.rs): `adopt` sets the env, forces a fresh daemon, and rewrites into
  `connect <relay-url>` (like `extension connect`) so the daemon attaches to the
  user's real Chrome before parse_command.

Extension (ab-connect 0.4.11): `reannounceAttachedTabs` now re-sends each tab's
url/title (it previously sent neither) so the relay's target list stays matchable
by URL after the MV3 service worker reconnects — otherwise reannounced tabs show
a blank url and `adopt <url>` can't find them. Repacked upload zip + crx.

Mechanism verified live (enumerated all 11 of the user's open tabs incl. the
target). 862 tests pass.
2026-06-17 21:18:01 +09:00
leeguooooo 10d196b6eb chore(ext): pack ab-connect 0.4.10 upload zip + crx (#40 group-scoped relay)
Rebuilt extensions/ab-connect.zip (key stripped for the Web Store) and the
reference .crx from the 0.4.10 source (openerTargetId + abGroup in the
synthesized Target.attachedToTarget).
2026-06-17 18:17:20 +09:00
leeguooooo 5be01e292d feat(relay): group-scoped Target.getTargets — restore follow-popup + cross-session adopt under isolation (#40)
Release binaries / Build macOS ARM64 (push) Has been cancelled
Release binaries / Build macOS x64 (push) Has been cancelled
Release binaries / Build Linux ARM64 (push) Has been cancelled
Release binaries / Build Linux musl ARM64 (push) Has been cancelled
Release binaries / Build Linux musl x64 (push) Has been cancelled
Release binaries / Build Linux x64 (push) Has been cancelled
Release binaries / Build Windows x64 (push) Has been cancelled
Release binaries / Attach binaries to GitHub Release (push) Has been cancelled
Move multi-agent isolation from blunt daemon-side filtering to relay-side
group scoping, so a session can adopt new tabs again (follow-popup, OAuth
results, cross-session adopt-by-targetId) without ever seeing the user's or
another agent's tabs.

Relay (relay.rs): track client->group (announced via new local ABRelay.setGroup,
or the first createTarget's agentGroup) and target->group (created tabs tagged
from the createTarget reply; an explicit attachToTarget tags the target into the
adopter's group = #21; a pop-up inherits its opener's group via openerTargetId).
Target.getTargets returns ONLY the requesting client's group; a client that never
announced a group (older daemon) gets the full list — fully backward-compatible.
+5 unit tests.

Daemon (browser.rs): announce_group() on connect sets relay_scoped. Adoption in
discover/resync/adopt_newly_opened is re-enabled ONLY when relay_scoped; without
it (launch / real CDP / older relay that didn't answer the announce) the daemon
keeps strict daemon-side isolation. So this can't regress the 125/125 isolation.

Extension (ab-connect 0.4.10): synthesized Target.attachedToTarget targetInfo now
carries openerTargetId (pop-ups inherit opener's group) and abGroup (the tab-group
title, so the relay re-attributes existing tabs after ITS own restart, since
createTarget tagging won't re-run). tabScopeHints().

Back-compat verified live: new daemon + OLD relay -> announce fails ->
relay_scoped=false -> strict fallback, open/eval/url all work. The new extension
(publish to CWS, strip manifest key) activates follow-popup; relay+daemon ship now.
861 tests pass.
2026-06-17 18:14:13 +09:00
leeguooooo a83d1b1df9 feat: rich-editor fill, box centers, screenshot downscale, disabled+docs (#41-#45)
Release binaries / Build macOS ARM64 (push) Has been cancelled
Release binaries / Build macOS x64 (push) Has been cancelled
Release binaries / Build Linux ARM64 (push) Has been cancelled
Release binaries / Build Linux musl ARM64 (push) Has been cancelled
Release binaries / Build Linux musl x64 (push) Has been cancelled
Release binaries / Build Linux x64 (push) Has been cancelled
Release binaries / Build Windows x64 (push) Has been cancelled
Release binaries / Attach binaries to GitHub Release (push) Has been cancelled
Dogfooding backlog from this session's embedded-form/editor work.

#41 fill on rich editors: detect CodeMirror 5 / Monaco / ProseMirror /
contenteditable and set via their own API or execCommand('insertText') so
beforeinput/input fire (a raw .value/textContent write no-op'd juejin's
CodeMirror and skipped React composers). Response echoes the `engine` used.
`fill <sel> --file <path>` / `--stdin` set large multiline text without
shell-escaping. `get value` now reads CodeMirror/Monaco/contenteditable too.

#42 screenshot --max-width/--max-height/--scale, plus a default 2000px
longest-edge cap (AGENT_BROWSER_SCREENSHOT_MAX_EDGE; 0 disables) so retina
full-page shots fit an agent's image reader and --scale 0.5 makes screenshot
px line up with click px. Annotated shots are never downscaled.

#43 `box @ref` (already a top-level alias of `get box`) now also returns
centerX/centerY/inViewport in CSS px — feed straight into `click x y` when a
ref-click no-ops (e.g. a button in a cross-origin iframe).

#44 no code change needed — disabled elements already list as
`button "Save" [disabled, ref=eN]`; the reporter's missing button was
DOM-gated on validity. Added a skill note: `find text` can't reach into a
cross-origin iframe — target those by snapshot @ref.

#45 core skill now distinguishes screenshot-to-locate (discouraged) from
screenshot-to-capture a reusable image asset via `screenshot [--clip] <file>`
(encouraged), so agents stop over-reading the prohibition.

#40 (group-scoped relay) stays deferred — needs an ab-connect extension change.

Verified live: fill --file round-trips multiline+CJK+backticks; contenteditable
engine=contenteditable + get value reads it back; box gives centerX/centerY/
inViewport; screenshot of retina example.com → 2000px; disabled button shows
[disabled]. 856 tests pass.
2026-06-17 17:49:38 +09:00
leeguooooo 50b27ac0e0 feat(site): auto-sync + auto-suggest adapters (auto-trigger)
Release binaries / Build macOS ARM64 (push) Has been cancelled
Release binaries / Build macOS x64 (push) Has been cancelled
Release binaries / Build Linux ARM64 (push) Has been cancelled
Release binaries / Build Linux musl ARM64 (push) Has been cancelled
Release binaries / Build Linux musl x64 (push) Has been cancelled
Release binaries / Build Linux x64 (push) Has been cancelled
Release binaries / Build Windows x64 (push) Has been cancelled
Release binaries / Attach binaries to GitHub Release (push) Has been cancelled
Make `site` trigger itself so an agent doesn't have to know adapters exist.

Auto-sync: the pack refreshes on first use and on a TTL (default 7d), both in
the `site` command path (blocking, fast) and as a non-blocking background task
on daemon startup — so ~/.chrome-use/sites/.index.json is always populated with
zero added latency. Tune via AGENT_BROWSER_SITES_TTL_DAYS; disable with
AGENT_BROWSER_SITES_NO_AUTO_UPDATE=1. `update` now writes .last_update + a
domain→adapters .index.json (read-only adapters ordered first).

Auto-suggest: `open`/`navigate`/`snapshot` onto a domain with adapters attaches
`siteAdapters: {domain, commands}` to the response; the CLI prints a
`💡 site adapters for <domain>` hint (stderr) and the field rides along in --json.
SKILL.md tells the agent to prefer the listed `site <name>/<cmd>` over scraping.
This keeps the 'never auto-disrupt user tabs' guarantee — it suggests, the agent
decides; nothing auto-runs on navigation.

site.rs: needs_refresh/adapters_for_domain/write_domain_index + timestamp/index
in update(). daemon.rs: background bootstrap. actions.rs: with_site_hint on
navigate + snapshot. output.rs: hint render. Verified live: open github.com →
hint leads with read-only github/issues; --json carries siteAdapters.
2026-06-17 17:15:34 +09:00
21 changed files with 1107 additions and 65 deletions
+16
View File
@@ -233,6 +233,22 @@ Positional args fill the adapter's declared args in order; `--key value` overrid
by name. Adapters are authored by the bb-sites community and remain their authors'
property — chrome-use just runs them.
**Auto-sync + auto-suggest.** You rarely type `site update` yourself: chrome-use
syncs the pack on first use and refreshes it weekly in the background (tune with
`AGENT_BROWSER_SITES_TTL_DAYS`, disable with `AGENT_BROWSER_SITES_NO_AUTO_UPDATE=1`).
And when you `open`/`snapshot` a page whose domain has adapters, chrome-use surfaces
them right in the output — a `💡 site adapters for <domain>` line, plus a
`siteAdapters` field under `--json` — so an agent reaches for the structured-data
adapter instead of scraping the DOM:
```text
$ chrome-use open https://github.com
💡 site adapters for github.com — prefer these for structured data:
github/issues, github/me, github/repo, …
e.g. chrome-use site github/issues --json
✓ GitHub
```
## Automated testing (`chrome-use test`)
Turn the repetitive "open it, click around, check it's right" work into a
+14
View File
@@ -188,6 +188,20 @@ chrome-use site bilibili/feed --json # 能用,因为走的是你的
位置参数按适配器声明的参数顺序填入;`--key value` 按名覆盖。适配器由 bb-sites 社区编写、
版权归各自作者所有 —— chrome-use 只负责运行它们。
**自动同步 + 自动提示。** 你基本不用手动 `site update`:chrome-use 首次使用时自动拉取,
之后每周后台刷新一次(`AGENT_BROWSER_SITES_TTL_DAYS` 调周期,`AGENT_BROWSER_SITES_NO_AUTO_UPDATE=1`
关闭)。而当你 `open`/`snapshot` 一个有适配器的域名时,chrome-use 会在输出里直接把可用命令
亮出来 —— 一行 `💡 site adapters for <域名>`,`--json` 下则是 `siteAdapters` 字段 —— 这样
agent 会直接改用结构化适配器,而不是去扒 DOM:
```text
$ chrome-use open https://github.com
💡 site adapters for github.com — prefer these for structured data:
github/issues, github/me, github/repo, …
e.g. chrome-use site github/issues --json
✓ GitHub
```
## 自动化测试(`chrome-use test`
把反复的「打开它、点一圈、看对不对」变成**可重跑的测试套件** —— 前端的单元测试。用 YAML 写用例;步骤复用 chrome-use 自己的命令,断言编译成一次检查:
+1 -1
View File
@@ -290,7 +290,7 @@ checksum = "613afe47fcd5fac7ccf1db93babcb082c5994d996f20b8b159f2ad1658eb5724"
[[package]]
name = "chrome-use"
version = "1.5.19"
version = "1.5.23"
dependencies = [
"aes",
"aes-gcm",
+1 -1
View File
@@ -1,6 +1,6 @@
[package]
name = "chrome-use"
version = "1.5.19"
version = "1.5.23"
edition = "2021"
description = "Fast browser automation CLI for AI agents"
license = "Apache-2.0"
+91 -2
View File
@@ -79,6 +79,8 @@ const KNOWN_COMMANDS: &[&str] = &[
"dialog",
"upload",
"site",
"box",
"adopt",
];
/// Levenshtein distance, capped — small inputs only (command names).
@@ -560,9 +562,37 @@ fn parse_command_inner(args: &[String], flags: &Flags) -> Result<Value, ParseErr
"fill" => {
let sel = rest.first().ok_or_else(|| ParseError::MissingArguments {
context: "fill".to_string(),
usage: "fill <selector> <text>",
usage: "fill <selector> <text> | fill <selector> --file <path> | fill <selector> --stdin",
})?;
Ok(json!({ "id": id, "action": "fill", "selector": sel, "value": rest[1..].join(" ") }))
// Large/multiline content without shell-escaping hell (issue #41):
// `fill <sel> --file <path>` reads the value from a UTF-8 file, and
// `fill <sel> --stdin` reads it from stdin — sent verbatim, so backticks,
// quotes, newlines and non-ASCII pass through untouched.
let value = match rest.get(1).copied() {
Some("--file") => {
let path = rest.get(2).ok_or(ParseError::InvalidValue {
message: "fill --file requires a path".to_string(),
usage: "fill <selector> --file <path>",
})?;
std::fs::read_to_string(path).map_err(|e| ParseError::InvalidValue {
message: format!("fill --file: cannot read {path}: {e}"),
usage: "fill <selector> --file <path>",
})?
}
Some("--stdin") => {
use std::io::Read;
let mut buf = String::new();
io::stdin()
.read_to_string(&mut buf)
.map_err(|e| ParseError::InvalidValue {
message: format!("fill --stdin: {e}"),
usage: "fill <selector> --stdin",
})?;
buf
}
_ => rest[1..].join(" "),
};
Ok(json!({ "id": id, "action": "fill", "selector": sel, "value": value }))
}
"type" => {
// `--key-events` (alias `--keys`): send real per-character keystrokes
@@ -1006,11 +1036,55 @@ fn parse_command_inner(args: &[String], flags: &Flags) -> Result<Value, ParseErr
// path: file path (contains / or . or ends with known extension)
let mut full_page = false;
let mut clip: Option<Value> = None;
let mut max_width: Option<u32> = None;
let mut max_height: Option<u32> = None;
let mut scale: Option<f64> = None;
let mut positional: Vec<&str> = Vec::new();
let mut i = 0;
// Parse a numeric value for a downscale flag (issue #42).
let parse_num = |i: &mut usize, flag: &str| -> Result<String, ParseError> {
let v = rest
.get(*i + 1)
.ok_or_else(|| ParseError::MissingArguments {
context: format!("screenshot {flag}"),
usage: "screenshot [--max-width <px>] [--max-height <px>] [--scale <0..1>]",
})?;
*i += 1;
Ok(v.to_string())
};
while i < rest.len() {
match rest[i] {
"--full" | "-f" => full_page = true,
// Downscale the saved image so retina/full-page shots fit an
// agent's image reader and screenshot px line up with click px (#42).
"--max-width" => {
let v = parse_num(&mut i, "--max-width")?;
max_width = Some(v.parse().map_err(|_| ParseError::InvalidValue {
message: format!("--max-width expects a number, got '{v}'"),
usage: "screenshot --max-width <px>",
})?);
}
"--max-height" => {
let v = parse_num(&mut i, "--max-height")?;
max_height = Some(v.parse().map_err(|_| ParseError::InvalidValue {
message: format!("--max-height expects a number, got '{v}'"),
usage: "screenshot --max-height <px>",
})?);
}
"--scale" => {
let v = parse_num(&mut i, "--scale")?;
let s: f64 = v.parse().map_err(|_| ParseError::InvalidValue {
message: format!("--scale expects a number like 0.5, got '{v}'"),
usage: "screenshot --scale <0..1>",
})?;
if s <= 0.0 || s > 1.0 {
return Err(ParseError::InvalidValue {
message: format!("--scale must be in (0, 1], got '{v}'"),
usage: "screenshot --scale <0..1>",
});
}
scale = Some(s);
}
// `--clip x,y,w,h` captures a pixel region (issue #34).
"--clip" => {
let raw = rest
@@ -1073,6 +1147,15 @@ fn parse_command_inner(args: &[String], flags: &Flags) -> Result<Value, ParseErr
if let Some(c) = clip {
cmd["clip"] = c;
}
if let Some(w) = max_width {
cmd["maxWidth"] = json!(w);
}
if let Some(h) = max_height {
cmd["maxHeight"] = json!(h);
}
if let Some(s) = scale {
cmd["scale"] = json!(s);
}
if let Some(ref fmt) = flags.screenshot_format {
cmd["format"] = json!(fmt);
}
@@ -1235,6 +1318,12 @@ fn parse_command_inner(args: &[String], flags: &Flags) -> Result<Value, ParseErr
Ok(json!({ "id": id, "action": "site", "domain": domain, "script": script }))
}
// `adopt <url|targetId>`: the adoption happens at daemon connect (driven by
// the AGENT_BROWSER_ADOPT env main.rs set + a forced-fresh daemon), so by
// the time this command runs the tab is already attached. Resolve to a
// `url` read so the response confirms which tab got adopted.
"adopt" => Ok(json!({ "id": id, "action": "url" })),
// === Stealth self-check ===
"stealth" => {
// `stealth [status]` — local stealth self-check: mode, live probes
+1 -1
View File
@@ -595,7 +595,7 @@ fn query_current_url(session: &str) -> Option<String> {
}
/// Kill a running daemon by reading its PID file and sending a kill signal.
fn kill_stale_daemon(session: &str) {
pub fn kill_stale_daemon(session: &str) {
// Remove the socket first so no new connections reach the old daemon
#[cfg(unix)]
{
+58
View File
@@ -840,6 +840,27 @@ fn main() {
// `info` are CLI-side (download/filesystem); `site <name>/<cmd> [args]` falls
// through to the daemon dispatch below (navigate to the adapter's domain + eval).
if clean.first().map(|s| s.as_str()) == Some("site") {
// Auto-sync the adapter pack on first use and periodically (TTL, default
// 7d) so adapters stay fresh without a manual `site update`. Skipped for an
// explicit `update` (full sync below). Best-effort: offline → cached pack.
// Disable with AGENT_BROWSER_SITES_NO_AUTO_UPDATE=1.
if clean.get(1).map(|s| s.as_str()) != Some("update") && site::needs_refresh() {
let rt = tokio::runtime::Runtime::new().expect("Failed to create tokio runtime");
match rt.block_on(site::update()) {
Ok(n) => {
eprintln!(
"{}",
color::dim(&format!("site: synced {n} adapters (auto)"))
)
}
Err(e) => eprintln!(
"{}",
color::dim(&format!(
"site: auto-sync skipped ({e}); using cached adapters"
))
),
}
}
match clean.get(1).map(|s| s.as_str()) {
Some("update") => {
let rt = tokio::runtime::Runtime::new().expect("Failed to create tokio runtime");
@@ -960,6 +981,43 @@ fn main() {
}
}
// `adopt <url|targetId>`: read a PRE-EXISTING tab (the user's own, or another
// session's) WITHOUT opening a new one. Forces a fresh daemon and points it at
// the relay (like `extension connect`); the AGENT_BROWSER_ADOPT env makes the
// daemon's first connect ADOPT the matching tab instead of creating an
// about:blank. Rewrites into `connect <relay-url>` BEFORE parse_command so the
// daemon attaches to the user's real Chrome. Must run before parse_command.
if clean.first().map(|s| s.as_str()) == Some("adopt") {
match clean.get(1) {
Some(spec) if !spec.trim().is_empty() => {
std::env::set_var("AGENT_BROWSER_ADOPT", spec.trim());
connection::kill_stale_daemon(&flags.session);
match connect::relay_url() {
Some(url) => {
flags.cdp = Some(url.clone());
flags.auto_connect = false;
clean = vec!["connect".to_string(), url];
}
None => {
eprintln!(
"{} extension relay not connected — open Chrome with the ab-connect \
extension first (this command reads an EXISTING tab, it won't launch one).",
color::error_indicator()
);
exit(1);
}
}
}
_ => {
eprintln!(
"{} usage: chrome-use adopt <url-substring|targetId> (reads an existing tab, no new tab)",
color::error_indicator()
);
exit(2);
}
}
}
// Handle session separately (doesn't need daemon)
if clean.first().map(|s| s.as_str()) == Some("session") {
run_session(&clean, &flags.session, flags.json);
+127 -5
View File
@@ -2478,7 +2478,10 @@ async fn handle_navigate(cmd: &Value, state: &mut DaemonState) -> Result<Value,
wb.navigate(url).await?;
let new_url = wb.get_url().await.unwrap_or_else(|_| url.to_string());
let title = wb.get_title().await.unwrap_or_default();
return Ok(json!({ "url": new_url, "title": title }));
return Ok(with_site_hint(
json!({ "url": new_url, "title": title }),
url,
));
}
}
@@ -2545,7 +2548,7 @@ async fn handle_navigate(cmd: &Value, state: &mut DaemonState) -> Result<Value,
.unwrap_or(false)
{
if let Ok(Some(switched)) = mgr.reuse_tab_for_url(url).await {
return Ok(switched);
return Ok(with_site_hint(switched, url));
}
}
@@ -2553,7 +2556,34 @@ async fn handle_navigate(cmd: &Value, state: &mut DaemonState) -> Result<Value,
// Adaptive humanize: sample the freshly loaded page for known behavioural
// anti-bot vendors and escalate this session to Human if any are present.
detect_and_set_humanize(mgr).await;
Ok(result)
Ok(with_site_hint(result, url))
}
/// Annotate a navigation/snapshot result with the `site` adapters available for
/// the page's domain (auto-trigger): when you land on e.g. github.com, the
/// response carries `siteAdapters: { domain, commands: ["github/issues", …] }` so
/// the agent reaches for a structured-data adapter instead of scraping. No-op
/// when nothing matches or the pack isn't synced yet.
fn with_site_hint(mut result: Value, fallback_url: &str) -> Value {
let url = result
.get("url")
.and_then(|v| v.as_str())
.unwrap_or(fallback_url);
let host = url::Url::parse(url)
.ok()
.and_then(|u| u.host_str().map(String::from));
if let Some(host) = host {
let adapters = crate::site::adapters_for_domain(&host);
if !adapters.is_empty() {
if let Some(obj) = result.as_object_mut() {
obj.insert(
"siteAdapters".to_string(),
json!({ "domain": host, "commands": adapters }),
);
}
}
}
result
}
/// After navigation, probe the page for known anti-bot vendor fingerprints
@@ -2936,6 +2966,13 @@ async fn handle_snapshot(cmd: &Value, state: &mut DaemonState) -> Result<Value,
let ref_count = refs.len();
let mut out = json!({ "snapshot": tree, "origin": url, "refs": refs });
// Auto-trigger: if this domain has site adapters, surface them so the agent
// pulls structured data instead of walking the tree. `with_site_hint` reads
// the `url` field, so pass it under that key.
if let Some(hint) = with_site_hint(json!({ "url": url }), &url).get("siteAdapters") {
out["siteAdapters"] = hint.clone();
}
// Canvas/WebGL apps (games, map/3D viewers, drawing tools) paint to a
// <canvas> and expose almost no accessibility tree, so `snapshot` comes back
// near-empty and agents get stuck looking for refs that will never exist
@@ -3077,7 +3114,40 @@ async fn handle_screenshot(cmd: &Value, state: &mut DaemonState) -> Result<Value
)
.await?;
// Downscale the saved image so retina/full-page shots fit an agent's image
// reader and screenshot pixels line up with `click x y` CSS px (issue #42).
// Explicit --scale / --max-width / --max-height win; otherwise a default cap
// (2000px longest edge, AGENT_BROWSER_SCREENSHOT_MAX_EDGE overrides, 0 = off)
// applies. Annotated shots are left untouched so ref overlays stay aligned.
let mut resized: Option<(u32, u32)> = None;
if !annotate {
let scale = cmd.get("scale").and_then(|v| v.as_f64());
let max_w = cmd
.get("maxWidth")
.and_then(|v| v.as_u64())
.map(|v| v as u32);
let max_h = cmd
.get("maxHeight")
.and_then(|v| v.as_u64())
.map(|v| v as u32);
let default_edge = if scale.is_none() && max_w.is_none() && max_h.is_none() {
std::env::var("AGENT_BROWSER_SCREENSHOT_MAX_EDGE")
.ok()
.and_then(|s| s.parse::<u32>().ok())
.or(Some(2000))
.filter(|&e| e > 0)
} else {
None
};
resized = downscale_screenshot(&result.path, scale, max_w, max_h, default_edge);
}
let mut response = json!({ "path": absolutize_saved_path(&result.path) });
if let Some((w, h)) = resized {
response["width"] = json!(w);
response["height"] = json!(h);
response["resized"] = json!(true);
}
if !result.annotations.is_empty() {
response["annotations"] = serde_json::to_value(&result.annotations)
.map_err(|e| format!("Failed to serialize annotations: {}", e))?;
@@ -3093,6 +3163,56 @@ async fn handle_screenshot(cmd: &Value, state: &mut DaemonState) -> Result<Value
Ok(response)
}
/// Downscale a saved screenshot in place (issue #42). Resolves the target longest
/// edge from `scale` (fraction of current), explicit `max_w`/`max_h` caps, or a
/// `default_edge` cap — whichever yields the smaller image. Only ever shrinks;
/// no-op (returns None) if the image is already within bounds or can't be read.
/// Returns the new (width, height) when it actually resized.
fn downscale_screenshot(
path: &str,
scale: Option<f64>,
max_w: Option<u32>,
max_h: Option<u32>,
default_edge: Option<u32>,
) -> Option<(u32, u32)> {
let img = image::open(path).ok()?;
let (w, h) = (img.width(), img.height());
if w == 0 || h == 0 {
return None;
}
// Collect candidate scale factors (≤ 1.0); the smallest wins.
let mut factor = 1.0f64;
if let Some(s) = scale {
factor = factor.min(s);
}
if let Some(mw) = max_w {
if w > mw {
factor = factor.min(mw as f64 / w as f64);
}
}
if let Some(mh) = max_h {
if h > mh {
factor = factor.min(mh as f64 / h as f64);
}
}
if let Some(edge) = default_edge {
let longest = w.max(h);
if longest > edge {
factor = factor.min(edge as f64 / longest as f64);
}
}
if factor >= 1.0 {
return None; // already within bounds — never upscale
}
let nw = ((w as f64 * factor).round() as u32).max(1);
let nh = ((h as f64 * factor).round() as u32).max(1);
let resized = img.resize(nw, nh, image::imageops::FilterType::Lanczos3);
resized.save(path).ok()?;
Some((resized.width(), resized.height()))
}
async fn handle_click(cmd: &Value, state: &mut DaemonState) -> Result<Value, String> {
// First-class coordinate click (issue #8.4): click a raw viewport point with
// no element resolution. Parsed from `click <x> <y>` / `click --coords x,y`.
@@ -3251,7 +3371,7 @@ async fn handle_fill(cmd: &Value, state: &mut DaemonState) -> Result<Value, Stri
let mgr = state.browser.as_ref().ok_or("Browser not launched")?;
let session_id = mgr.active_session_id()?.to_string();
interaction::fill(
let engine = interaction::fill(
&mgr.client,
&session_id,
&state.ref_map,
@@ -3260,7 +3380,9 @@ async fn handle_fill(cmd: &Value, state: &mut DaemonState) -> Result<Value, Stri
&state.iframe_sessions,
)
.await?;
Ok(json!({ "filled": selector }))
// Echo the input path used (input/contenteditable/codemirror5/monaco/select)
// so the agent can confirm a rich editor was handled, not silently no-op'd (#41).
Ok(json!({ "filled": selector, "engine": engine }))
}
async fn handle_type(cmd: &Value, state: &mut DaemonState) -> Result<Value, String> {
+166 -17
View File
@@ -509,6 +509,13 @@ pub struct BrowserManager {
/// for `RELAY_PRUNE_MISSES` consecutive snapshots; any snapshot that includes
/// it resets the counter. Keyed by stable target_id.
relay_target_misses: HashMap<String, u32>,
/// Whether the relay accepted this session's group announcement and is
/// therefore scoping `Target.getTargets` to our own tab group (issue #40).
/// When true the daemon can safely adopt new targets again (follow-popup,
/// cross-session adopt) — the relay has already filtered out foreign tabs.
/// When false (launch-on-real-CDP, or an older relay that didn't answer the
/// announce) the daemon keeps strict daemon-side isolation.
relay_scoped: bool,
next_tab_id: u32,
/// Whether to enable the CDP `Runtime` domain (console / error / exception capture).
/// OFF by default for stealth: a live `Runtime.enable` is a detectable CDP signal
@@ -646,6 +653,7 @@ impl BrowserManager {
created_targets: HashSet::new(),
active_target_id: None,
relay_target_misses: HashMap::new(),
relay_scoped: false,
next_tab_id: 1,
capture_console: console_capture_enabled(),
};
@@ -749,6 +757,7 @@ impl BrowserManager {
created_targets: HashSet::new(),
active_target_id: None,
relay_target_misses: HashMap::new(),
relay_scoped: false,
next_tab_id: 1,
capture_console: console_capture_enabled(),
};
@@ -815,6 +824,106 @@ impl BrowserManager {
Ok(by_id.into_values().collect())
}
/// Every tab the relay knows, UNSCOPED (ignores group scoping) — for explicit
/// cross-group adoption (`chrome-use adopt`). Falls back to the scoped
/// `collect_page_targets` on a relay/browser that doesn't support the
/// unscoped query. Retries a few times over the relay (discovery is eventual).
async fn collect_all_targets(&self) -> Result<Vec<TargetInfo>, String> {
let rounds = if crate::connect::relay_url().is_some() {
3
} else {
1
};
let mut by_id: HashMap<String, TargetInfo> = HashMap::new();
let mut any_ok = false;
for i in 0..rounds {
if i > 0 {
tokio::time::sleep(Duration::from_millis(150)).await;
}
if let Ok(result) = self
.client
.send_command_typed::<_, GetTargetsResult>(
"ABRelay.getAllTargets",
&json!({}),
None,
)
.await
{
any_ok = true;
for t in result.target_infos.into_iter().filter(should_track_target) {
by_id.entry(t.target_id.clone()).or_insert(t);
}
}
}
if any_ok {
Ok(by_id.into_values().collect())
} else {
// Older relay without ABRelay.getAllTargets → best-effort scoped list.
self.collect_page_targets().await
}
}
/// Adopt a specific pre-existing tab matched by `spec` (an exact CDP
/// `targetId`, or a case-insensitive substring of the tab URL) WITHOUT opening
/// a new tab — for `chrome-use adopt`. Attaches it (the relay tags it into our
/// group), tracks + pins it. Errors if nothing matches (never creates a tab).
async fn adopt_existing_target(&mut self, spec: &str) -> Result<(), String> {
let all = self.collect_all_targets().await?;
let spec_l = spec.to_lowercase();
let target = all
.iter()
.find(|t| t.target_id == spec)
.or_else(|| all.iter().find(|t| t.url.to_lowercase().contains(&spec_l)))
.ok_or_else(|| {
let mut open: Vec<String> = all
.iter()
.map(|t| {
let u = if t.url.len() > 80 {
&t.url[..80]
} else {
&t.url
};
u.to_string()
})
.collect();
open.sort();
open.dedup();
format!(
"adopt: no open tab matching `{spec}` (by targetId or URL substring).\n\
{} tab(s) the extension can see:\n {}",
open.len(),
open.join("\n ")
)
})?
.clone();
let attach: AttachToTargetResult = self
.client
.send_command_typed(
"Target.attachToTarget",
&AttachToTargetParams {
target_id: target.target_id.clone(),
flatten: true,
},
None,
)
.await?;
let tab_id = self.assign_tab_id();
self.pages.push(PageInfo {
tab_id,
label: None,
target_id: target.target_id.clone(),
session_id: attach.session_id.clone(),
url: target.url.clone(),
title: sanitize_title(&target.title),
target_type: target.target_type.clone(),
});
self.active_page_index = self.pages.len() - 1;
self.pin_active_target();
self.enable_domains(&attach.session_id).await?;
Ok(())
}
async fn discover_and_attach_targets(&mut self) -> Result<(), String> {
self.client
.send_command_typed::<_, Value>(
@@ -824,6 +933,21 @@ impl BrowserManager {
)
.await?;
// Announce our group FIRST so the relay scopes the getTargets below to our
// own tab group (issue #40). On a launched browser this is a no-op.
let scoped = self.announce_group().await;
// `chrome-use adopt <spec>`: adopt a specific PRE-EXISTING tab instead of
// creating one — true zero-new-tab reading of the user's own tab. The
// directive rides in via env so it takes effect at first connect (before
// any about:blank would be made). If nothing matches, error out rather
// than fall back to creating a tab.
if let Ok(spec) = std::env::var("AGENT_BROWSER_ADOPT") {
if !spec.trim().is_empty() {
return self.adopt_existing_target(spec.trim()).await;
}
}
let page_targets: Vec<TargetInfo> = self.collect_page_targets().await?;
if page_targets.is_empty() {
@@ -870,19 +994,19 @@ impl BrowserManager {
self.active_page_index = 0;
self.pin_active_target();
self.enable_domains(&attach_result.session_id).await?;
} else if self.agent_group().is_some() {
// STRICT MULTI-AGENT ISOLATION (relay / the user's real Chrome).
// `page_targets` here are the USER's and OTHER agents' tabs. A tab
// group belongs to exactly ONE agent, so this session must NOT adopt
// any of them — it tracks ONLY tabs it creates (its own colored group)
// plus popups it opens. Adopting foreign tabs is precisely what let
// another concurrent agent's tab churn drop the tab we were driving and
// drift eval/click onto the wrong page (multi-agent failure). Open our
// own dedicated background tab in the session's group and pin it; the
// user's / other agents' tabs stay invisible to us.
} else if self.agent_group().is_some() && !scoped {
// STRICT MULTI-AGENT ISOLATION fallback (relay, but the group announce
// didn't take — e.g. an older relay). Without relay-side scoping,
// `page_targets` could be the USER's and OTHER agents' tabs, so this
// session must NOT adopt any of them — adopting foreign tabs is what let
// another agent's tab churn drop the tab we were driving (multi-agent
// failure). Open our own dedicated background tab and pin it instead.
self.tab_new(None, None).await?;
} else {
// A browser WE launched: every tab is ours, so adopt them all.
// Either a browser WE launched (every tab is ours) or the relay has
// scoped getTargets to our own tab group (#40) — so `page_targets` are
// all ours: adopt them (this restores follow-popup + cross-session
// adopt under isolation, since foreign tabs were already filtered out).
for target in &page_targets {
let attach_result: AttachToTargetResult = self
.client
@@ -1580,7 +1704,11 @@ impl BrowserManager {
// in the synthesized targetInfo), so don't adopt anything: the agent drives
// only tabs it explicitly created, and pop-ups (e.g. an OAuth/login window)
// are the user's. A launched browser (every tab ours) still follows pop-ups.
if self.agent_group().is_some() {
// Strict isolation only when on the relay WITHOUT group scoping: there a
// pop-up can't be told apart from a foreign tab, so adopt nothing. When the
// relay IS scoping (#40), getTargets returns only our group, so a tab that
// appeared after our own action is genuinely ours (a pop-up) — adopt it.
if self.agent_group().is_some() && !self.relay_scoped {
return None;
}
let result: GetTargetsResult = self
@@ -1658,16 +1786,17 @@ impl BrowserManager {
.collect();
let live_ids: HashSet<String> = live.iter().map(|t| t.target_id.clone()).collect();
let on_relay = self.agent_group().is_some();
// When the relay scopes getTargets to our group (#40), `live` is already
// only our own tabs, so adopting unknown ones is safe (a freshly-opened
// pop-up). Without scoping, keep strict isolation: never adopt a tab we
// didn't create — it belongs to the user or another agent.
let strict_isolation = on_relay && !self.relay_scoped;
for target in &live {
if self.update_page_target_info(target) {
continue;
}
// STRICT MULTI-AGENT ISOLATION: on the relay (the user's real Chrome,
// shared with other agents), NEVER adopt a tab this session didn't
// create — it belongs to the user or another agent's group. Only a
// browser we launched (every tab ours) adopts unknown targets.
if on_relay {
if strict_isolation {
continue;
}
let attach_result: AttachToTargetResult = match self
@@ -1837,6 +1966,25 @@ impl BrowserManager {
}
}
/// Tell the relay which tab group this session owns so it can scope
/// `Target.getTargets` to us (issue #40). Only meaningful on the relay; a
/// no-op (returns false) on a launched/real-CDP connection. Sets and returns
/// `relay_scoped`: when true, the daemon can trust getTargets to contain only
/// our group and re-enable adopting new tabs (pop-ups, cross-session adopt).
async fn announce_group(&mut self) -> bool {
let Some(group) = self.agent_group() else {
self.relay_scoped = false;
return false;
};
let ok = self
.client
.send_command_typed::<_, Value>("ABRelay.setGroup", &json!({ "group": group }), None)
.await
.is_ok();
self.relay_scoped = ok;
ok
}
pub async fn tab_new(
&mut self,
url: Option<&str>,
@@ -2630,6 +2778,7 @@ async fn initialize_lightpanda_manager(
created_targets: HashSet::new(),
active_target_id: None,
relay_target_misses: HashMap::new(),
relay_scoped: false,
next_tab_id: 1,
capture_console: console_capture_enabled(),
};
+10
View File
@@ -21,6 +21,16 @@ pub async fn run_daemon(session: &str) {
// (via the ab-connect extension) land in a per-session Chrome tab group.
let _ = super::browser::DAEMON_SESSION.set(session.to_string());
// Bootstrap / refresh the site-adapter pack in the background (first-run +
// periodic TTL). This populates ~/.chrome-use/sites/.index.json so navigation
// can auto-suggest `site` commands for the page you land on, with zero added
// latency to any command. Best-effort; offline is a no-op.
if crate::site::needs_refresh() {
tokio::spawn(async {
let _ = crate::site::update().await;
});
}
let socket_dir = get_daemon_socket_dir();
if !socket_dir.exists() {
let _ = fs::create_dir_all(&socket_dir);
+30 -4
View File
@@ -1529,9 +1529,27 @@ pub async fn get_element_input_value(
.send_command_typed(
"Runtime.callFunctionOn",
&CallFunctionOnParams {
function_declaration:
"function() { return typeof this.value === 'string' ? this.value : ''; }"
.to_string(),
// Read rich-editor content too (issue #41): CodeMirror 5 / Monaco
// keep their text in a model, not `.value`; contenteditable keeps
// it as innerText. Falls back to `.value` for plain inputs.
function_declaration: r#"function() {
const el = this;
const cm5 = el.closest && el.closest('.CodeMirror');
if (cm5 && cm5.CodeMirror) return cm5.CodeMirror.getValue();
if (window.monaco && monaco.editor) {
try {
const eds = monaco.editor.getEditors ? monaco.editor.getEditors() : [];
const ed = eds.find(e => e.getDomNode && e.getDomNode().contains(el)) || eds[0];
if (ed) return ed.getValue();
const m = monaco.editor.getModels ? monaco.editor.getModels() : [];
if (m[0]) return m[0].getValue();
} catch (e) {}
}
if (typeof el.value === 'string') return el.value;
if (el.isContentEditable) return el.innerText;
return '';
}"#
.to_string(),
object_id: Some(object_id),
arguments: None,
return_by_value: Some(true),
@@ -1609,7 +1627,15 @@ pub async fn get_element_bounding_box(
&CallFunctionOnParams {
function_declaration: r#"function() {
const r = this.getBoundingClientRect();
return { x: r.x, y: r.y, width: r.width, height: r.height };
const inViewport = r.bottom > 0 && r.right > 0
&& r.top < (innerHeight || document.documentElement.clientHeight)
&& r.left < (innerWidth || document.documentElement.clientWidth);
return {
x: r.x, y: r.y, width: r.width, height: r.height,
centerX: Math.round(r.x + r.width / 2),
centerY: Math.round(r.y + r.height / 2),
inViewport,
};
}"#
.to_string(),
object_id: Some(object_id),
+52 -17
View File
@@ -579,7 +579,7 @@ pub async fn fill(
selector_or_ref: &str,
value: &str,
iframe_sessions: &HashMap<String, String>,
) -> Result<(), String> {
) -> Result<String, String> {
let (object_id, effective_session_id) = resolve_element_object_id(
client,
session_id,
@@ -590,14 +590,15 @@ pub async fn fill(
.await?;
// Emulate a real edit so framework-controlled inputs (React/Vue) and
// site-side listeners actually see the change (issue #25): the old path set
// `this.value` directly and used Input.insertText, which left React's
// internal value-tracker out of sync and never fired change/blur — so
// dependent logic (e.g. Mercari's postal-code → 都道府県 autocomplete) never
// ran even though the value was visible. Set the value through the element's
// PROTOTYPE setter (which React's _valueTracker hooks), then dispatch
// input → change → blur/focusout. `type <sel> <text>` remains for sites that
// need per-keystroke events.
// site-side listeners actually see the change (issue #25): set the value
// through the element's PROTOTYPE setter (which React's _valueTracker hooks),
// then dispatch input → changeblur/focusout. Beyond plain inputs, detect
// rich editors and use their own API/events (issue #41): CodeMirror 5 and
// Monaco have a model that `.value`/`textContent` can't touch; ProseMirror /
// contenteditable need `execCommand('insertText')` so beforeinput/input fire
// (a raw `textContent =` corrupts PM's doc and skips React composers).
// Returns the engine used so the caller can report it. `type <sel> <text>`
// remains for sites that need per-keystroke events.
let fill_js = format!(
r#"function() {{
const el = this;
@@ -605,13 +606,43 @@ pub async fn fill(
try {{ el.focus(); }} catch (e) {{}}
const tag = el.tagName;
const fire = (type, ctor) => el.dispatchEvent(new (ctor || Event)(type, {{ bubbles: true }}));
if (tag === 'SELECT') {{
el.value = v; fire('input'); fire('change'); return true;
// CodeMirror 5: a hidden <textarea> inside .CodeMirror with a live instance.
const cm5 = el.closest && el.closest('.CodeMirror');
if (cm5 && cm5.CodeMirror) {{ cm5.CodeMirror.setValue(v); return 'codemirror5'; }}
// Monaco: global `monaco`; prefer the editor whose DOM contains el.
if (window.monaco && monaco.editor) {{
try {{
const eds = monaco.editor.getEditors ? monaco.editor.getEditors() : [];
const ed = eds.find(e => e.getDomNode && e.getDomNode().contains(el)) || eds[0];
if (ed) {{ ed.setValue(v); return 'monaco'; }}
const models = monaco.editor.getModels ? monaco.editor.getModels() : [];
if (models[0]) {{ models[0].setValue(v); return 'monaco'; }}
}} catch (e) {{}}
}}
if (tag === 'SELECT') {{ el.value = v; fire('input'); fire('change'); return 'select'; }}
if (el.isContentEditable) {{
el.textContent = v; fire('input', window.InputEvent || Event); fire('change');
try {{ el.blur(); }} catch (e) {{}} fire('focusout'); return true;
// ProseMirror / contenteditable: select-all then insertText fires
// beforeinput/input that PM and React composers listen for.
let ok = false;
try {{
const sel = window.getSelection();
const range = document.createRange();
range.selectNodeContents(el);
sel.removeAllRanges();
sel.addRange(range);
ok = document.execCommand('insertText', false, v);
}} catch (e) {{}}
if (!ok) {{ el.textContent = v; fire('input', window.InputEvent || Event); }}
fire('change');
try {{ el.blur(); }} catch (e) {{}}
fire('focusout');
return ok ? 'contenteditable' : 'contenteditable-fallback';
}}
const proto = tag === 'TEXTAREA' ? window.HTMLTextAreaElement.prototype
: window.HTMLInputElement.prototype;
const desc = Object.getOwnPropertyDescriptor(proto, 'value');
@@ -623,13 +654,13 @@ pub async fn fill(
fire('change');
try {{ el.blur(); }} catch (e) {{}}
fire('focusout'); // blur-triggered lookups/validation
return true;
return 'input';
}}"#,
val = serde_json::to_string(value).unwrap_or_default()
);
client
.send_command_typed::<_, Value>(
let result: EvaluateResult = client
.send_command_typed(
"Runtime.callFunctionOn",
&CallFunctionOnParams {
function_declaration: fill_js,
@@ -642,7 +673,11 @@ pub async fn fill(
)
.await?;
Ok(())
Ok(result
.result
.value
.and_then(|v| v.as_str().map(String::from))
.unwrap_or_else(|| "input".to_string()))
}
#[allow(clippy::too_many_arguments)]
+273 -4
View File
@@ -52,6 +52,22 @@ pub struct RelayState {
pending: HashMap<i64, (ClientId, Value)>,
/// monotonic source of relay-global command ids
next_global_id: i64,
/// Group-scoped isolation (issue #40). A tab group belongs to exactly one
/// agent/session; the relay scopes `Target.getTargets` per client to its own
/// group so the daemon can safely adopt new tabs (follow-popup, cross-session
/// adopt) without ever seeing the user's or another agent's tabs.
///
/// clientId -> group name. A client that never announced a group (older
/// daemon) is absent here and gets the full, UNSCOPED target list — so this
/// is fully backward-compatible.
client_groups: HashMap<ClientId, String>,
/// targetId -> group name. Created tabs are tagged from `Target.createTarget`'s
/// `agentGroup`; an explicitly adopted tab is tagged to the adopter; a pop-up
/// inherits its opener's group (needs the extension to report `openerTargetId`).
target_group: HashMap<String, String>,
/// relay-global id of an in-flight `Target.createTarget` -> the `agentGroup`
/// it carried, so the reply's `targetId` can be tagged with that group.
pending_create: HashMap<i64, String>,
}
/// What to do with a raw CDP command received from a `CdpClient`.
@@ -95,6 +111,7 @@ impl RelayState {
/// `pending` entries don't leak.
pub fn drop_client(&mut self, client_id: ClientId) {
self.pending.retain(|_, (cid, _)| *cid != client_id);
self.client_groups.remove(&client_id);
}
/// Route a raw CDP command `{id, method, params?, sessionId?}` from a
@@ -123,12 +140,28 @@ impl RelayState {
"jsVersion": ""
}
})),
// Non-CDP control message: a daemon announces which tab group
// (session) it owns, so getTargets can be scoped to it (issue #40).
"ABRelay.setGroup" => {
if let Some(g) = params.get("group").and_then(|g| g.as_str()) {
if !g.is_empty() {
self.client_groups.insert(client_id, g.to_string());
}
}
ClientRoute::Local(json!({ "id": id, "result": {} }))
}
// Discovery is best-effort and event-driven in real CDP; abs only
// reads the getTargets result, so an empty ack is enough here.
"Target.setDiscoverTargets" | "Target.setAutoAttach" => {
ClientRoute::Local(json!({ "id": id, "result": {} }))
}
"Target.getTargets" => {
// Unscoped discovery for EXPLICIT cross-group adoption (`chrome-use
// adopt`): returns every target the extension has attached, ignoring
// group scoping, so an agent can find a specific pre-existing tab (the
// user's, another session's) by URL/targetId and adopt it. Isolation
// is preserved because the daemon only acts on the one tab it then
// attaches (which the relay re-tags into the adopter's group).
"ABRelay.getAllTargets" => {
let infos: Vec<Value> = self
.targets
.values()
@@ -136,15 +169,44 @@ impl RelayState {
.collect();
ClientRoute::Local(json!({ "id": id, "result": { "targetInfos": infos } }))
}
"Target.getTargets" => {
// Scope to the client's own group when it announced one; an
// un-announced (legacy) client gets the full list (back-compat).
let scoped = self.client_groups.get(&client_id).cloned();
let infos: Vec<Value> = self
.targets
.iter()
.filter(|(tid, _)| match &scoped {
Some(g) => self
.target_group
.get(*tid)
.map(|tg| tg == g)
.unwrap_or(false),
None => true,
})
.map(|(_, t)| t.target_info.clone())
.collect();
ClientRoute::Local(json!({ "id": id, "result": { "targetInfos": infos } }))
}
"Target.attachToTarget" => {
let target_id = params
.get("targetId")
.and_then(|t| t.as_str())
.unwrap_or("");
match self.targets.get(target_id) {
Some(entry) => ClientRoute::Local(
json!({ "id": id, "result": { "sessionId": entry.session_id } }),
),
Some(entry) => {
let session_id = entry.session_id.clone();
// Explicitly adopting a target makes it this client's
// (cross-session adopt, #21) — tag it into the adopter's
// group so it stays in that client's scoped getTargets and
// isn't churn-pruned.
if let Some(g) = self.client_groups.get(&client_id).cloned() {
self.target_group.insert(target_id.to_string(), g);
}
ClientRoute::Local(
json!({ "id": id, "result": { "sessionId": session_id } }),
)
}
None => ClientRoute::Local(json!({
"id": id,
"error": { "code": -32602, "message": format!("No such target {target_id}") }
@@ -157,6 +219,18 @@ impl RelayState {
self.next_global_id += 1;
let gid = self.next_global_id;
self.pending.insert(gid, (client_id, id));
// Remember the group a createTarget carries so the reply's
// targetId can be tagged to the creating session (issue #40).
if method == "Target.createTarget" {
if let Some(g) = params.get("agentGroup").and_then(|g| g.as_str()) {
if !g.is_empty() {
self.pending_create.insert(gid, g.to_string());
self.client_groups
.entry(client_id)
.or_insert_with(|| g.to_string());
}
}
}
ClientRoute::Forward(json!({
"id": gid,
"method": "forwardCDPCommand",
@@ -201,6 +275,17 @@ impl RelayState {
&& msg.get("method").is_none()
{
let gid = msg.get("id").and_then(|i| i.as_i64());
// A createTarget reply: tag the new tab's targetId with the group the
// command carried, so it lands in the creating session's scope (#40).
if let Some(g) = gid.and_then(|g| self.pending_create.remove(&g)) {
if let Some(tid) = msg
.get("result")
.and_then(|r| r.get("targetId"))
.and_then(|t| t.as_str())
{
self.target_group.insert(tid.to_string(), g);
}
}
let (to, orig_id) = match gid.and_then(|g| self.pending.remove(&g)) {
Some((client_id, orig)) => (Some(client_id), orig),
// No mapping (stale/unknown id) — fall back to broadcasting with
@@ -241,6 +326,27 @@ impl RelayState {
.and_then(|s| s.as_str())
.unwrap_or("")
.to_string();
// Attribute the tab to a group for scoping (issue #40),
// unless we already know it (createTarget tag). An
// explicit `abGroup` from the extension wins; otherwise a
// pop-up inherits its opener's group via `openerTargetId`.
if !self.target_group.contains_key(tid) {
if let Some(g) = info
.get("abGroup")
.and_then(|g| g.as_str())
.filter(|g| !g.is_empty())
{
self.target_group.insert(tid.to_string(), g.to_string());
} else if let Some(opener) = info
.get("openerTargetId")
.and_then(|o| o.as_str())
.filter(|o| !o.is_empty())
{
if let Some(g) = self.target_group.get(opener).cloned() {
self.target_group.insert(tid.to_string(), g);
}
}
}
self.targets.insert(
tid.to_string(),
TargetEntry {
@@ -255,6 +361,15 @@ impl RelayState {
"Target.detachedFromTarget" => {
let gone = inner_params.get("sessionId").and_then(|s| s.as_str());
if let Some(gone) = gone {
let gone_tids: Vec<String> = self
.targets
.iter()
.filter(|(_, e)| e.session_id == gone)
.map(|(tid, _)| tid.clone())
.collect();
for tid in gone_tids {
self.target_group.remove(&tid);
}
self.targets.retain(|_, e| e.session_id != gone);
}
return vec![];
@@ -557,4 +672,158 @@ mod tests {
_ => panic!("expected ToExt"),
}
}
// === Group-scoped isolation (issue #40) ===
/// Drive the real create path: announce group, createTarget(agentGroup), feed
/// the ext reply (tags target→group) + the attachedToTarget event (creates the
/// entry). Returns nothing; mutates `s`.
fn create_in_group(s: &mut RelayState, client: ClientId, group: &str, tid: &str, sid: &str) {
s.route_client_command(
client,
&json!({ "id": 1, "method": "ABRelay.setGroup", "params": { "group": group } }),
);
let route = s.route_client_command(
client,
&json!({ "id": 2, "method": "Target.createTarget",
"params": { "url": "about:blank", "agentGroup": group } }),
);
let gid = match route {
ClientRoute::Forward(env) => env["id"].as_i64().unwrap(),
_ => panic!("createTarget must forward"),
};
s.handle_ext_message(&json!({ "id": gid, "result": { "targetId": tid } }), "");
s.handle_ext_message(
&json!({ "method": "forwardCDPEvent", "params": {
"method": "Target.attachedToTarget",
"params": { "sessionId": sid, "targetInfo": {
"targetId": tid, "type": "page", "url": "about:blank", "attached": true } } } }),
"",
);
}
fn get_target_ids(s: &mut RelayState, client: ClientId) -> Vec<String> {
match s.route_client_command(client, &json!({ "id": 9, "method": "Target.getTargets" })) {
ClientRoute::Local(v) => v["result"]["targetInfos"]
.as_array()
.unwrap()
.iter()
.map(|t| t["targetId"].as_str().unwrap().to_string())
.collect(),
_ => panic!("getTargets must be local"),
}
}
#[test]
fn get_targets_is_scoped_to_each_clients_group() {
let mut s = RelayState::new();
create_in_group(&mut s, 1, "agent-a", "ta", "sa");
create_in_group(&mut s, 2, "agent-b", "tb", "sb");
// Each client sees ONLY its own group's tab — never the other agent's.
assert_eq!(get_target_ids(&mut s, 1), vec!["ta"]);
assert_eq!(get_target_ids(&mut s, 2), vec!["tb"]);
}
#[test]
fn legacy_client_without_group_sees_all_targets() {
let mut s = RelayState::new();
create_in_group(&mut s, 1, "agent-a", "ta", "sa");
create_in_group(&mut s, 2, "agent-b", "tb", "sb");
// Client 3 never announced a group → full, unscoped list (back-compat).
let mut all = get_target_ids(&mut s, 3);
all.sort();
assert_eq!(all, vec!["ta", "tb"]);
}
#[test]
fn popup_inherits_opener_group_and_is_visible_to_that_client_only() {
let mut s = RelayState::new();
create_in_group(&mut s, 1, "agent-a", "ta", "sa");
create_in_group(&mut s, 2, "agent-b", "tb", "sb");
// A pop-up that agent-a's tab opened: extension reports openerTargetId=ta.
s.handle_ext_message(
&json!({ "method": "forwardCDPEvent", "params": {
"method": "Target.attachedToTarget",
"params": { "sessionId": "sp", "targetInfo": {
"targetId": "tp", "type": "page", "url": "https://oauth.example/",
"attached": true, "openerTargetId": "ta" } } } }),
"",
);
// Only agent-a sees the pop-up; agent-b never does.
let mut a = get_target_ids(&mut s, 1);
a.sort();
assert_eq!(a, vec!["ta", "tp"]);
assert_eq!(get_target_ids(&mut s, 2), vec!["tb"]);
}
#[test]
fn explicit_attach_tags_target_into_adopter_group() {
let mut s = RelayState::new();
// A pre-existing, ungrouped tab the extension reported (e.g. user's tab).
s.handle_ext_message(
&json!({ "method": "forwardCDPEvent", "params": {
"method": "Target.attachedToTarget",
"params": { "sessionId": "su", "targetInfo": {
"targetId": "tu", "type": "page", "url": "https://user.example/", "attached": true } } } }),
"",
);
// Client 1 (group agent-a) explicitly adopts it by targetId (#21).
s.route_client_command(
1,
&json!({ "id": 1, "method": "ABRelay.setGroup", "params": { "group": "agent-a" } }),
);
s.route_client_command(
1,
&json!({ "id": 2, "method": "Target.attachToTarget", "params": { "targetId": "tu" } }),
);
// Now it's in agent-a's scope and survives the scoped getTargets.
assert_eq!(get_target_ids(&mut s, 1), vec!["tu"]);
// A different agent still doesn't see it.
s.route_client_command(
2,
&json!({ "id": 1, "method": "ABRelay.setGroup", "params": { "group": "agent-b" } }),
);
assert!(get_target_ids(&mut s, 2).is_empty());
}
#[test]
fn get_all_targets_is_unscoped() {
let mut s = RelayState::new();
create_in_group(&mut s, 1, "agent-a", "ta", "sa");
create_in_group(&mut s, 2, "agent-b", "tb", "sb");
// Client 1's scoped getTargets sees only its own group...
assert_eq!(get_target_ids(&mut s, 1), vec!["ta"]);
// ...but ABRelay.getAllTargets returns EVERY target regardless of group
// (for explicit cross-group adoption).
let all = match s
.route_client_command(1, &json!({ "id": 1, "method": "ABRelay.getAllTargets" }))
{
ClientRoute::Local(v) => {
let mut ids: Vec<String> = v["result"]["targetInfos"]
.as_array()
.unwrap()
.iter()
.map(|t| t["targetId"].as_str().unwrap().to_string())
.collect();
ids.sort();
ids
}
_ => panic!("getAllTargets must be local"),
};
assert_eq!(all, vec!["ta", "tb"]);
}
#[test]
fn detach_clears_target_group() {
let mut s = RelayState::new();
create_in_group(&mut s, 1, "agent-a", "ta", "sa");
assert_eq!(get_target_ids(&mut s, 1), vec!["ta"]);
s.handle_ext_message(
&json!({ "method": "forwardCDPEvent", "params": {
"method": "Target.detachedFromTarget", "params": { "sessionId": "sa" } } }),
"",
);
assert!(get_target_ids(&mut s, 1).is_empty());
assert!(!s.target_group.contains_key("ta"));
}
}
+65 -5
View File
@@ -186,6 +186,26 @@ pub fn print_response_with_opts(resp: &Response, action: Option<&str>, opts: &Ou
}
if let Some(data) = &resp.data {
// Auto-trigger: when you land on / read a page whose domain has site
// adapters, surface them so the agent pulls structured data via
// `chrome-use site <name>/<cmd>` instead of scraping the DOM. (In --json
// mode this same info rides along in the `siteAdapters` field above.)
if let Some(hint) = data.get("siteAdapters") {
let domain = hint.get("domain").and_then(|v| v.as_str()).unwrap_or("");
let cmds: Vec<&str> = hint
.get("commands")
.and_then(|v| v.as_array())
.map(|a| a.iter().filter_map(|v| v.as_str()).collect())
.unwrap_or_default();
if !cmds.is_empty() {
eprintln!("💡 site adapters for {domain} — prefer these for structured data:");
eprintln!(" {}", color::dim(&cmds.join(", ")));
eprintln!(
" {}",
color::dim(&format!("e.g. chrome-use site {} --json", cmds[0]))
);
}
}
// A click that opened a new tab: surface it so the agent doesn't read the
// unchanged old page as a failed click (issue #24-A).
if let Some(opened) = data.get("openedTab") {
@@ -469,6 +489,17 @@ pub fn print_response_with_opts(resp: &Response, action: Option<&str>, opts: &Ou
println!("y: {}", y);
println!("width: {}", w);
println!("height: {}", h);
if let (Some(cx), Some(cy)) = (
obj.get("centerX").and_then(|v| v.as_i64()),
obj.get("centerY").and_then(|v| v.as_i64()),
) {
// Echoed in click-ready CSS px so the agent can paste straight
// into `click <centerX> <centerY>` (issue #43).
println!("center: {} {}", cx, cy);
}
if let Some(iv) = obj.get("inViewport").and_then(|v| v.as_bool()) {
println!("inViewport: {}", iv);
}
}
return;
}
@@ -1470,9 +1501,20 @@ Examples:
chrome-use fill - Clear and fill an input field
Usage: chrome-use fill <selector> <text>
chrome-use fill <selector> --file <path>
chrome-use fill <selector> --stdin
Clears the input field and fills it with the specified text.
This replaces any existing content in the field.
Clears the field and fills it with the text, replacing existing content.
Works on rich editors too (issue #41): CodeMirror 5, Monaco, ProseMirror and
plain contenteditable are detected and set via their own API / input events,
not a raw `.value` write and the response echoes which `engine` was used.
For framework inputs (React/Vue/Angular) the value goes through the native
setter so the form registers it (no more "pristine" Save no-ops).
Options:
--file <path> Read the value from a UTF-8 file (large/multiline text,
backticks/quotes/newlines/non-ASCII no shell escaping)
--stdin Read the value from stdin
Global Options:
--json Output as JSON
@@ -1481,7 +1523,8 @@ Global Options:
Examples:
chrome-use fill "#email" "user@example.com"
chrome-use fill @e3 "Hello World"
chrome-use fill "input[name='search']" "query"
chrome-use fill ".CodeMirror" --file ./article.md # set a CodeMirror editor
cat post.md | chrome-use fill @e7 --stdin
"##
}
"type" => {
@@ -1883,6 +1926,13 @@ Options:
--full, -f Capture full page (not just viewport)
[selector] Capture just an element (CSS or @ref), e.g. `screenshot ".header" h.png`
--clip <x,y,w,h> Capture a pixel region, e.g. `screenshot --clip 0,0,200,40 corner.png`
--max-width <px> Downscale so the image's width px (preserves aspect)
--max-height <px> Downscale so the image's height px
--scale <0..1> Downscale by a factor, e.g. 0.5 (DPR-1, so screenshot px
line up 1:1 with `click x y`)
Default: capped at 2000px longest edge unless overridden
(AGENT_BROWSER_SCREENSHOT_MAX_EDGE; 0 disables). Annotated
shots are never downscaled, so ref overlays stay aligned.
--annotate Overlay numbered labels on interactive elements.
Each label [N] corresponds to ref @eN from snapshot.
Prints a legend mapping labels to element roles/names.
@@ -1905,6 +1955,8 @@ Examples:
chrome-use screenshot --full ./full-page.png
chrome-use screenshot ".header .indicator" corner.png # just one element
chrome-use screenshot --clip 1600,0,200,40 corner.png # a pixel region
chrome-use screenshot --scale 0.5 ./half.png # DPR-1: screenshot px == click px
chrome-use screenshot --max-width 1400 ./shot.png # cap width for image readers
chrome-use screenshot --annotate # Labeled screenshot + legend
chrome-use screenshot --annotate ./page.png # Save annotated screenshot
chrome-use screenshot --annotate --json # JSON output with annotations
@@ -3243,7 +3295,8 @@ Core Commands:
click <sel|x y> Click element/@ref, or a viewport coordinate
dblclick <sel> Double-click element
type <sel> <text> Type into element
fill <sel> <text> Clear and fill
fill <sel> <text> Clear and fill (handles CodeMirror/Monaco/ProseMirror/
contenteditable; `--file <path>`/`--stdin` for large text)
press <key> [--hold <ms>] Press key (Enter, Tab, Control+a). --hold keeps it
down <ms> then releases precise (in-daemon), for
games/charge: `press d --hold 800`
@@ -3263,7 +3316,8 @@ Core Commands:
scroll <dir> [px] Scroll (up/down/left/right)
scrollintoview <sel> Scroll element into view
wait <sel|ms> Wait for element or time
screenshot [path] Take screenshot
screenshot [path] Take screenshot (auto-downscaled to 2000px long edge;
--max-width/--max-height/--scale to override)
pdf <path> Save as PDF
snapshot Accessibility tree with refs (for AI)
eval <js> Run JavaScript
@@ -3277,6 +3331,8 @@ Navigation:
Get Info: chrome-use get <what> [selector]
text, html, value, attr <name>, title, url, count, box, styles, cdp-url
box <sel> x,y,width,height,centerX,centerY,inViewport in CSS px (feed
centerX/centerY into `click x y`); value reads CodeMirror/Monaco too
text (no selector = whole page, all frames), text --main, frames (list)
Check State: chrome-use is <what> <selector>
@@ -3319,6 +3375,10 @@ Tabs:
stable targetId, no reload preserves in-page state
open <url> --reuse-tab Reuse an existing tab on that URL instead of spawning
a duplicate (matches origin+path; preserves state)
adopt <url|targetId> Read a PRE-EXISTING tab (the user's own, or another
session's) WITHOUT opening a new one matches by URL
substring or stable targetId, then drives it. e.g.
`adopt "github.com/owner/repo"`
Diff:
diff snapshot Compare current vs last snapshot
+118
View File
@@ -199,9 +199,127 @@ pub async fn update() -> Result<usize, String> {
count += 1;
}
}
// Build the domain→adapters index and stamp the sync time so navigation can
// suggest adapters (auto-trigger) and `needs_refresh` can pace re-syncs.
write_domain_index(&dir);
if let Some(p) = last_update_path() {
let _ = std::fs::write(p, now_secs().to_string());
}
Ok(count)
}
/// `~/.chrome-use/sites/.last_update` — unix-seconds marker of the last sync.
fn last_update_path() -> Option<PathBuf> {
sites_dir().map(|d| d.join(".last_update"))
}
/// `~/.chrome-use/sites/.index.json` — `{ "github.com": ["github/issues", …], … }`,
/// built on `update` so navigation can look up adapters by domain without parsing
/// all ~145 adapter files on every command.
fn index_path() -> Option<PathBuf> {
sites_dir().map(|d| d.join(".index.json"))
}
fn now_secs() -> u64 {
std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.map(|d| d.as_secs())
.unwrap_or(0)
}
/// Parse every installed adapter and write the domain→adapters index. Within a
/// domain, read-only adapters are listed first (then alphabetical) so the
/// auto-suggested example leads with a safe read, not a write action.
fn write_domain_index(dir: &std::path::Path) {
let mut by_domain: std::collections::BTreeMap<String, Vec<(bool, String)>> = Default::default();
for spec in list_adapters().unwrap_or_default() {
if let Ok(a) = load_adapter(&spec) {
if let Some(d) = a.domain() {
let read_only = a
.meta
.get("readOnly")
.and_then(|v| v.as_bool())
.unwrap_or(false);
by_domain
.entry(d.to_string())
.or_default()
.push((read_only, spec));
}
}
}
let ordered: std::collections::BTreeMap<String, Vec<String>> = by_domain
.into_iter()
.map(|(domain, mut v)| {
// read-only (true) first, then by spec name
v.sort_by(|a, b| b.0.cmp(&a.0).then_with(|| a.1.cmp(&b.1)));
(domain, v.into_iter().map(|(_, s)| s).collect())
})
.collect();
if let Ok(json) = serde_json::to_string(&ordered) {
let _ = std::fs::write(dir.join(".index.json"), json);
}
}
const DEFAULT_TTL_DAYS: u64 = 7;
/// Whether the adapter pack should be (re)synced: true on first use (nothing
/// installed) or when the last sync is older than the TTL. Disabled by
/// `AGENT_BROWSER_SITES_NO_AUTO_UPDATE=1`; TTL overridable via
/// `AGENT_BROWSER_SITES_TTL_DAYS` (0 = always).
pub fn needs_refresh() -> bool {
if std::env::var_os("AGENT_BROWSER_SITES_NO_AUTO_UPDATE").is_some() {
return false;
}
let Some(dir) = sites_dir() else {
return false;
};
// First use: no adapters installed yet.
if list_adapters().map(|l| l.is_empty()).unwrap_or(true) {
let _ = &dir;
return true;
}
let ttl_days = std::env::var("AGENT_BROWSER_SITES_TTL_DAYS")
.ok()
.and_then(|s| s.parse::<u64>().ok())
.unwrap_or(DEFAULT_TTL_DAYS);
let ttl = ttl_days.saturating_mul(86_400);
match last_update_path().and_then(|p| std::fs::read_to_string(p).ok()) {
Some(s) => match s.trim().parse::<u64>() {
Ok(ts) => now_secs().saturating_sub(ts) >= ttl,
Err(_) => true,
},
None => true, // no marker → treat as stale
}
}
/// Adapters whose `@meta.domain` matches `host` (exact, or `host` is a subdomain
/// of it) — for auto-suggesting `site` commands when you land on a known site.
/// Reads the prebuilt `.index.json`; empty if the pack isn't synced yet.
pub fn adapters_for_domain(host: &str) -> Vec<String> {
let host = host.trim_start_matches("www.");
let Some(raw) = index_path().and_then(|p| std::fs::read_to_string(p).ok()) else {
return Vec::new();
};
let Ok(idx) = serde_json::from_str::<std::collections::BTreeMap<String, Vec<String>>>(&raw)
else {
return Vec::new();
};
// Preserve the index's per-domain ordering (read-only adapters first); just
// dedup if a host somehow matches multiple domain keys.
let mut out: Vec<String> = Vec::new();
for (domain, specs) in idx {
let d = domain.trim_start_matches("www.");
if host == d || host.ends_with(&format!(".{d}")) {
for s in specs {
if !out.contains(&s) {
out.push(s);
}
}
}
}
out
}
/// Map CLI args to the adapter's `args` object. Positional args fill the adapter's
/// declared `args` keys in order; `--key value` overrides by name. The adapter
/// validates required args itself.
Binary file not shown.
Binary file not shown.
+57 -6
View File
@@ -81,6 +81,35 @@ async function groupTabInto(tabId, name) {
groupIdByName.set(name, gid)
}
// Group-scoped relay isolation hints (issue #40). The relay scopes
// Target.getTargets per agent by tab group; report two things in the synthesized
// targetInfo so it can attribute each tab:
// - abGroup: the tab's Chrome tab-group TITLE (= the owning session name), so
// the relay can re-attribute existing tabs after a restart (createTarget
// tagging won't re-run for already-open tabs).
// - openerTargetId: the targetId of the tab that opened this one, so a pop-up
// (window.open / target=_blank / OAuth result) inherits its opener's group
// and the agent that opened it can follow it — without foreign tabs leaking.
// Best-effort: any failure yields empty strings, which the relay ignores.
async function tabScopeHints(tabId) {
let openerTargetId = ''
let abGroup = ''
try {
const t = await chrome.tabs.get(tabId)
if (t) {
if (typeof t.openerTabId === 'number') {
const op = tabs.get(t.openerTabId)
if (op) openerTargetId = op.targetId
}
if (t.groupId != null && t.groupId >= 0 && chrome.tabGroups) {
const g = await chrome.tabGroups.get(t.groupId).catch(() => null)
if (g && g.title) abGroup = g.title
}
}
} catch {}
return { openerTargetId, abGroup }
}
function postToHost(msg) {
try {
if (port) port.postMessage(msg)
@@ -126,7 +155,7 @@ function connectHost() {
} catch {}
// Tell the daemon about everything we already have attached, then attach
// anything new.
reannounceAttachedTabs()
void reannounceAttachedTabs()
void attachAllTabs()
}
@@ -140,7 +169,7 @@ async function onHostMessage(msg) {
// Daemon (re)connected — (re)attach and announce every tab so it discovers
// the user's existing tabs rather than racing an empty target list.
if (msg.method === 'attachAll') {
reannounceAttachedTabs()
void reannounceAttachedTabs()
await attachAllTabs()
return
}
@@ -387,12 +416,16 @@ async function attachTab(tabId) {
sessionToTab.set(sessionId, tabId)
rememberSessionTarget(sessionId, targetId)
setBadge(tabId, port ? 'on' : 'connecting')
const { openerTargetId, abGroup } = await tabScopeHints(tabId)
postToHost({
method: 'forwardCDPEvent',
params: {
sessionId,
method: 'Target.attachedToTarget',
params: { sessionId, targetInfo: { ...targetInfo, attached: true } },
params: {
sessionId,
targetInfo: { ...targetInfo, attached: true, openerTargetId, abGroup },
},
},
})
return entry
@@ -434,14 +467,32 @@ async function attachAllTabs() {
}
}
function reannounceAttachedTabs() {
for (const [, entry] of tabs.entries()) {
async function reannounceAttachedTabs() {
for (const [tabId, entry] of tabs.entries()) {
// Re-send the group hint too (issue #40) so the relay can rebuild its
// targetId→group map after its own restart (createTarget tagging won't
// re-run for tabs that are already open). Include the live url/title so the
// relay's target list stays matchable by URL after a reconnect (otherwise a
// reannounced tab shows a blank url and `adopt <url>` can't find it).
const { openerTargetId, abGroup } = await tabScopeHints(tabId)
let url = ''
let title = ''
try {
const t = await chrome.tabs.get(tabId)
if (t) {
url = t.url || t.pendingUrl || ''
title = t.title || ''
}
} catch {}
postToHost({
method: 'forwardCDPEvent',
params: {
sessionId: entry.sessionId,
method: 'Target.attachedToTarget',
params: { sessionId: entry.sessionId, targetInfo: { targetId: entry.targetId, type: 'page', attached: true } },
params: {
sessionId: entry.sessionId,
targetInfo: { targetId: entry.targetId, type: 'page', url, title, attached: true, openerTargetId, abGroup },
},
},
})
}
+1 -1
View File
@@ -1,7 +1,7 @@
{
"manifest_version": 3,
"name": "chrome-use",
"version": "0.4.9",
"version": "0.4.11",
"description": "Let chrome-use drive your logged-in Chrome \u2014 install once, no token, no per-use confirmation.",
"key": "MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA6vQIyscGIPYPZdSpPwPL0+0gxUROyRgCpmvCSDoc8XUm4qm97VbKnD9Ijc1lV22lNWZtE78gaRjt6BeSfuMgnBymnhLKjN1gU6AI5QUU0mrJyeHdWKvrKQR5FmsM2A7Xr1ykE2SiiS8zNUS3Y/6O5l+Nva7wrVy6E4a2dkBVQkOsu+DV+nEZvhIyuDY5D5SPXqNwUTWTaglwj5mjvHz36xSwCWlPmrtJ+ED0AUyrb2z4GIOmvk4kqtBVrh/UD058klLo4CkYOnIybB5aV6WYuwarfPY4bF/dLggPem+ewLNTUNBuwrxj/A4nUv0LJTuRO8rR7f8WR9qnRCY0Ic5saQIDAQAB",
"icons": {
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "chrome-use",
"version": "1.5.19",
"version": "1.5.23",
"description": "chrome-use — drive your real, logged-in Chrome from any AI agent, stealth by default",
"type": "module",
"packageManager": "pnpm@11.1.3",
+25
View File
@@ -59,6 +59,18 @@ next ref interaction.
> anyway.) Driving off pixels on the relay also risks a coordinate event drifting
> onto the user's foreground tab — refs never do. See issue #37.
> **Two different intents — only one is discouraged.** The rule above is about
> *screenshot-to-locate* (using a picture to find/hit an element) — that's the bug.
> *screenshot-to-capture* — saving a region or element to a file as a **reusable
> image asset** (maps, charts, og-images, visual-diff baselines, report figures) —
> is fully supported and encouraged: `screenshot [selector] [--clip x,y,w,h] <file>`.
> Capturing a rendered map region to a PNG for a blog post is the right tool, not a
> smell. Screenshots are auto-downscaled to ≤2000px (longest edge) so they fit an
> image reader and their pixels line up with `click x y`; override with
> `--max-width`/`--max-height`/`--scale`. To click something you couldn't hit by
> ref, `box @ref` gives the element's CSS-px box + `centerX/centerY` to feed
> straight into `click <centerX> <centerY>` — no screenshot needed.
## Before you automate: pick the cheapest tool
Driving a browser is the heavy option. chrome-use earns its keep when you
@@ -218,6 +230,13 @@ chrome-use site github/issues owner/repo --json # run it → JSON (navigates t
- If no adapter fits, fall back to the normal `snapshot`/`eval` loop. Adapters come from the
[bb-sites](https://github.com/epiral/bb-sites) community pack; chrome-use fetches & runs them.
> **Auto-trigger — act on it.** chrome-use keeps the pack synced automatically (first use +
> weekly), and when you `open`/`navigate`/`snapshot` a page whose domain has adapters it tells
> you: a `💡 site adapters for <domain>` line on stderr, and a `siteAdapters: {domain, commands}`
> field in `--json`. **When you see that, prefer the listed `site <name>/<cmd>` over snapshot+click
> for reading data** — it's the cheaper, more reliable path and it's already installed. You don't
> need to run `site update` yourself; just use the command it names.
## Quickstart
```bash
@@ -366,6 +385,12 @@ foreground, so prefer refs. For below-the-fold content in such a frame, scroll i
with `scroll down N --at x,y` (a pixel over the frame) or `--frame n`. For a
postal/autocomplete box inside the frame, `type @e "…" --key-events`.
> **Caveat: `find text "…"` can't reach into a cross-origin iframe** — it errors
> "Element not found" even though `snapshot -i` lists those nodes and
> `get text` reads them. Inside cross-origin iframes, target elements by their
> **snapshot `@ref`**, not by `find`. (`box @ref` also works on iframe refs when
> you need a coordinate fallback.)
### When refs don't work or you don't want to snapshot
Use semantic locators: