Compare commits

...
4 Commits
Author SHA1 Message Date
leeguooooo 02e23ebe11 fix(build): include browser.rs clear_viewport/via_relay (#47) + cargo fmt
Release binaries / Build macOS ARM64 (push) Has been cancelled
Release binaries / Build macOS x64 (push) Has been cancelled
Release binaries / Build Linux ARM64 (push) Has been cancelled
Release binaries / Build Linux musl ARM64 (push) Has been cancelled
Release binaries / Build Linux musl x64 (push) Has been cancelled
Release binaries / Build Linux x64 (push) Has been cancelled
Release binaries / Build Windows x64 (push) Has been cancelled
Release binaries / Attach binaries to GitHub Release (push) Has been cancelled
v1.5.24 (d5cd9cd) shipped a commands.rs caller of `clear_viewport` but not the
browser.rs method it lives in (a concurrent in-progress #47 viewport/resize edit
was only partly staged), so main didn't compile and the format check failed.
Commit the matching browser.rs method + via_relay() helper and run cargo fmt.
Full tree builds; 863 tests pass.
2026-06-18 11:39:11 +09:00
leeguooooo d5cd9cd621 feat(canvas): extract WebGL/canvas-app content + fix site arg-order + adopt skill doc
canvas — chrome-use can now read canvas/WebGL apps (Figma, games, maps, charts,
drawing tools) that expose no DOM/refs:
  - canvas list: enumerate <canvas> (backing+CSS size, visibility, toDataUrl/tainted)
  - canvas capture [selector] [path]: save rendered pixels to PNG — toDataURL
    (full backing-store resolution), with a CDP screenshot fallback for WebGL
    without preserveDrawingBuffer or cross-origin-tainted canvases. --screenshot
    forces the screenshot path. Gets the RENDER, not hidden source data.
  Verified live: captured Figma's canvas at full 2522x1904 via toDataURL.

site — fix map_args losing the adapter's declared arg order: serde sorts @meta
keys alphabetically, so a 2-arg adapter like {projectId, path} mapped positionals
to {path, projectId} (swapped). Now parses declaration order from the raw @meta
text (Adapter.arg_order) + regression test. Affects any multi-arg adapter.

skill — core skill now documents `adopt <url|targetId>` (read a pre-existing tab,
the explicit way through strict isolation) and `canvas list`/`canvas capture` in
the canvas/WebGL section.

863 tests pass.
2026-06-18 11:31:24 +09:00
leeguooooo 284a60a54c feat(adopt): read a pre-existing tab without opening a new one
Release binaries / Build macOS ARM64 (push) Has been cancelled
Release binaries / Build macOS x64 (push) Has been cancelled
Release binaries / Build Linux ARM64 (push) Has been cancelled
Release binaries / Build Linux musl ARM64 (push) Has been cancelled
Release binaries / Build Linux musl x64 (push) Has been cancelled
Release binaries / Build Linux x64 (push) Has been cancelled
Release binaries / Build Windows x64 (push) Has been cancelled
Release binaries / Attach binaries to GitHub Release (push) Has been cancelled
New `chrome-use adopt <url-substring|targetId>`: drive a tab the user (or
another session) already has open, with ZERO new tabs. After group-scoped
isolation (#40) a session can't see foreign tabs, so adopt adds an explicit,
opt-in path:

- Relay (relay.rs): `ABRelay.getAllTargets` returns every attached target
  UNSCOPED (ignores group scoping), so the agent can find a specific tab by URL
  or targetId. +1 unit test.
- Daemon (browser.rs): `collect_all_targets` (unscoped, falls back to scoped on
  older relays) + `adopt_existing_target` — matches by exact targetId or
  case-insensitive URL substring, attaches it (the relay re-tags it into the
  adopter's group, so isolation holds), pins it; never creates a tab. On no
  match it errors AND lists the open tabs it can see, rather than launching.
  discover_and_attach_targets honors AGENT_BROWSER_ADOPT at first connect, so no
  about:blank is ever created.
- CLI (main.rs): `adopt` sets the env, forces a fresh daemon, and rewrites into
  `connect <relay-url>` (like `extension connect`) so the daemon attaches to the
  user's real Chrome before parse_command.

Extension (ab-connect 0.4.11): `reannounceAttachedTabs` now re-sends each tab's
url/title (it previously sent neither) so the relay's target list stays matchable
by URL after the MV3 service worker reconnects — otherwise reannounced tabs show
a blank url and `adopt <url>` can't find them. Repacked upload zip + crx.

Mechanism verified live (enumerated all 11 of the user's open tabs incl. the
target). 862 tests pass.
2026-06-17 21:18:01 +09:00
leeguooooo 10d196b6eb chore(ext): pack ab-connect 0.4.10 upload zip + crx (#40 group-scoped relay)
Rebuilt extensions/ab-connect.zip (key stripped for the Web Store) and the
reference .crx from the 0.4.10 source (openerTargetId + abGroup in the
synthesized Target.attachedToTarget).
2026-06-17 18:17:20 +09:00
16 changed files with 757 additions and 77 deletions
+1 -1
View File
@@ -290,7 +290,7 @@ checksum = "613afe47fcd5fac7ccf1db93babcb082c5994d996f20b8b159f2ad1658eb5724"
[[package]]
name = "chrome-use"
version = "1.5.22"
version = "1.5.24"
dependencies = [
"aes",
"aes-gcm",
+1 -1
View File
@@ -1,6 +1,6 @@
[package]
name = "chrome-use"
version = "1.5.22"
version = "1.5.24"
edition = "2021"
description = "Fast browser automation CLI for AI agents"
license = "Apache-2.0"
+214 -33
View File
@@ -80,6 +80,10 @@ const KNOWN_COMMANDS: &[&str] = &[
"upload",
"site",
"box",
"adopt",
"canvas",
"viewport",
"resize",
];
/// Levenshtein distance, capped — small inputs only (command names).
@@ -1172,6 +1176,54 @@ fn parse_command_inner(args: &[String], flags: &Flags) -> Result<Value, ParseErr
}
Ok(cmd)
}
// Canvas/WebGL apps (Figma, games, maps, charts, drawing tools) render to a
// <canvas> with no DOM/refs to read. `canvas` extracts what's actually
// rendered: `list` enumerates canvases; `capture` saves a canvas to PNG via
// toDataURL (full backing-store resolution) with a CDP-screenshot fallback
// for WebGL contexts (no preserveDrawingBuffer) or cross-origin-tainted ones.
"canvas" => {
match rest.first().copied() {
Some("list") => Ok(json!({ "id": id, "action": "canvas_list" })),
Some("capture") => {
let force_screenshot = rest.contains(&"--screenshot");
let pos: Vec<&str> = rest[1..]
.iter()
.copied()
.filter(|a| !a.starts_with("--"))
.collect();
// `capture [selector] [path]`: a selector starts with . # @ or is
// a tag; a path contains / or ends in an image extension.
let is_path = |s: &str| {
s.contains('/')
|| s.ends_with(".png")
|| s.ends_with(".jpg")
|| s.ends_with(".jpeg")
|| s.ends_with(".webp")
};
let (selector, path) = match (pos.first(), pos.get(1)) {
(Some(a), Some(b)) => (Some(*a), Some(*b)),
(Some(a), None) if is_path(a) => (None, Some(*a)),
(Some(a), None) => (Some(*a), None),
_ => (None, None),
};
let mut cmd = json!({ "id": id, "action": "canvas_capture" });
if let Some(s) = selector {
cmd["selector"] = json!(s);
}
if let Some(p) = path {
cmd["path"] = json!(p);
}
if force_screenshot {
cmd["forceScreenshot"] = json!(true);
}
Ok(cmd)
}
_ => Err(ParseError::InvalidValue {
message: "canvas needs a subcommand".to_string(),
usage: "canvas list | canvas capture [selector] [path] [--screenshot]",
}),
}
}
"pdf" => {
let path = rest.first().ok_or_else(|| ParseError::MissingArguments {
context: "pdf".to_string(),
@@ -1317,6 +1369,12 @@ fn parse_command_inner(args: &[String], flags: &Flags) -> Result<Value, ParseErr
Ok(json!({ "id": id, "action": "site", "domain": domain, "script": script }))
}
// `adopt <url|targetId>`: the adoption happens at daemon connect (driven by
// the AGENT_BROWSER_ADOPT env main.rs set + a forced-fresh daemon), so by
// the time this command runs the tab is already attached. Resolve to a
// `url` read so the response confirms which tab got adopted.
"adopt" => Ok(json!({ "id": id, "action": "url" })),
// === Stealth self-check ===
"stealth" => {
// `stealth [status]` — local stealth self-check: mode, live probes
@@ -1640,6 +1698,13 @@ fn parse_command_inner(args: &[String], flags: &Flags) -> Result<Value, ParseErr
// === Mouse ===
"mouse" => parse_mouse(&rest, &id),
// === Viewport / window size ===
// Top-level shortcut for `set viewport` — agents (and the author, in
// issue #47) reach for `viewport`/`resize` first. Uses a CDP virtual
// viewport, so it also works on the extension relay without yanking the
// user's real window around.
"viewport" | "resize" => parse_viewport(&rest, &id),
// === Set (browser settings) ===
"set" => parse_set(&rest, &id),
@@ -3088,6 +3153,97 @@ fn parse_mouse(rest: &[&str], id: &str) -> Result<Value, ParseError> {
}
}
/// Parse a viewport / resize spec, shared by the top-level `viewport` / `resize`
/// commands and `set viewport`. Sets a CDP device-metrics override
/// (`Emulation.setDeviceMetricsOverride`) — a *virtual* viewport for the tab, so
/// it works headless AND on the extension relay without physically resizing the
/// user's real Chrome window (issue #47). Forms:
/// viewport <width> <height> [scale] [--dpr N] [--mobile]
/// viewport <width>x<height> (e.g. 1280x800)
/// viewport reset | clear (drop the override, restore real size)
fn parse_viewport(rest: &[&str], id: &str) -> Result<Value, ParseError> {
const USAGE: &str = "viewport <width> <height> [scale] [--dpr N] [--mobile] | viewport reset";
if matches!(
rest.first().copied(),
Some("reset") | Some("clear") | Some("off")
) {
return Ok(json!({ "id": id, "action": "viewport", "reset": true }));
}
// Positional (non-flag) tokens. A `WxH` token counts as one positional.
let positionals: Vec<&str> = rest
.iter()
.copied()
.filter(|a| !a.starts_with("--"))
.collect();
let (w, h, scale_tok): (i32, i32, Option<&str>) = match positionals.first() {
Some(first) if first.contains('x') || first.contains('X') => {
let mut parts = first.split(|c| c == 'x' || c == 'X');
let w = parts.next().and_then(|s| s.parse::<i32>().ok());
let h = parts.next().and_then(|s| s.parse::<i32>().ok());
match (w, h) {
(Some(w), Some(h)) => (w, h, positionals.get(1).copied()),
_ => {
return Err(ParseError::InvalidValue {
message: format!("Invalid viewport size: {}", first),
usage: USAGE,
})
}
}
}
Some(w_str) => {
let h_str = positionals.get(1).ok_or(ParseError::MissingArguments {
context: "viewport".to_string(),
usage: USAGE,
})?;
let w = w_str.parse::<i32>().map_err(|_| ParseError::InvalidValue {
message: format!("Invalid width: {}", w_str),
usage: USAGE,
})?;
let h = h_str.parse::<i32>().map_err(|_| ParseError::InvalidValue {
message: format!("Invalid height: {}", h_str),
usage: USAGE,
})?;
(w, h, positionals.get(2).copied())
}
None => {
return Err(ParseError::MissingArguments {
context: "viewport".to_string(),
usage: USAGE,
})
}
};
let mut cmd = json!({ "id": id, "action": "viewport", "width": w, "height": h });
// Device-scale-factor: positional, overridden by --dpr / --scale.
let mut scale: Option<f64> = match scale_tok {
Some(s) => Some(s.parse::<f64>().map_err(|_| ParseError::InvalidValue {
message: format!("Invalid scale: {}", s),
usage: USAGE,
})?),
None => None,
};
if let Some(i) = rest.iter().position(|a| *a == "--dpr" || *a == "--scale") {
let v = rest.get(i + 1).and_then(|s| s.parse::<f64>().ok()).ok_or(
ParseError::InvalidValue {
message: "--dpr/--scale needs a number".to_string(),
usage: USAGE,
},
)?;
scale = Some(v);
}
if let Some(s) = scale {
cmd["deviceScaleFactor"] = json!(s);
}
if rest.iter().any(|a| *a == "--mobile") {
cmd["mobile"] = json!(true);
}
Ok(cmd)
}
fn parse_set(rest: &[&str], id: &str) -> Result<Value, ParseError> {
const VALID: &[&str] = &[
"viewport",
@@ -3102,39 +3258,8 @@ fn parse_set(rest: &[&str], id: &str) -> Result<Value, ParseError> {
];
match rest.first().copied() {
Some("viewport") => {
let w_str = rest.get(1).ok_or_else(|| ParseError::MissingArguments {
context: "set viewport".to_string(),
usage: "set viewport <width> <height> [scale]",
})?;
let h_str = rest.get(2).ok_or_else(|| ParseError::MissingArguments {
context: "set viewport".to_string(),
usage: "set viewport <width> <height> [scale]",
})?;
let w = w_str
.parse::<i32>()
.map_err(|_| ParseError::MissingArguments {
context: "set viewport".to_string(),
usage: "set viewport <width> <height> [scale]",
})?;
let h = h_str
.parse::<i32>()
.map_err(|_| ParseError::MissingArguments {
context: "set viewport".to_string(),
usage: "set viewport <width> <height> [scale]",
})?;
let mut cmd = json!({ "id": id, "action": "viewport", "width": w, "height": h });
if let Some(scale_str) = rest.get(3) {
let scale = scale_str
.parse::<f64>()
.map_err(|_| ParseError::MissingArguments {
context: "set viewport".to_string(),
usage: "set viewport <width> <height> [scale]",
})?;
cmd["deviceScaleFactor"] = json!(scale);
}
Ok(cmd)
}
// `set viewport ...` is an alias for the top-level `viewport` command.
Some("viewport") => parse_viewport(&rest[1..], id),
Some("device") => {
let dev = rest.get(1).ok_or_else(|| ParseError::MissingArguments {
context: "set device".to_string(),
@@ -5295,6 +5420,62 @@ mod tests {
assert!(result.is_err());
}
#[test]
fn test_viewport_toplevel() {
let cmd = parse_command(&args("viewport 1280 800"), &default_flags()).unwrap();
assert_eq!(cmd["action"], "viewport");
assert_eq!(cmd["width"], 1280);
assert_eq!(cmd["height"], 800);
assert!(cmd.get("deviceScaleFactor").is_none());
assert!(cmd.get("mobile").is_none());
}
#[test]
fn test_resize_alias() {
let cmd = parse_command(&args("resize 700 800"), &default_flags()).unwrap();
assert_eq!(cmd["action"], "viewport");
assert_eq!(cmd["width"], 700);
assert_eq!(cmd["height"], 800);
}
#[test]
fn test_viewport_wxh_form() {
let cmd = parse_command(&args("viewport 375x812"), &default_flags()).unwrap();
assert_eq!(cmd["action"], "viewport");
assert_eq!(cmd["width"], 375);
assert_eq!(cmd["height"], 812);
}
#[test]
fn test_viewport_dpr_and_mobile_flags() {
let cmd =
parse_command(&args("viewport 375 812 --dpr 3 --mobile"), &default_flags()).unwrap();
assert_eq!(cmd["action"], "viewport");
assert_eq!(cmd["width"], 375);
assert_eq!(cmd["height"], 812);
assert_eq!(cmd["deviceScaleFactor"], 3.0);
assert_eq!(cmd["mobile"], true);
}
#[test]
fn test_viewport_reset() {
for spec in ["viewport reset", "viewport clear", "resize reset"] {
let cmd = parse_command(&args(spec), &default_flags()).unwrap();
assert_eq!(cmd["action"], "viewport", "{spec}");
assert_eq!(cmd["reset"], true, "{spec}");
}
}
#[test]
fn test_viewport_missing_height() {
assert!(parse_command(&args("viewport 1280"), &default_flags()).is_err());
}
#[test]
fn test_viewport_invalid_width() {
assert!(parse_command(&args("viewport abc 800"), &default_flags()).is_err());
}
#[test]
fn test_find_first_no_value() {
let cmd = parse_command(&args("find first a click"), &default_flags()).unwrap();
+1 -1
View File
@@ -595,7 +595,7 @@ fn query_current_url(session: &str) -> Option<String> {
}
/// Kill a running daemon by reading its PID file and sending a kill signal.
fn kill_stale_daemon(session: &str) {
pub fn kill_stale_daemon(session: &str) {
// Remove the socket first so no new connections reach the old daemon
#[cfg(unix)]
{
+37
View File
@@ -981,6 +981,43 @@ fn main() {
}
}
// `adopt <url|targetId>`: read a PRE-EXISTING tab (the user's own, or another
// session's) WITHOUT opening a new one. Forces a fresh daemon and points it at
// the relay (like `extension connect`); the AGENT_BROWSER_ADOPT env makes the
// daemon's first connect ADOPT the matching tab instead of creating an
// about:blank. Rewrites into `connect <relay-url>` BEFORE parse_command so the
// daemon attaches to the user's real Chrome. Must run before parse_command.
if clean.first().map(|s| s.as_str()) == Some("adopt") {
match clean.get(1) {
Some(spec) if !spec.trim().is_empty() => {
std::env::set_var("AGENT_BROWSER_ADOPT", spec.trim());
connection::kill_stale_daemon(&flags.session);
match connect::relay_url() {
Some(url) => {
flags.cdp = Some(url.clone());
flags.auto_connect = false;
clean = vec!["connect".to_string(), url];
}
None => {
eprintln!(
"{} extension relay not connected — open Chrome with the ab-connect \
extension first (this command reads an EXISTING tab, it won't launch one).",
color::error_indicator()
);
exit(1);
}
}
}
_ => {
eprintln!(
"{} usage: chrome-use adopt <url-substring|targetId> (reads an existing tab, no new tab)",
color::error_indicator()
);
exit(2);
}
}
}
// Handle session separately (doesn't need daemon)
if clean.first().map(|s| s.as_str()) == Some("session") {
run_session(&clean, &flags.session, flags.json);
+163
View File
@@ -1320,6 +1320,8 @@ pub async fn execute_command(cmd: &Value, state: &mut DaemonState) -> Value {
"stealth_status" => handle_stealth_status(state).await,
"snapshot" => handle_snapshot(cmd, state).await,
"screenshot" => handle_screenshot(cmd, state).await,
"canvas_list" => handle_canvas_list(state).await,
"canvas_capture" => handle_canvas_capture(cmd, state).await,
"click" => handle_click(cmd, state).await,
"dblclick" => handle_dblclick(cmd, state).await,
"fill" => handle_fill(cmd, state).await,
@@ -3213,6 +3215,154 @@ fn downscale_screenshot(
Some((resized.width(), resized.height()))
}
/// `canvas list` — enumerate <canvas> elements (size, visibility, whether
/// toDataURL is usable) so an agent can pick one to capture on a canvas/WebGL app
/// where snapshot/DOM reads see nothing.
async fn handle_canvas_list(state: &DaemonState) -> Result<Value, String> {
let mgr = state.browser.as_ref().ok_or("Browser not launched")?;
let js = r#"(() => [...document.querySelectorAll('canvas')].map((c, i) => {
const r = c.getBoundingClientRect();
let toDataUrl = true, tainted = false;
try { c.toDataURL('image/png'); } catch (e) { toDataUrl = false; tainted = true; }
return {
index: i,
backingWidth: c.width, backingHeight: c.height,
cssWidth: Math.round(r.width), cssHeight: Math.round(r.height),
visible: r.width > 0 && r.height > 0 && r.bottom > 0 && r.top < innerHeight,
id: c.id || null, className: c.className || null,
toDataUrl, tainted,
};
}))()"#;
let canvases = mgr.evaluate(js, None).await?;
Ok(json!({ "canvases": canvases }))
}
/// `canvas capture [selector] [path]` — save a canvas's rendered pixels to PNG.
/// Prefers `toDataURL` (full backing-store resolution); falls back to a CDP
/// screenshot of the canvas element when toDataURL is blank (WebGL without
/// preserveDrawingBuffer, e.g. Figma) or throws (cross-origin tainted). This is
/// how chrome-use "sees" canvas/WebGL apps that expose no DOM or refs.
async fn handle_canvas_capture(cmd: &Value, state: &mut DaemonState) -> Result<Value, String> {
let selector = cmd
.get("selector")
.and_then(|v| v.as_str())
.map(String::from);
let force_screenshot = cmd
.get("forceScreenshot")
.and_then(|v| v.as_bool())
.unwrap_or(false);
let out_path = cmd.get("path").and_then(|v| v.as_str()).map(String::from);
let id = cmd.get("id").and_then(|v| v.as_str()).unwrap_or("0");
// Probe: locate the canvas, get its bbox, and try toDataURL (unless forced).
let probe = {
let mgr = state.browser.as_ref().ok_or("Browser not launched")?;
let sel_lit = match &selector {
Some(s) => serde_json::to_string(s).unwrap_or_else(|_| "null".to_string()),
None => "null".to_string(),
};
let js = format!(
r#"(() => {{
const sel = {sel_lit};
const c = sel ? document.querySelector(sel)
: [...document.querySelectorAll('canvas')].sort((a,b)=>(b.width*b.height)-(a.width*a.height))[0];
if (!c) return {{ found: false, count: document.querySelectorAll('canvas').length }};
const r = c.getBoundingClientRect();
let dataUrl = null, err = null;
if ({try_data}) {{ try {{ dataUrl = c.toDataURL('image/png'); }} catch (e) {{ err = String(e && e.message || e); }} }}
return {{ found: true, w: c.width, h: c.height, x: r.x, y: r.y, cw: r.width, ch: r.height, dataUrl, err }};
}})()"#,
try_data = !force_screenshot
);
mgr.evaluate(&js, None).await?
};
if !probe
.get("found")
.and_then(|v| v.as_bool())
.unwrap_or(false)
{
let count = probe.get("count").and_then(|v| v.as_u64()).unwrap_or(0);
return Err(format!(
"canvas: no canvas matched{} ({count} canvas element(s) on the page — try `canvas list`)",
selector
.as_deref()
.map(|s| format!(" `{s}`"))
.unwrap_or_default()
));
}
let backing_w = probe.get("w").and_then(|v| v.as_u64()).unwrap_or(0);
let backing_h = probe.get("h").and_then(|v| v.as_u64()).unwrap_or(0);
// Decode toDataURL if present; a WebGL canvas without preserveDrawingBuffer
// returns a blank PNG (tiny when compressed), so reject suspiciously small
// results and fall back to the screenshot path.
let decoded: Option<Vec<u8>> = probe
.get("dataUrl")
.and_then(|v| v.as_str())
.and_then(|u| u.split_once(",").map(|(_, b)| b.to_string()))
.and_then(|b64| {
base64::Engine::decode(&base64::engine::general_purpose::STANDARD, b64.as_bytes()).ok()
});
let use_data = !force_screenshot && decoded.as_ref().map(|d| d.len() > 1024).unwrap_or(false);
if use_data {
let bytes = decoded.unwrap();
let path = out_path.unwrap_or_else(|| {
std::env::temp_dir()
.join(format!("canvas-{id}.png"))
.to_string_lossy()
.into_owned()
});
std::fs::write(&path, &bytes).map_err(|e| format!("canvas: write {path}: {e}"))?;
Ok(json!({
"path": absolutize_saved_path(&path),
"method": "toDataURL",
"width": backing_w,
"height": backing_h,
}))
} else {
// Fallback: screenshot the canvas element (or its bbox clip).
let clip = if selector.is_none() {
let x = probe.get("x").and_then(|v| v.as_f64()).unwrap_or(0.0);
let y = probe.get("y").and_then(|v| v.as_f64()).unwrap_or(0.0);
let cw = probe.get("cw").and_then(|v| v.as_f64()).unwrap_or(0.0);
let ch = probe.get("ch").and_then(|v| v.as_f64()).unwrap_or(0.0);
Some((x, y, cw, ch))
} else {
None
};
let options = ScreenshotOptions {
selector: selector.clone(),
path: out_path,
clip,
..ScreenshotOptions::default()
};
let mgr = state.browser.as_ref().ok_or("Browser not launched")?;
let session_id = mgr.active_session_id()?.to_string();
let result = screenshot::take_screenshot(
&mgr.client,
&session_id,
&state.ref_map,
&options,
&state.iframe_sessions,
)
.await?;
let why = probe
.get("err")
.and_then(|v| v.as_str())
.map(|e| format!("toDataURL failed ({e})"))
.unwrap_or_else(|| {
"toDataURL blank/unavailable (WebGL no preserveDrawingBuffer)".into()
});
Ok(json!({
"path": absolutize_saved_path(&result.path),
"method": "screenshot",
"note": format!("captured rendered pixels via screenshot — {why}"),
}))
}
}
async fn handle_click(cmd: &Value, state: &mut DaemonState) -> Result<Value, String> {
// First-class coordinate click (issue #8.4): click a raw viewport point with
// no element resolution. Parsed from `click <x> <y>` / `click --coords x,y`.
@@ -5102,6 +5252,19 @@ async fn handle_tab_close(cmd: &Value, state: &mut DaemonState) -> Result<Value,
async fn handle_viewport(cmd: &Value, state: &mut DaemonState) -> Result<Value, String> {
let mgr = state.browser.as_ref().ok_or("Browser not launched")?;
// `viewport reset` clears the device-metrics override and restores the real
// layout viewport (the launched window's size, or — over the relay — the
// user's actual Chrome window).
if cmd.get("reset").and_then(|v| v.as_bool()).unwrap_or(false) {
mgr.clear_viewport().await?;
state.viewport = None;
if let Some(ref server) = state.stream_server {
server.set_viewport(1280, 720).await;
}
return Ok(json!({ "reset": true }));
}
let width = cmd.get("width").and_then(|v| v.as_i64()).unwrap_or(1280) as i32;
let height = cmd.get("height").and_then(|v| v.as_i64()).unwrap_or(720) as i32;
let scale = cmd
+170 -28
View File
@@ -824,6 +824,106 @@ impl BrowserManager {
Ok(by_id.into_values().collect())
}
/// Every tab the relay knows, UNSCOPED (ignores group scoping) — for explicit
/// cross-group adoption (`chrome-use adopt`). Falls back to the scoped
/// `collect_page_targets` on a relay/browser that doesn't support the
/// unscoped query. Retries a few times over the relay (discovery is eventual).
async fn collect_all_targets(&self) -> Result<Vec<TargetInfo>, String> {
let rounds = if crate::connect::relay_url().is_some() {
3
} else {
1
};
let mut by_id: HashMap<String, TargetInfo> = HashMap::new();
let mut any_ok = false;
for i in 0..rounds {
if i > 0 {
tokio::time::sleep(Duration::from_millis(150)).await;
}
if let Ok(result) = self
.client
.send_command_typed::<_, GetTargetsResult>(
"ABRelay.getAllTargets",
&json!({}),
None,
)
.await
{
any_ok = true;
for t in result.target_infos.into_iter().filter(should_track_target) {
by_id.entry(t.target_id.clone()).or_insert(t);
}
}
}
if any_ok {
Ok(by_id.into_values().collect())
} else {
// Older relay without ABRelay.getAllTargets → best-effort scoped list.
self.collect_page_targets().await
}
}
/// Adopt a specific pre-existing tab matched by `spec` (an exact CDP
/// `targetId`, or a case-insensitive substring of the tab URL) WITHOUT opening
/// a new tab — for `chrome-use adopt`. Attaches it (the relay tags it into our
/// group), tracks + pins it. Errors if nothing matches (never creates a tab).
async fn adopt_existing_target(&mut self, spec: &str) -> Result<(), String> {
let all = self.collect_all_targets().await?;
let spec_l = spec.to_lowercase();
let target = all
.iter()
.find(|t| t.target_id == spec)
.or_else(|| all.iter().find(|t| t.url.to_lowercase().contains(&spec_l)))
.ok_or_else(|| {
let mut open: Vec<String> = all
.iter()
.map(|t| {
let u = if t.url.len() > 80 {
&t.url[..80]
} else {
&t.url
};
u.to_string()
})
.collect();
open.sort();
open.dedup();
format!(
"adopt: no open tab matching `{spec}` (by targetId or URL substring).\n\
{} tab(s) the extension can see:\n {}",
open.len(),
open.join("\n ")
)
})?
.clone();
let attach: AttachToTargetResult = self
.client
.send_command_typed(
"Target.attachToTarget",
&AttachToTargetParams {
target_id: target.target_id.clone(),
flatten: true,
},
None,
)
.await?;
let tab_id = self.assign_tab_id();
self.pages.push(PageInfo {
tab_id,
label: None,
target_id: target.target_id.clone(),
session_id: attach.session_id.clone(),
url: target.url.clone(),
title: sanitize_title(&target.title),
target_type: target.target_type.clone(),
});
self.active_page_index = self.pages.len() - 1;
self.pin_active_target();
self.enable_domains(&attach.session_id).await?;
Ok(())
}
async fn discover_and_attach_targets(&mut self) -> Result<(), String> {
self.client
.send_command_typed::<_, Value>(
@@ -837,6 +937,17 @@ impl BrowserManager {
// own tab group (issue #40). On a launched browser this is a no-op.
let scoped = self.announce_group().await;
// `chrome-use adopt <spec>`: adopt a specific PRE-EXISTING tab instead of
// creating one — true zero-new-tab reading of the user's own tab. The
// directive rides in via env so it takes effect at first connect (before
// any about:blank would be made). If nothing matches, error out rather
// than fall back to creating a tab.
if let Ok(spec) = std::env::var("AGENT_BROWSER_ADOPT") {
if !spec.trim().is_empty() {
return self.adopt_existing_target(spec.trim()).await;
}
}
let page_targets: Vec<TargetInfo> = self.collect_page_targets().await?;
if page_targets.is_empty() {
@@ -1839,9 +1950,17 @@ impl BrowserManager {
/// CDP browser the endpoint is strict, so we must NOT send the custom param —
/// hence `None` there. We detect the relay by matching our `ws_url` against
/// the live relay URL the native-messaging host published.
/// Whether this manager is driving the user's real Chrome through the
/// `ab-connect` extension relay (vs. a browser we launched or a direct CDP
/// endpoint). Detected by matching our `ws_url` against the live relay URL
/// the native-messaging host published. Used to avoid relay-unsafe CDP that
/// would disturb the user's window (e.g. Browser.setContentsSize, issue #47).
fn via_relay(&self) -> bool {
crate::connect::relay_url().as_deref() == Some(self.ws_url.as_str())
}
fn agent_group(&self) -> Option<String> {
let via_relay = crate::connect::relay_url().as_deref() == Some(self.ws_url.as_str());
if !via_relay {
if !self.via_relay() {
return None;
}
let name = DAEMON_SESSION
@@ -2050,32 +2169,39 @@ impl BrowserManager {
.await?;
// Screencast captures the actual content area, not the emulated CSS
// viewport, so resize the content area to match.
if let Ok(target_id) = self.active_target_id() {
if let Ok(window_info) = self
.client
.send_command(
"Browser.getWindowForTarget",
Some(json!({ "targetId": target_id })),
None,
)
.await
{
if let Some(window_id) = window_info.get("windowId").and_then(|v| v.as_i64()) {
if let Err(e) = self
.client
.send_command(
"Browser.setContentsSize",
Some(json!({
"windowId": window_id,
"width": width,
"height": height,
})),
None,
)
.await
{
eprintln!("Browser.setContentsSize failed (experimental CDP): {e}");
// viewport, so resize the content area to match — but ONLY for a browser
// we launched. Over the ab-connect relay the "window" is the user's real
// Chrome window, and Browser.setContentsSize would physically resize it
// (issue #47) — the exact thing the CDP device-metrics override exists to
// avoid. The Emulation override above already gives the tab the requested
// CSS viewport without touching the OS window, so skip the resize there.
if !self.via_relay() {
if let Ok(target_id) = self.active_target_id() {
if let Ok(window_info) = self
.client
.send_command(
"Browser.getWindowForTarget",
Some(json!({ "targetId": target_id })),
None,
)
.await
{
if let Some(window_id) = window_info.get("windowId").and_then(|v| v.as_i64()) {
if let Err(e) = self
.client
.send_command(
"Browser.setContentsSize",
Some(json!({
"windowId": window_id,
"width": width,
"height": height,
})),
None,
)
.await
{
eprintln!("Browser.setContentsSize failed (experimental CDP): {e}");
}
}
}
}
@@ -2084,6 +2210,22 @@ impl BrowserManager {
Ok(())
}
/// Clear the CDP device-metrics override (`viewport reset`), restoring the
/// tab's real layout viewport. Never touches the OS window, so it is safe on
/// the relay (we never physically resized the user's window — see
/// `set_viewport`).
pub async fn clear_viewport(&self) -> Result<(), String> {
let session_id = self.active_session_id()?;
self.client
.send_command(
"Emulation.clearDeviceMetricsOverride",
Some(json!({})),
Some(session_id),
)
.await?;
Ok(())
}
pub async fn set_user_agent(&self, user_agent: &str) -> Result<(), String> {
let session_id = self.active_session_id()?;
self.client
+41
View File
@@ -155,6 +155,20 @@ impl RelayState {
"Target.setDiscoverTargets" | "Target.setAutoAttach" => {
ClientRoute::Local(json!({ "id": id, "result": {} }))
}
// Unscoped discovery for EXPLICIT cross-group adoption (`chrome-use
// adopt`): returns every target the extension has attached, ignoring
// group scoping, so an agent can find a specific pre-existing tab (the
// user's, another session's) by URL/targetId and adopt it. Isolation
// is preserved because the daemon only acts on the one tab it then
// attaches (which the relay re-tags into the adopter's group).
"ABRelay.getAllTargets" => {
let infos: Vec<Value> = self
.targets
.values()
.map(|t| t.target_info.clone())
.collect();
ClientRoute::Local(json!({ "id": id, "result": { "targetInfos": infos } }))
}
"Target.getTargets" => {
// Scope to the client's own group when it announced one; an
// un-announced (legacy) client gets the full list (back-compat).
@@ -772,6 +786,33 @@ mod tests {
assert!(get_target_ids(&mut s, 2).is_empty());
}
#[test]
fn get_all_targets_is_unscoped() {
let mut s = RelayState::new();
create_in_group(&mut s, 1, "agent-a", "ta", "sa");
create_in_group(&mut s, 2, "agent-b", "tb", "sb");
// Client 1's scoped getTargets sees only its own group...
assert_eq!(get_target_ids(&mut s, 1), vec!["ta"]);
// ...but ABRelay.getAllTargets returns EVERY target regardless of group
// (for explicit cross-group adoption).
let all = match s
.route_client_command(1, &json!({ "id": 1, "method": "ABRelay.getAllTargets" }))
{
ClientRoute::Local(v) => {
let mut ids: Vec<String> = v["result"]["targetInfos"]
.as_array()
.unwrap()
.iter()
.map(|t| t["targetId"].as_str().unwrap().to_string())
.collect();
ids.sort();
ids
}
_ => panic!("getAllTargets must be local"),
};
assert_eq!(all, vec!["ta", "tb"]);
}
#[test]
fn detach_clears_target_group() {
let mut s = RelayState::new();
+8
View File
@@ -3319,6 +3319,10 @@ Core Commands:
screenshot [path] Take screenshot (auto-downscaled to 2000px long edge;
--max-width/--max-height/--scale to override)
pdf <path> Save as PDF
canvas list List <canvas> elements (size, type) on the page
canvas capture [sel] [path] Save a canvas's rendered pixels to PNG for
WebGL/canvas apps (Figma, games, maps, charts) that
expose no DOM. toDataURL, with a screenshot fallback.
snapshot Accessibility tree with refs (for AI)
eval <js> Run JavaScript
connect <port|url> Connect to browser via CDP
@@ -3375,6 +3379,10 @@ Tabs:
stable targetId, no reload preserves in-page state
open <url> --reuse-tab Reuse an existing tab on that URL instead of spawning
a duplicate (matches origin+path; preserves state)
adopt <url|targetId> Read a PRE-EXISTING tab (the user's own, or another
session's) WITHOUT opening a new one matches by URL
substring or stable targetId, then drives it. e.g.
`adopt "github.com/owner/repo"`
Diff:
diff snapshot Compare current vs last snapshot
+88 -8
View File
@@ -31,6 +31,10 @@ fn dirs_home() -> Option<PathBuf> {
pub struct Adapter {
pub meta: Value,
pub func_src: String,
/// The adapter's declared `args` keys in DECLARATION order. Parsed from the
/// raw @meta text because `serde_json` sorts object keys alphabetically, which
/// would otherwise scramble positional-arg mapping for multi-arg adapters.
pub arg_order: Vec<String>,
}
impl Adapter {
@@ -111,7 +115,66 @@ pub fn parse_adapter(raw: &str, spec: &str) -> Result<Adapter, String> {
if func_src.is_empty() {
return Err(format!("site: {spec} has no function body after @meta"));
}
Ok(Adapter { meta, func_src })
let arg_order = arg_order_from_meta(&raw[start..end]);
Ok(Adapter {
meta,
func_src,
arg_order,
})
}
/// Extract the `args` object's keys in DECLARATION order from the raw @meta JSON
/// text (serde sorts them, losing order). Brace/string-aware: finds the `"args"`
/// value object and collects only its top-level keys.
fn arg_order_from_meta(meta_json: &str) -> Vec<String> {
let bytes = meta_json.as_bytes();
// Locate the `"args"` key, then the `{` that opens its value object.
let Some(args_pos) = meta_json.find("\"args\"") else {
return Vec::new();
};
let Some(brace_off) = meta_json[args_pos..].find('{') else {
return Vec::new();
};
let open = args_pos + brace_off;
let mut keys = Vec::new();
let mut depth = 0i32;
let mut in_str = false;
let mut esc = false;
let mut cur = String::new();
let mut last_str: Option<String> = None;
for &b in bytes.iter().skip(open) {
if in_str {
if esc {
esc = false;
} else if b == b'\\' {
esc = true;
} else if b == b'"' {
in_str = false;
last_str = Some(std::mem::take(&mut cur));
} else {
cur.push(b as char);
}
continue;
}
match b {
b'"' => in_str = true,
b'{' => depth += 1,
b'}' => {
depth -= 1;
if depth == 0 {
break; // end of the args object
}
}
// A `:` at depth 1 means the preceding string was a key of `args`.
b':' if depth == 1 => {
if let Some(k) = last_str.take() {
keys.push(k);
}
}
_ => {}
}
}
keys
}
/// Build the JS to eval: `(<adapter function>)(<args JSON>)`. The adapter's
@@ -325,14 +388,11 @@ pub fn adapters_for_domain(host: &str) -> Vec<String> {
/// validates required args itself.
pub fn map_args(adapter: &Adapter, positional: &[String], named: &[(String, String)]) -> Value {
let mut obj = serde_json::Map::new();
let keys: Vec<String> = adapter
.meta
.get("args")
.and_then(|a| a.as_object())
.map(|m| m.keys().cloned().collect())
.unwrap_or_default();
// Positional args fill the adapter's declared args in DECLARATION order
// (`arg_order`), not serde's alphabetized key order — otherwise a 2-arg
// adapter like `{projectId, path}` would map positionals to `{path, projectId}`.
for (i, val) in positional.iter().enumerate() {
if let Some(k) = keys.get(i) {
if let Some(k) = adapter.arg_order.get(i) {
obj.insert(k.clone(), Value::String(val.clone()));
}
}
@@ -382,4 +442,24 @@ async function(args) { return { repo: args.repo }; }"#;
assert!(load_adapter("noslash").is_err());
assert!(load_adapter("../etc/passwd").is_err());
}
// Regression: positional args must follow DECLARATION order, not serde's
// alphabetical key order. With `{projectId, path}` (not alphabetical),
// `<uuid> <file>` must map projectId←uuid, path←file — not swapped.
#[test]
fn positional_args_follow_declaration_order_not_alphabetical() {
let raw = r#"/* @meta
{
"name": "claude-design/get-file",
"domain": "claude.ai",
"args": { "projectId": {"required": true}, "path": {"required": true} }
}
*/
async function(args){ return args; }"#;
let a = parse_adapter(raw, "claude-design/get-file").unwrap();
assert_eq!(a.arg_order, vec!["projectId", "path"]);
let args = map_args(&a, &["the-uuid".into(), "misonote.dc.html".into()], &[]);
assert_eq!(args["projectId"], "the-uuid");
assert_eq!(args["path"], "misonote.dc.html");
}
}
Binary file not shown.
Binary file not shown.
+13 -2
View File
@@ -471,8 +471,19 @@ async function reannounceAttachedTabs() {
for (const [tabId, entry] of tabs.entries()) {
// Re-send the group hint too (issue #40) so the relay can rebuild its
// targetId→group map after its own restart (createTarget tagging won't
// re-run for tabs that are already open).
// re-run for tabs that are already open). Include the live url/title so the
// relay's target list stays matchable by URL after a reconnect (otherwise a
// reannounced tab shows a blank url and `adopt <url>` can't find it).
const { openerTargetId, abGroup } = await tabScopeHints(tabId)
let url = ''
let title = ''
try {
const t = await chrome.tabs.get(tabId)
if (t) {
url = t.url || t.pendingUrl || ''
title = t.title || ''
}
} catch {}
postToHost({
method: 'forwardCDPEvent',
params: {
@@ -480,7 +491,7 @@ async function reannounceAttachedTabs() {
method: 'Target.attachedToTarget',
params: {
sessionId: entry.sessionId,
targetInfo: { targetId: entry.targetId, type: 'page', attached: true, openerTargetId, abGroup },
targetInfo: { targetId: entry.targetId, type: 'page', url, title, attached: true, openerTargetId, abGroup },
},
},
})
+1 -1
View File
@@ -1,7 +1,7 @@
{
"manifest_version": 3,
"name": "chrome-use",
"version": "0.4.10",
"version": "0.4.11",
"description": "Let chrome-use drive your logged-in Chrome \u2014 install once, no token, no per-use confirmation.",
"key": "MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA6vQIyscGIPYPZdSpPwPL0+0gxUROyRgCpmvCSDoc8XUm4qm97VbKnD9Ijc1lV22lNWZtE78gaRjt6BeSfuMgnBymnhLKjN1gU6AI5QUU0mrJyeHdWKvrKQR5FmsM2A7Xr1ykE2SiiS8zNUS3Y/6O5l+Nva7wrVy6E4a2dkBVQkOsu+DV+nEZvhIyuDY5D5SPXqNwUTWTaglwj5mjvHz36xSwCWlPmrtJ+ED0AUyrb2z4GIOmvk4kqtBVrh/UD058klLo4CkYOnIybB5aV6WYuwarfPY4bF/dLggPem+ewLNTUNBuwrxj/A4nUv0LJTuRO8rR7f8WR9qnRCY0Ic5saQIDAQAB",
"icons": {
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "chrome-use",
"version": "1.5.22",
"version": "1.5.24",
"description": "chrome-use — drive your real, logged-in Chrome from any AI agent, stealth by default",
"type": "module",
"packageManager": "pnpm@11.1.3",
+18 -1
View File
@@ -154,7 +154,17 @@ real Chrome concurrently without ever dropping or stealing each other's tabs —
another agent's tab churn can't make your bound tab vanish or drift your commands
onto the wrong page. Consequence: `tab list` shows only *your* session's tabs; to
drive a specific page, navigate to it in your own tab instead of expecting a
pre-existing or popped-up tab to appear in the list. **Anti-detection ranking: this real logged-in Chrome (extension
pre-existing or popped-up tab to appear in the list.
> **Need to read a tab the user already has open?** Use `chrome-use adopt
> <url-substring|targetId>` — it finds that pre-existing tab (the user's own, or
> another session's) across groups and drives it **without opening a new tab**.
> e.g. `adopt "claude.ai/design"` then `snapshot`/`eval`/`get text` on it. On no
> match it errors and lists the tabs it can see. This is the explicit, opt-in way
> through the isolation above (it tags the adopted tab into your group). Great for
> "read/extract from the page I'm looking at" without disturbing it.
**Anti-detection ranking: this real logged-in Chrome (extension
connect) > a headed launched browser > headless (forbidden).** A genuine human
browser has no headless/automation tells at all, so prefer it for anything
anti-bot-sensitive.
@@ -444,6 +454,13 @@ tree**, so `snapshot` comes back near-empty and refs are a dead end. `snapshot`
detects this and prints a one-line hint. Drive them the screenshot way:
```bash
chrome-use canvas list # enumerate <canvas> elements (size, type)
chrome-use canvas capture out.png # save the canvas's RENDERED pixels to PNG —
# toDataURL (full backing-store res, e.g.
# Figma 2522x1904), screenshot fallback for
# WebGL w/o preserveDrawingBuffer / tainted.
# Gets the RENDER, not hidden source data
# (those live in the app's binary store/API).
chrome-use screenshot /tmp/s.png # SEE the state (your only read path —
# eval/get text return nothing useful)
chrome-use click 640 360 # interact by viewport coordinate