Compare commits

...
Author SHA1 Message Date
leeguooooo 9efcb56651 chore(release): bump to 0.27.0-fork.18 — extension connect (zero-token native-messaging control of real Chrome) + click reliability + eval-first/find-url/site-notes
Release binaries / Build macOS ARM64 (push) Has been cancelled
Release binaries / Build macOS x64 (push) Has been cancelled
Release binaries / Build Linux ARM64 (push) Has been cancelled
Release binaries / Build Linux musl ARM64 (push) Has been cancelled
Release binaries / Build Linux musl x64 (push) Has been cancelled
Release binaries / Build Linux x64 (push) Has been cancelled
Release binaries / Build Windows x64 (push) Has been cancelled
Release binaries / Attach binaries to GitHub Release (push) Has been cancelled
2026-06-09 17:32:44 +09:00
leeguooooo 091a4ec02e docs(skills): teach agents the extension-connect flow + computer-use for setup
So an agent can operate the zero-confirmation real-Chrome feature itself:
- SKILL.md: tool matrix gains "the user's own already-open, logged-in window →
  extension connect", plus a short section pointing at the flow.
- commands.md: the one-time "Load unpacked" is a privileged GUI step the CLI
  can't do — call it out that the agent can perform it with a computer-use /
  GUI-automation tool (cua-driver), with the live gotchas (synthetic-keystroke
  tools like peekaboo don't reach Chrome; cua-driver does; the native file
  dialog may need the user to pick the folder).
2026-06-09 17:14:50 +09:00
leeguooooo 6c0f5cbaa1 feat(connect): attach existing tabs + extension connect one-command UX
Completes the zero-confirmation real-Chrome feature.

- Drive the user's EXISTING logged-in tabs (not just newly-created ones):
  extension attachTab now treats "already attached" (a lingering chrome.debugger
  binding after a service-worker restart) as success and announces the tab
  anyway, instead of skipping it. The nm-host also sends {method:"attachAll"}
  when an agent-browser CDP client connects, so the daemon doesn't race an empty
  target list.
- `agent-browser extension connect` auto-discovers the relay's CDP url
  (~/.agent-browser/relay-cdp-url) and attaches — no copying a ws URL. Rewrites
  into the normal `connect <url>` flow; `extension install/status/uninstall`
  unchanged.
- Skill docs: a "drive your real, logged-in Chrome (extension)" section.

Verified end-to-end: `extension connect` listed the user's real tabs (Lark,
LINUX DO, Rakuten, Discord) and read a logged-in Lark doc's title — zero token,
zero confirmation. Full suite 768 passed.
2026-06-09 17:11:59 +09:00
leeguooooo 0d72e0d889 feat(connect): bridge native-messaging host to a CDP endpoint — end-to-end works
The __nm-host now exposes a Chrome-compatible CDP WebSocket endpoint and bridges
it to the extension over native messaging via the relay translation core
(relay.rs): incoming raw CDP commands are answered locally for browser-level
Target discovery or forwarded to the extension as forwardCDPCommand; the
extension's forwardCDPEvent/results are relayed back as raw CDP.

Security without a token or user interaction: the ws URL carries an unguessable
guid and is written to ~/.agent-browser/relay-cdp-url (perms 600), so only this
user's agent-browser can drive the browser — mirroring how Chrome guards its own
remote-debugging URL.

Verified end-to-end on real Chrome: `agent-browser connect <relay-url>` then an
eval navigated a tab and read back "Example Domain | https://example.com/" —
abs → CDP → relay → native messaging → extension → chrome.debugger → real tab,
zero token, zero confirmation. Adds the tokio io-std feature for the host's
stdio.

Remaining polish: re-attach the user's EXISTING tabs after a service-worker
restart (currently attaches new tabs cleanly; existing ones need detach+reattach
since chrome.debugger may still be bound), and an `open --extension` UX that
reads relay-cdp-url so the URL isn't passed by hand.
2026-06-09 16:54:29 +09:00
leeguooooo 528de4230f feat(connect): native-messaging transport — zero-token connect to real Chrome
Optimal architecture (chosen over the WS+token copy): the ab-connect extension
talks to a local agent-browser native-messaging host. No localhost port, no
token — Chrome authenticates the extension to the host by id. This is the
codex/claude-style "install once, no per-use confirmation" model.

- extensions/ab-connect: rewritten transport WebSocket+token → native messaging
  (chrome.runtime.connectNative). Pinned the extension id via a manifest `key`
  (→ bdoiejojpjogcjojeladhioioijhgade) so the host manifest can authorize it.
  Kept the proven chrome.debugger attach + Target.attachedToTarget emulation;
  dropped WS/token/options. Rebranded to "agent-browser connect".
- cli connect.rs: `agent-browser extension install` writes the native-messaging
  host manifest (Chrome/Chromium/Edge/Brave) + a launcher; hidden `__nm-host`
  speaks the 4-byte-length native-messaging framing.

Validated end-to-end on real Chrome: Chrome spawned the host (origin matched the
pinned id) and the extension attached the user's real logged-in tabs, streaming
Target.attachedToTarget over native messaging — zero token, zero port.

Next: bridge the host to the daemon relay (relay.rs) + CdpClient so
`agent-browser click/eval/...` drives those tabs.
2026-06-09 16:39:38 +09:00
leeguooooo 7f672494c1 feat(connect): relay translation core (envelope <-> raw CDP + Target emulation)
Pure, unit-tested core of the daemon-side relay that bridges the ab-connect
extension to the existing CdpClient. The extension exposes per-tab
chrome.debugger + synthesized Target events; CdpClient expects a browser-level
endpoint. So RelayState:

- answers Target.getTargets / attachToTarget / setDiscoverTargets LOCALLY from
  targets learned via the extension's forwardCDPEvent(Target.attachedToTarget),
  returning the extension's cb-tab-N sessionId (consumes those synth events
  rather than double-forwarding them);
- forwards every other command as a forwardCDPCommand envelope (carrying
  method/params/sessionId);
- maps forwardCDPCommand responses and forwardCDPEvent events back to raw CDP;
- validates the connect-handshake token; emits challenge/ping.

Keeps CdpClient and browser.rs unchanged. 8 unit tests; clippy clean. Still
inert — the tokio WS server + `connect` command wire it next.
2026-06-09 14:59:54 +09:00
leeguooooo 8a8106ad75 feat(connect): vendor MV3 connect extension (adapted from openclaw-browser-relay)
First step toward zero-confirmation direct connect to the user's real Chrome:
Chrome 136 killed --remote-debugging-port on the default profile, so the only
sanctioned way to drive the user's live logged-in window is an extension using
chrome.debugger (same approach as Codex/Claude, whose extensions are closed).

Vendors the MIT-licensed openclaw-browser-relay extension into
extensions/ab-connect/, rebranded to "agent-browser connect" (NOTICE.md keeps
attribution). It already handles the hard parts: chrome.debugger auto-attach all
tabs, new-tab auto-attach, MV3 service-worker keepalive (alarms) + reconnect,
sessionId↔tab mapping, token auth, and a CDP-over-WebSocket envelope
(connect handshake / forwardCDPCommand / forwardCDPEvent / ping-pong).

Inert for now — not wired. Next: an abs-daemon relay that speaks this envelope
and bridges it to the existing CdpClient (raw CDP), then a `connect` command.
2026-06-09 14:54:03 +09:00
leeguooooo f9cc31d003 chore(release): bump to 0.27.0-fork.17 — eval-first skill + find-url (local bookmark search) + site-notes convention
Release binaries / Build macOS ARM64 (push) Has been cancelled
Release binaries / Build macOS x64 (push) Has been cancelled
Release binaries / Build Linux ARM64 (push) Has been cancelled
Release binaries / Build Linux musl ARM64 (push) Has been cancelled
Release binaries / Build Linux musl x64 (push) Has been cancelled
Release binaries / Build Linux x64 (push) Has been cancelled
Release binaries / Build Windows x64 (push) Has been cancelled
Release binaries / Attach binaries to GitHub Release (push) Has been cancelled
2026-06-05 17:15:49 +09:00
leeguooooo a7a3f924b0 docs(skills): site-notes convention for remembering site quirks
Borrow web-access's site-experience persistence as an agent-workflow convention
(no CLI code): keep one markdown file per domain under
~/.agent-browser/site-patterns/<domain>.md. Read it before working a domain
(hints, not guarantees); update it after learning something durable — working
selectors, required hidden fields, anti-bot traps, login needs. Makes repeat
visits fast instead of re-solving the same page every run.
2026-06-05 16:55:47 +09:00
leeguooooo 7572c34229 feat(find-url): search local Chrome/Edge bookmarks by keyword
Borrow web-access's find-url: locate an internal system or a previously-saved
page that public search can't reach, without opening a browser.

- `agent-browser find-url <keywords> [--browser chrome|edge] [--profile X]
  [--limit N] [--json]` — local command, no daemon. All keywords must match a
  bookmark's name or url; results are most-recently-added first.
- Cross-platform Bookmarks JSON paths (macOS / Linux / Windows), zero new deps
  (serde_json). Skips javascript:/data: bookmarklets.
- Skill docs: "pick the cheapest tool" matrix now points at find-url, plus a
  commands.md section.

Bookmarks only for now — visited-history is a locked SQLite DB and would need a
SQLite dependency (deferred to avoid C-dep cross-compile risk in the release
pipeline).
2026-06-05 16:54:46 +09:00
leeguooooo 06f5f9e8f1 docs(skills): lead with eval-first + tool-choice matrix
Real dogfooding showed the skill pushed agents straight into the fragile
snapshot/@ref path. Reframe the core guidance toward how a developer actually
drives a real browser:

- "Pick the cheapest tool" matrix: WebSearch / WebFetch+curl for static, reach
  for agent-browser only when you need a real logged-in / interactive / dynamic
  browser. Plus: don't hand-build deep URLs — use links found by interacting.
- "Two ways to drive a page": structured (@ref/find) is convenient but lossy &
  fragile; eval-first (`eval "<js>"`) is the real DOM — read hidden inputs,
  Shadow DOM, form.elements/.validity, or el.click() directly. Drop to eval the
  moment the structured path fights you, instead of retrying it.
- Escalation ladder rewritten (refs → find → CSS → eval) and a note to retry a
  no-op click with AGENT_BROWSER_CLICK_MODE=dom.

Doc-only; closes the biggest part of the "abs feels worse than web-access" gap.
2026-06-05 16:44:29 +09:00
leeguooooo 5f50ca075c chore(release): bump to 0.27.0-fork.16 — click reliability (scroll-into-view + DOM fallback) + skill docs (console opt-in, CLICK_MODE, form/hidden-input eval)
Release binaries / Build macOS ARM64 (push) Has been cancelled
Release binaries / Build macOS x64 (push) Has been cancelled
Release binaries / Build Linux ARM64 (push) Has been cancelled
Release binaries / Build Linux musl ARM64 (push) Has been cancelled
Release binaries / Build Linux musl x64 (push) Has been cancelled
Release binaries / Build Linux x64 (push) Has been cancelled
Release binaries / Build Windows x64 (push) Has been cancelled
Release binaries / Attach binaries to GitHub Release (push) Has been cancelled
2026-06-05 11:00:57 +09:00
leeguooooo e7548c3eb5 fix(click): scroll into view + DOM-dispatch fallback for reliable clicks
Real-world dogfooding surfaced clicks that resolve a valid @ref but still miss:

- Scroll the target into view before computing click coordinates
  (scrollIntoViewIfNeeded). Without it, an element below the fold — or revealed
  after a scroll/popup — yields off-viewport coordinates and the click lands on
  whatever occupies that screen point.
- Fall back to a DOM-dispatched `.click()` when the coordinate path fails (a
  persistent floating layer failing the occlusion guard, or coordinates that
  won't resolve). The DOM dispatch targets the intended element directly instead
  of a screen point, so an overlay or portal can't divert it.
- AGENT_BROWSER_CLICK_MODE: "" (default: scroll + coordinate + DOM fallback),
  "coord" (strict coordinate, hard-fail on occlusion), "dom" (always
  element.click() — best for autocomplete/menu <li> that close on input blur).

Fallback is limited to left single-clicks (DOM .click() can't express
right/middle/double). Non-left/multi and "coord" mode keep the original error.

Docs: README knob table + skill commands.md gain CLICK_MODE, a click-reliability
note, and a "debug forms/hidden inputs with eval" section (snapshot doesn't show
hidden inputs — the fast path to bugs like a hidden point_choice=none).

6 click/interaction e2e green; full suite 760 passed.
2026-06-05 10:58:28 +09:00
leeguooooo b77a1e4568 docs(skills): document console-capture opt-in + stealth env knobs
console/errors capture is off by default in this fork (Runtime.enable is a
detectable CDP signal). Update the agent-facing skill docs so agents don't
treat empty console output as a bug:

- commands.md: new "Stealth / anti-detection knobs" env-var block
  (CAPTURE_CONSOLE, TIMEZONE, BLOCK_WEBRTC, HIDE_CANVAS, ADAPTIVE_REF) plus a
  heads-up note; annotate the console/errors lines.
- dogfood/slack SKILL.md: note that console/errors need
  AGENT_BROWSER_CAPTURE_CONSOLE=1.
2026-06-04 17:08:14 +09:00
leeguooooo 7c499885e5 chore(release): bump to 0.27.0-fork.15 — stealth hardening (lazy Runtime.enable, native timezone/WebRTC, opt-in canvas noise) + adaptive @ref relocation
Release binaries / Build macOS ARM64 (push) Has been cancelled
Release binaries / Build macOS x64 (push) Has been cancelled
Release binaries / Build Linux ARM64 (push) Has been cancelled
Release binaries / Build Linux musl ARM64 (push) Has been cancelled
Release binaries / Build Linux musl x64 (push) Has been cancelled
Release binaries / Build Linux x64 (push) Has been cancelled
Release binaries / Build Windows x64 (push) Has been cancelled
Release binaries / Attach binaries to GitHub Release (push) Has been cancelled
2026-06-04 16:26:51 +09:00
leeguooooo fc2621559b style: clear clippy warnings from the stealth/adaptive work
- snapshot: make collect_fingerprints private (TreeNode is private, so a
  pub(super) fn leaked a more-private type)
- adaptive: if-let instead of single-arm match in attr_score
- stealth: move timezone test module to end of file (items-after-test-module)

No behavior change. Pre-release cleanup.
2026-06-04 14:26:08 +09:00
leeguooooo 8b55c553e6 feat(adaptive): relocate stale @refs by AX fingerprint similarity
Borrow Scrapling's adaptive element finding, adapted to this project's
in-session AX-ref model. When a saved @ref's node is gone (or its identity
no longer matches) and the role/name/nth re-query also fails, score the
current page's candidate elements against an AX fingerprint captured at
snapshot time and relocate to the best match.

- New `adaptive` module: pure, browser-free scoring (role, accessible name
  via Levenshtein, AX properties, ancestor-role LCS, parent/sibling) plus
  pick_best with a high absolute threshold (0.70) AND a clear margin (0.15)
  over the runner-up — so ambiguous twins are refused rather than mis-clicked,
  matching the existing "fail loudly over wrong click" posture.
- Fingerprint captured during the existing AX-tree snapshot walk — no extra
  CDP round-trips. TreeNode is AX-only (no DOM tag/attrs), so we use AX role
  as the type and a few discriminating AX properties (value/url/level/checked);
  DOM id/class would have cost an N×describeNode storm per snapshot.
- Wired into both resolve_element_center and resolve_element_object_id: on a
  verify-identity mismatch or a stale-node fallback miss, relocation is tried
  before erroring. A confident match overrides the identity guard; otherwise
  the original error is surfaced. Opt out with AGENT_BROWSER_ADAPTIVE_REF=0.

README documents the new tuning knobs. Adds 9 unit tests; full suite 760 passed.
2026-06-04 14:10:50 +09:00
leeguooooo 6b99d304b1 feat(stealth): shrink detectable surface — lazy Runtime.enable, native timezone/WebRTC, opt-in canvas noise
Borrow anti-detection hardening from Scrapling/patchright, preferring native
CDP/Chrome overrides over JS lies:

- Runtime.enable is now opt-in via AGENT_BROWSER_CAPTURE_CONSOLE (default off).
  It was called on every session INCLUDING CdpAttach (the user's real Chrome),
  leaking the patchright/rebrowser "runtime" CDP signal and undermining the
  "real browser, no lies" guarantee. Runtime.evaluate/callFunctionOn and
  runIfWaitingForDebugger work without it; only console/error capture needs it.
  The console/errors commands now return a hint when capture is disabled.
- Timezone alignment via native Emulation.setTimezoneOverride, opt-in with
  AGENT_BROWSER_TIMEZONE=<IANA>|auto (FullLaunch only). Intl and Date both
  follow with no JS artifact.
- WebRTC IP-leak handling via the --force-webrtc-ip-handling-policy Chrome
  flag: auto disable_non_proxied_udp when a proxy is set (so the real IP can't
  leak past the proxy); AGENT_BROWSER_BLOCK_WEBRTC=1 hides the local IP when
  there is no proxy; =0 opts out.
- Opt-in canvas/audio fingerprint noise via AGENT_BROWSER_HIDE_CANVAS=1
  (FullLaunch only). Session-stable seed so reads stay consistent within a
  session while differing from the headless-stable hash.

Adds 5 unit tests; full suite 751 passed, 0 failed.
2026-06-04 13:28:34 +09:00
leeguooooo 900a5b5cde fix(install): also create the agent-browser-stealth command name
install.sh created `agent-browser` + `abs` but not `agent-browser-stealth`, so
users who invoke `agent-browser-stealth` (the fork's package name) weren't
getting it updated on curl-install/upgrade. Now all three names — agent-browser,
agent-browser-stealth, abs — symlink to the same binary, so an upgrade refreshes
whichever name you actually run.
2026-06-01 18:55:58 +09:00
26 changed files with 2783 additions and 52 deletions
+14 -2
View File
@@ -115,9 +115,10 @@ In CI environments, standalone mode is used automatically.
## Anti-detection
When connected to your real Chrome, we inject **zero** JavaScript patches. Your browser's fingerprint is completely genuine.
When connected to your real Chrome, we inject **zero** JavaScript patches. Your browser's fingerprint is completely genuine. The guiding rule is **native CDP/Chrome overrides over JS lies** — a re-defined getter is itself detectable; a native override isn't.
The only thing we do is call `Emulation.setAutomationOverride` via CDP to set `navigator.webdriver = false` at the native Chrome level — undetectable by lie-detection systems like CreepJS.
- `navigator.webdriver = false` via `Emulation.setAutomationOverride` (native, undetectable by CreepJS-style lie tests).
- **`Runtime.enable` is left OFF by default.** A live `Runtime` domain is a detectable CDP signal (the patchright/rebrowser "runtime leak") — even when attached to your real Chrome. We only enable it when you opt into console/error capture (see below). `click`, `fill`, `eval`, etc. work without it.
**Test results (connected to real Chrome):**
@@ -129,6 +130,17 @@ The only thing we do is call `Emulation.setAutomationOverride` via CDP to set `n
When using `--launch` mode (standalone browser), a full suite of 32 stealth patches is applied for headless Chrome.
### Tuning knobs (environment variables)
| Variable | Default | Effect |
|---|---|---|
| `AGENT_BROWSER_CAPTURE_CONSOLE` | off | Enable `Runtime` domain so `console` / `errors` capture page output. Off keeps the stealthiest profile. |
| `AGENT_BROWSER_TIMEZONE` | unset | `--launch` only. An IANA id (e.g. `Asia/Tokyo`) sets the timezone natively (Intl + Date follow, no JS lie) to match a proxy; `auto` derives one from the locale. |
| `AGENT_BROWSER_BLOCK_WEBRTC` | auto | `--launch` only. Auto-forces WebRTC through the proxy when one is set (no real-IP leak). `1` hides the local IP without a proxy; `0` opts out. |
| `AGENT_BROWSER_HIDE_CANVAS` | off | `--launch` only. Adds session-stable canvas/audio fingerprint noise. Off by default (noise is itself a "lie"). |
| `AGENT_BROWSER_ADAPTIVE_REF` | on | When a saved `@ref` moves and the role/name re-query fails, relocate it by fingerprint similarity (high score + clear margin required, else it fails loudly). `0` disables. |
| `AGENT_BROWSER_CLICK_MODE` | _(auto)_ | Click strategy. Default scrolls the target into view, dispatches a coordinate click, and falls back to a DOM `.click()` if a floating layer occludes the point. `dom` always uses `.click()` (best for autocomplete/menu items that close on blur); `coord` is strict coordinate-only (hard-fail on occlusion). |
## Differences from upstream
Based on [agent-browser v0.27.0](https://github.com/vercel-labs/agent-browser). Changes:
+1 -1
View File
@@ -45,7 +45,7 @@ dependencies = [
[[package]]
name = "agent-browser-stealth"
version = "0.27.0-fork.14"
version = "0.27.0-fork.18"
dependencies = [
"aes-gcm",
"async-trait",
+2 -2
View File
@@ -1,6 +1,6 @@
[package]
name = "agent-browser-stealth"
version = "0.27.0-fork.14"
version = "0.27.0-fork.18"
edition = "2021"
description = "Fast browser automation CLI for AI agents"
license = "Apache-2.0"
@@ -22,7 +22,7 @@ dirs = "5.0"
include_dir = "0.7"
base64 = "0.22"
getrandom = "0.2"
tokio = { version = "1", features = ["rt-multi-thread", "macros", "net", "io-util", "time", "sync", "signal", "process"] }
tokio = { version = "1", features = ["rt-multi-thread", "macros", "net", "io-util", "io-std", "time", "sync", "signal", "process"] }
tokio-tungstenite = { version = "0.24", features = ["rustls-tls-webpki-roots"] }
futures-util = "0.3"
url = "2"
+439
View File
@@ -0,0 +1,439 @@
//! `agent-browser connect` — zero-confirmation control of the user's real,
//! logged-in Chrome via the `ab-connect` MV3 extension over Chrome **native
//! messaging** (no localhost port, no token; Chrome authenticates the extension
//! to this host by id).
//!
//! Two pieces live here:
//! - `run_connect` — `--install` writes the native-messaging host manifest (and
//! a tiny launcher) so Chrome will spawn us; with no flag it reports status.
//! - `run_nm_host` — the hidden `__nm-host` mode Chrome launches: it speaks the
//! native-messaging stdio framing (4-byte little-endian length + JSON).
//!
//! This step wires the transport end-to-end (Chrome ⇄ host). Bridging the host
//! to the daemon's relay + CdpClient is layered on next.
use std::io::Write;
use std::path::PathBuf;
/// Native-messaging host name; must match `HOST_NAME` in the extension and the
/// manifest filename.
pub const HOST_NAME: &str = "com.agent_browser.connect";
/// Stable id of the `ab-connect` extension, pinned by the `key` in its
/// manifest.json. Chrome only lets that extension talk to this host.
pub const EXTENSION_ID: &str = "bdoiejojpjogcjojeladhioioijhgade";
/// `agent-browser extension <install|uninstall|status>` (local; no daemon).
/// `args` is the cleaned argv including the leading "extension".
pub fn run_connect(args: &[String], json: bool) {
let install = args.iter().any(|a| a == "--install" || a == "install");
let uninstall = args.iter().any(|a| a == "--uninstall" || a == "uninstall");
if uninstall {
let removed = remove_host_manifests();
report(json, true, &format!("removed {removed} native-host manifest(s)"));
return;
}
if install {
match install_native_host() {
Ok(paths) => {
if json {
println!(
"{}",
serde_json::to_string(&serde_json::json!({
"success": true,
"data": { "installed": paths, "extensionId": EXTENSION_ID }
}))
.unwrap_or_default()
);
} else {
println!("✓ native-messaging host installed:");
for p in &paths {
println!(" {p}");
}
println!(
"\nNext: load the ab-connect extension in Chrome (chrome://extensions →\n\
Developer mode → Load unpacked → extensions/ab-connect), then this host\n\
is reachable with no token and no per-use confirmation."
);
}
}
Err(e) => report(json, false, &format!("install failed: {e}")),
}
return;
}
// Status.
let manifest = host_manifest_path_for_chrome();
let installed = manifest.as_ref().map(|p| p.exists()).unwrap_or(false);
if json {
println!(
"{}",
serde_json::to_string(&serde_json::json!({
"success": true,
"data": {
"installed": installed,
"manifest": manifest.as_ref().map(|p| p.display().to_string()),
"extensionId": EXTENSION_ID,
}
}))
.unwrap_or_default()
);
} else if installed {
println!("✓ native-messaging host installed ({HOST_NAME}).");
println!(" Load the ab-connect extension and it connects automatically.");
} else {
println!("✗ not installed. Run: agent-browser connect --install");
}
}
/// Write the launcher script + native-messaging host manifest(s).
fn install_native_host() -> Result<Vec<String>, String> {
let home = dirs::home_dir().ok_or("no home dir")?;
let ab_dir = home.join(".agent-browser");
std::fs::create_dir_all(&ab_dir).map_err(|e| e.to_string())?;
// Chrome execs the manifest `path` directly with the calling extension's
// origin as argv[1]; a launcher lets us run the binary in __nm-host mode
// regardless of how/where agent-browser is installed.
let exe = std::env::current_exe().map_err(|e| e.to_string())?;
let launcher = ab_dir.join("nm-host.sh");
let script = format!(
"#!/bin/sh\n# agent-browser native-messaging host launcher (auto-generated)\nexec \"{}\" __nm-host \"$@\"\n",
exe.display()
);
std::fs::write(&launcher, script).map_err(|e| e.to_string())?;
#[cfg(unix)]
{
use std::os::unix::fs::PermissionsExt;
let _ = std::fs::set_permissions(&launcher, std::fs::Permissions::from_mode(0o755));
}
let manifest = serde_json::json!({
"name": HOST_NAME,
"description": "agent-browser connect — native messaging host",
"path": launcher.display().to_string(),
"type": "stdio",
"allowed_origins": [format!("chrome-extension://{EXTENSION_ID}/")],
});
let body = serde_json::to_string_pretty(&manifest).map_err(|e| e.to_string())?;
let mut written = Vec::new();
for dir in native_messaging_dirs() {
if let Some(parent) = dir.parent() {
if !parent.exists() {
continue; // that browser isn't installed
}
}
std::fs::create_dir_all(&dir).map_err(|e| e.to_string())?;
let path = dir.join(format!("{HOST_NAME}.json"));
std::fs::write(&path, &body).map_err(|e| e.to_string())?;
written.push(path.display().to_string());
}
if written.is_empty() {
return Err("no Chrome/Chromium NativeMessagingHosts directory found".into());
}
Ok(written)
}
fn remove_host_manifests() -> usize {
let mut n = 0;
for dir in native_messaging_dirs() {
let path = dir.join(format!("{HOST_NAME}.json"));
if path.exists() && std::fs::remove_file(&path).is_ok() {
n += 1;
}
}
n
}
/// Per-OS NativeMessagingHosts directories for Chrome + Chromium-family browsers.
fn native_messaging_dirs() -> Vec<PathBuf> {
let mut dirs_out = Vec::new();
#[cfg(target_os = "macos")]
{
if let Some(app_support) = dirs::config_dir() {
for sub in [
"Google/Chrome",
"Google/Chrome Beta",
"Google/Chrome Canary",
"Chromium",
"Microsoft Edge",
"BraveSoftware/Brave-Browser",
] {
dirs_out.push(app_support.join(sub).join("NativeMessagingHosts"));
}
}
}
#[cfg(all(unix, not(target_os = "macos")))]
{
if let Some(config) = dirs::config_dir() {
for sub in ["google-chrome", "chromium", "microsoft-edge", "BraveSoftware/Brave-Browser"] {
dirs_out.push(config.join(sub).join("NativeMessagingHosts"));
}
}
}
dirs_out
}
fn host_manifest_path_for_chrome() -> Option<PathBuf> {
native_messaging_dirs()
.into_iter()
.map(|d| d.join(format!("{HOST_NAME}.json")))
.find(|p| p.exists())
.or_else(|| {
native_messaging_dirs()
.into_iter()
.next()
.map(|d| d.join(format!("{HOST_NAME}.json")))
})
}
fn report(json: bool, ok: bool, msg: &str) {
if json {
println!(
"{}",
serde_json::to_string(&serde_json::json!({ "success": ok, "error": if ok { serde_json::Value::Null } else { serde_json::json!(msg) }, "message": msg }))
.unwrap_or_default()
);
} else if ok {
println!("{msg}");
} else {
eprintln!("{msg}");
}
if !ok {
std::process::exit(1);
}
}
// ---- native messaging host (`__nm-host`) ----------------------------------
fn nm_log(line: &str) {
let path = dirs::home_dir()
.map(|h| h.join(".agent-browser").join("nm-host.log"))
.unwrap_or_else(|| PathBuf::from("/tmp/ab-nm-host.log"));
if let Some(p) = path.parent() {
let _ = std::fs::create_dir_all(p);
}
if let Ok(mut f) = std::fs::OpenOptions::new().create(true).append(true).open(&path) {
let _ = writeln!(f, "{line}");
}
}
fn random_guid() -> String {
let mut b = [0u8; 16];
let _ = getrandom::getrandom(&mut b);
b.iter().map(|x| format!("{x:02x}")).collect()
}
/// Where the daemon/CLI reads the relay's CDP WebSocket URL (perms 600).
fn relay_url_path() -> PathBuf {
dirs::home_dir()
.map(|h| h.join(".agent-browser").join("relay-cdp-url"))
.unwrap_or_else(|| PathBuf::from("/tmp/ab-relay-cdp-url"))
}
/// The live relay CDP WebSocket URL, if the native-messaging host is running
/// (it writes the file on connect and removes it on exit). Used by
/// `agent-browser extension connect` to attach without the user copying a URL.
pub fn relay_url() -> Option<String> {
let s = std::fs::read_to_string(relay_url_path()).ok()?;
let s = s.trim().to_string();
if s.starts_with("ws://") {
Some(s)
} else {
None
}
}
/// Hidden `__nm-host` mode: launched by Chrome for the ab-connect extension.
///
/// Bridges the extension (native-messaging stdio, envelope protocol) to a local
/// **CDP WebSocket endpoint** that agent-browser connects to like any Chrome.
/// `relay::RelayState` translates envelope ⇄ raw CDP and emulates browser-level
/// Target discovery. The ws URL carries an unguessable guid (written to a 600
/// file) so only this user's agent-browser — not arbitrary local processes —
/// can drive the browser. No token, no user interaction.
pub fn run_nm_host() {
let rt = match tokio::runtime::Builder::new_multi_thread().enable_all().build() {
Ok(rt) => rt,
Err(e) => {
nm_log(&format!("[nm-host] runtime build failed: {e}"));
return;
}
};
rt.block_on(nm_host_main());
}
async fn nm_host_main() {
use crate::native::relay::{RelayOut, RelayState};
use std::sync::Arc;
use tokio::io::{AsyncReadExt, AsyncWriteExt};
use tokio::sync::{broadcast, mpsc, Mutex};
nm_log(&format!(
"[nm-host] start argv={:?}",
std::env::args().skip(1).collect::<Vec<_>>()
));
let listener = match tokio::net::TcpListener::bind("127.0.0.1:0").await {
Ok(l) => l,
Err(e) => {
nm_log(&format!("[nm-host] bind failed: {e}"));
return;
}
};
let port = listener.local_addr().map(|a| a.port()).unwrap_or(0);
let guid = random_guid();
let url = format!("ws://127.0.0.1:{port}/{guid}");
let url_path = relay_url_path();
if let Some(p) = url_path.parent() {
let _ = std::fs::create_dir_all(p);
}
if std::fs::write(&url_path, &url).is_ok() {
#[cfg(unix)]
{
use std::os::unix::fs::PermissionsExt;
let _ = std::fs::set_permissions(&url_path, std::fs::Permissions::from_mode(0o600));
}
}
nm_log(&format!("[nm-host] cdp endpoint {url}"));
let state = Arc::new(Mutex::new(RelayState::new()));
let (to_clients, _) = broadcast::channel::<String>(4096);
let (to_ext, mut to_ext_rx) = mpsc::channel::<Vec<u8>>(4096);
// Single writer to Chrome (extension) over stdout, native-messaging framed.
tokio::spawn(async move {
let mut out = tokio::io::stdout();
while let Some(frame) = to_ext_rx.recv().await {
let len = (frame.len() as u32).to_ne_bytes();
if out.write_all(&len).await.is_err() || out.write_all(&frame).await.is_err() {
break;
}
let _ = out.flush().await;
}
});
// Accept agent-browser CDP clients on the guid-scoped ws endpoint.
{
let state = state.clone();
let to_clients = to_clients.clone();
let to_ext = to_ext.clone();
let guid = guid.clone();
tokio::spawn(async move {
loop {
let (stream, _) = match listener.accept().await {
Ok(x) => x,
Err(_) => break,
};
let st = state.clone();
let rx = to_clients.subscribe();
let tx = to_ext.clone();
let g = guid.clone();
tokio::spawn(async move {
handle_cdp_client(stream, g, st, rx, tx).await;
});
}
});
}
// Extension → host frames.
let mut stdin = tokio::io::stdin();
loop {
let mut len_buf = [0u8; 4];
if stdin.read_exact(&mut len_buf).await.is_err() {
break;
}
let len = u32::from_ne_bytes(len_buf) as usize;
let mut buf = vec![0u8; len];
if stdin.read_exact(&mut buf).await.is_err() {
break;
}
let v: serde_json::Value = match serde_json::from_slice(&buf) {
Ok(v) => v,
Err(_) => continue,
};
let outs = {
let mut s = state.lock().await;
s.handle_ext_message(&v, "")
};
for o in outs {
match o {
RelayOut::ToClient(m) => {
let _ = to_clients.send(m.to_string());
}
RelayOut::ToExt(m) => {
let _ = to_ext.send(m.to_string().into_bytes()).await;
}
}
}
}
nm_log("[nm-host] stdin EOF — Chrome closed the port");
let _ = std::fs::remove_file(relay_url_path());
}
async fn handle_cdp_client(
stream: tokio::net::TcpStream,
guid: String,
state: std::sync::Arc<tokio::sync::Mutex<crate::native::relay::RelayState>>,
mut from_relay: tokio::sync::broadcast::Receiver<String>,
to_ext: tokio::sync::mpsc::Sender<Vec<u8>>,
) {
use crate::native::relay::ClientRoute;
use futures_util::{SinkExt, StreamExt};
use tokio::sync::broadcast::error::RecvError;
use tokio_tungstenite::tungstenite::Message;
let want_path = format!("/{guid}");
let cb = |req: &tokio_tungstenite::tungstenite::handshake::server::Request,
resp: tokio_tungstenite::tungstenite::handshake::server::Response| {
if req.uri().path() == want_path {
Ok(resp)
} else {
let mut reject = tokio_tungstenite::tungstenite::handshake::server::ErrorResponse::new(
Some("forbidden".to_string()),
);
*reject.status_mut() = tokio_tungstenite::tungstenite::http::StatusCode::FORBIDDEN;
Err(reject)
}
};
let ws = match tokio_tungstenite::accept_hdr_async(stream, cb).await {
Ok(ws) => ws,
Err(_) => return,
};
nm_log("[nm-host] cdp client connected");
// Ask the extension to (re)attach + announce every tab so this client
// discovers the user's existing tabs instead of racing an empty list.
let _ = to_ext.send(br#"{"method":"attachAll"}"#.to_vec()).await;
let (mut tx, mut rx) = ws.split();
loop {
tokio::select! {
relayed = from_relay.recv() => match relayed {
Ok(text) => { if tx.send(Message::Text(text)).await.is_err() { break } }
Err(RecvError::Lagged(_)) => continue,
Err(RecvError::Closed) => break,
},
incoming = rx.next() => match incoming {
Some(Ok(Message::Text(text))) => {
let v: serde_json::Value = match serde_json::from_str(&text) {
Ok(v) => v,
Err(_) => continue,
};
let route = { state.lock().await.route_client_command(&v) };
match route {
ClientRoute::Local(reply) => {
if tx.send(Message::Text(reply.to_string())).await.is_err() { break }
}
ClientRoute::Forward(env) => {
let _ = to_ext.send(env.to_string().into_bytes()).await;
}
}
}
Some(Ok(Message::Close(_))) | None => break,
_ => {}
},
}
}
nm_log("[nm-host] cdp client disconnected");
}
+244
View File
@@ -0,0 +1,244 @@
//! `find-url` — search the user's local Chrome/Edge **bookmarks** for pages they
//! saved, by keyword. Borrowed from web-access's `find-url.mjs`; lets an agent
//! locate an internal system or a previously-saved page that public search
//! can't reach, without opening a browser.
//!
//! v1 covers bookmarks only (a zero-dependency JSON read). Visited-history lives
//! in a locked SQLite DB and would need a SQLite dependency — not included yet.
use std::path::PathBuf;
use serde_json::Value;
use crate::color;
struct Hit {
name: String,
url: String,
folder: String,
date_added: i64,
}
/// Entry point for the `find-url` subcommand. `args` is the full cleaned argv
/// (including the leading "find-url").
pub fn run_find_url(args: &[String], json: bool) {
// Parse flags out of args[1..]; everything else is a keyword.
let mut browser = "chrome".to_string();
let mut profile = "Default".to_string();
let mut limit: usize = 20;
let mut keywords: Vec<String> = Vec::new();
let mut i = 1;
while i < args.len() {
match args[i].as_str() {
"--browser" => {
if let Some(v) = args.get(i + 1) {
browser = v.to_lowercase();
i += 1;
}
}
"--profile" => {
if let Some(v) = args.get(i + 1) {
profile = v.clone();
i += 1;
}
}
"--limit" => {
if let Some(v) = args.get(i + 1).and_then(|s| s.parse::<usize>().ok()) {
limit = v;
i += 1;
}
}
"--json" => {}
other if other.starts_with("--") => {}
other => keywords.push(other.to_lowercase()),
}
i += 1;
}
let path = match bookmarks_path(&browser, &profile) {
Some(p) => p,
None => {
emit_error(
json,
&format!("Could not locate {browser} bookmarks for profile '{profile}'"),
);
return;
}
};
let raw = match std::fs::read_to_string(&path) {
Ok(r) => r,
Err(e) => {
emit_error(json, &format!("Failed to read {}: {e}", path.display()));
return;
}
};
let root: Value = match serde_json::from_str(&raw) {
Ok(v) => v,
Err(e) => {
emit_error(json, &format!("Failed to parse bookmarks JSON: {e}"));
return;
}
};
let mut hits: Vec<Hit> = Vec::new();
if let Some(roots) = root.get("roots").and_then(|r| r.as_object()) {
for node in roots.values() {
walk(node, "", &keywords, &mut hits);
}
}
// Most-recently-added first (date_added is microseconds since 1601).
hits.sort_by(|a, b| b.date_added.cmp(&a.date_added));
hits.truncate(limit);
if json {
let arr: Vec<Value> = hits
.iter()
.map(|h| {
serde_json::json!({
"name": h.name,
"url": h.url,
"folder": h.folder,
})
})
.collect();
println!(
"{}",
serde_json::to_string(&serde_json::json!({
"success": true,
"data": { "results": arr, "count": hits.len() },
}))
.unwrap_or_default()
);
return;
}
if hits.is_empty() {
let kw = if keywords.is_empty() {
String::new()
} else {
format!(" matching {:?}", keywords.join(" "))
};
println!("No {browser} bookmarks found{kw}.");
return;
}
for h in &hits {
if h.folder.is_empty() {
println!("{}\n {}", h.name, h.url);
} else {
println!("{} ({})\n {}", h.name, h.folder, h.url);
}
}
}
/// Recursively walk a bookmark node, collecting URL entries that match every
/// keyword (in name or url). Empty keyword list matches everything.
fn walk(node: &Value, folder: &str, keywords: &[String], out: &mut Vec<Hit>) {
match node.get("type").and_then(|t| t.as_str()) {
Some("url") => {
let name = node.get("name").and_then(|v| v.as_str()).unwrap_or("");
let url = node.get("url").and_then(|v| v.as_str()).unwrap_or("");
// Skip non-navigable bookmarks: javascript: bookmarklets and data:
// URIs aren't pages you can visit, and their bodies can be huge.
if url.is_empty()
|| url.starts_with("javascript:")
|| url.starts_with("data:")
{
return;
}
let hay = format!("{} {}", name.to_lowercase(), url.to_lowercase());
if keywords.iter().all(|k| hay.contains(k.as_str())) {
let date_added = node
.get("date_added")
.and_then(|v| v.as_str())
.and_then(|s| s.parse::<i64>().ok())
.unwrap_or(0);
out.push(Hit {
name: name.to_string(),
url: url.to_string(),
folder: folder.to_string(),
date_added,
});
}
}
Some("folder") => {
let fname = node.get("name").and_then(|v| v.as_str()).unwrap_or("");
let child_folder = if folder.is_empty() {
fname.to_string()
} else {
format!("{folder}/{fname}")
};
if let Some(children) = node.get("children").and_then(|c| c.as_array()) {
for child in children {
walk(child, &child_folder, keywords, out);
}
}
}
_ => {}
}
}
/// Resolve the Bookmarks file path for a browser + profile across platforms.
fn bookmarks_path(browser: &str, profile: &str) -> Option<PathBuf> {
let base = browser_user_data_dir(browser)?;
let path = base.join(profile).join("Bookmarks");
if path.exists() {
Some(path)
} else {
None
}
}
/// The "User Data" directory that holds per-profile folders, per OS/browser.
fn browser_user_data_dir(browser: &str) -> Option<PathBuf> {
let is_edge = browser == "edge" || browser == "msedge";
#[cfg(target_os = "macos")]
{
let app_support = dirs::config_dir()?; // ~/Library/Application Support
let sub = if is_edge {
"Microsoft Edge"
} else {
"Google/Chrome"
};
Some(app_support.join(sub))
}
#[cfg(target_os = "windows")]
{
let local = dirs::data_local_dir()?; // %LOCALAPPDATA%
let sub = if is_edge {
"Microsoft/Edge/User Data"
} else {
"Google/Chrome/User Data"
};
Some(local.join(sub))
}
#[cfg(all(unix, not(target_os = "macos")))]
{
let config = dirs::config_dir()?; // ~/.config
let sub = if is_edge {
"microsoft-edge"
} else {
"google-chrome"
};
Some(config.join(sub))
}
}
fn emit_error(json: bool, msg: &str) {
if json {
println!(
"{}",
serde_json::to_string(&serde_json::json!({
"success": false,
"error": msg,
}))
.unwrap_or_default()
);
} else {
eprintln!("{} {msg}", color::error_indicator());
}
std::process::exit(1);
}
+45 -1
View File
@@ -2,7 +2,9 @@ mod chat;
mod color;
mod commands;
mod connection;
mod connect;
mod doctor;
mod findurl;
mod flags;
mod install;
mod native;
@@ -502,6 +504,14 @@ fn main() {
env::set_var("MSYS2_ARG_CONV_EXCL", "*");
}
// Native-messaging host mode: Chrome launches `agent-browser __nm-host
// <extension-origin> [...]` for the ab-connect extension. Must run before
// ANY stdout write — stdout is the Chrome native-messaging channel.
if env::args().nth(1).as_deref() == Some("__nm-host") {
connect::run_nm_host();
return;
}
// Native daemon mode: when AGENT_BROWSER_DAEMON is set, run as the daemon process
if env::var("AGENT_BROWSER_DAEMON").is_ok() {
// Ignore SIGPIPE so the daemon isn't killed when the parent drops
@@ -529,7 +539,7 @@ fn main() {
let args: Vec<String> = env::args().skip(1).collect();
let mut flags = parse_flags(&args);
let clean = clean_args(&args);
let mut clean = clean_args(&args);
// Loudly warn when launching a fresh browser with no profile: it gets a
// temporary EMPTY profile (no cookies / no login). For logged-in sites the
@@ -631,6 +641,40 @@ fn main() {
return;
}
// Handle find-url (doesn't need daemon): search local bookmarks
if matches!(
clean.first().map(|s| s.as_str()),
Some("find-url") | Some("findurl")
) {
findurl::run_find_url(&clean, flags.json);
return;
}
// Handle extension: native-messaging host install/status, and
// `extension connect` which attaches to the live relay (auto-discovers the
// CDP url the host wrote) by rewriting into the normal `connect <url>` flow.
// (`connect <port>` stays the plain CDP-attach command.)
if clean.first().map(|s| s.as_str()) == Some("extension") {
if clean.get(1).map(|s| s.as_str()) == Some("connect") {
match connect::relay_url() {
Some(url) => {
clean = vec!["connect".to_string(), url];
// fall through to the normal connect handling below
}
None => {
eprintln!(
"{} extension not connected. Run `agent-browser extension install`, load the\n ab-connect extension in Chrome (chrome://extensions → Developer mode →\n Load unpacked → extensions/ab-connect), then retry.",
color::error_indicator()
);
exit(1);
}
}
} else {
connect::run_connect(&clean, flags.json);
return;
}
}
// Handle session separately (doesn't need daemon)
if clean.first().map(|s| s.as_str()) == Some("session") {
run_session(&clean, &flags.session, flags.json);
+27 -2
View File
@@ -3690,13 +3690,38 @@ async fn handle_console(cmd: &Value, state: &mut DaemonState) -> Result<Value, S
state.event_tracker.clear_console();
Ok(json!({ "cleared": true }))
} else {
let result = state.event_tracker.get_console_json();
let mut result = state.event_tracker.get_console_json();
if !console_capture_active(state) {
if let Some(obj) = result.as_object_mut() {
obj.insert("hint".to_string(), json!(CONSOLE_DISABLED_HINT));
}
}
Ok(result)
}
}
/// Whether the active browser session has the CDP `Runtime` domain enabled
/// (required to receive console/error events). OFF by default for stealth.
fn console_capture_active(state: &DaemonState) -> bool {
state
.browser
.as_ref()
.map(|b| b.capture_console)
.unwrap_or(false)
}
const CONSOLE_DISABLED_HINT: &str =
"console/error capture is disabled for stealth (Runtime.enable is a detectable CDP \
signal). Restart the session with AGENT_BROWSER_CAPTURE_CONSOLE=1 to capture page output.";
async fn handle_errors(state: &DaemonState) -> Result<Value, String> {
Ok(state.event_tracker.get_errors_json())
let mut result = state.event_tracker.get_errors_json();
if !console_capture_active(state) {
if let Some(obj) = result.as_object_mut() {
obj.insert("hint".to_string(), json!(CONSOLE_DISABLED_HINT));
}
}
Ok(result)
}
async fn handle_state_save(cmd: &Value, state: &DaemonState) -> Result<Value, String> {
+364
View File
@@ -0,0 +1,364 @@
//! Adaptive @ref relocation.
//!
//! When a saved `@ref`'s DOM node is gone (stale `backendNodeId`) and the
//! role/name/nth re-query also fails, we score the current page's candidate
//! elements against the ref's stored [`ElementFingerprint`] and relocate to the
//! best match — but ONLY when confident: the best candidate must clear a high
//! absolute threshold AND beat the runner-up by a clear margin. This matches the
//! project's "fail loudly rather than mis-click" posture (see the identity and
//! occlusion guards in `element.rs`).
//!
//! Everything in this module is pure and browser-free so the scoring can be
//! unit-tested directly.
use std::collections::BTreeMap;
/// Minimum absolute similarity (0..1) for a relocation candidate to be accepted.
pub const ADAPTIVE_THRESHOLD: f64 = 0.70;
/// Minimum gap between the best and second-best candidate to avoid ambiguity.
pub const ADAPTIVE_MARGIN: f64 = 0.15;
/// A structural/semantic fingerprint of an element, captured at snapshot time so
/// a moved element can be re-identified after the page mutates.
///
/// Populated purely from the accessibility tree we already walk (`TreeNode`), so
/// capturing it costs no extra CDP round-trips — `TreeNode` has no DOM tag or
/// attributes (those would need an N×`DOM.describeNode` storm per snapshot), so
/// `tag` holds the AX **role** and `attrs` holds discriminating AX properties
/// (value/url/level/checked), not DOM `id`/`class`.
#[derive(Debug, Clone, Default, PartialEq)]
pub struct ElementFingerprint {
/// AX role, e.g. "button" (used where a DOM tag would otherwise go).
pub tag: String,
/// Accessible name / visible text — the dominant identity signal.
pub text: String,
/// Discriminating AX properties: value, url, level, checked. Keyed by name.
pub attrs: BTreeMap<String, String>,
/// Ancestor role signatures from nearest to farthest, e.g. "form" / "list".
pub ancestors: Vec<String>,
/// Parent role.
pub parent_tag: String,
/// Parent accessible name / text.
pub parent_text: String,
/// Index among same-role siblings.
pub sibling_index: u32,
/// Count of same-role siblings.
pub sibling_count: u32,
}
/// Component weights. They sum to 1.0 so the total score lands in 0..1.
/// Tuned for AX-derived fingerprints: the accessible name dominates, with role
/// and tree structure carrying disambiguation when the name has changed (which
/// is exactly when the exact role+name+nth fallback failed and we got here).
const W_TAG: f64 = 0.20;
const W_TEXT: f64 = 0.40;
const W_ATTRS: f64 = 0.10;
const W_ANCESTORS: f64 = 0.20;
const W_PARENT_SIBLING: f64 = 0.10;
/// Per-attribute importance for the attribute-overlap score. Strong identity
/// signals (a link's url) outweigh weak ones (heading level).
fn attr_weight(name: &str) -> f64 {
match name {
"url" | "value" => 3.0,
"checked" => 2.0,
_ => 1.0,
}
}
/// Levenshtein-based string similarity in 0..1 (1.0 = identical). Two empty
/// strings are treated as a perfect match (consistent absence of text).
pub fn string_similarity(a: &str, b: &str) -> f64 {
if a == b {
return 1.0;
}
let a: Vec<char> = a.chars().collect();
let b: Vec<char> = b.chars().collect();
let max_len = a.len().max(b.len());
if max_len == 0 {
return 1.0;
}
let dist = levenshtein(&a, &b);
1.0 - (dist as f64 / max_len as f64)
}
fn levenshtein(a: &[char], b: &[char]) -> usize {
if a.is_empty() {
return b.len();
}
if b.is_empty() {
return a.len();
}
let mut prev: Vec<usize> = (0..=b.len()).collect();
let mut cur = vec![0usize; b.len() + 1];
for (i, &ca) in a.iter().enumerate() {
cur[0] = i + 1;
for (j, &cb) in b.iter().enumerate() {
let cost = if ca == cb { 0 } else { 1 };
cur[j + 1] = (prev[j + 1] + 1).min(cur[j] + 1).min(prev[j] + cost);
}
std::mem::swap(&mut prev, &mut cur);
}
prev[b.len()]
}
/// Jaccard similarity over whitespace-separated tokens (used for `class`).
fn token_jaccard(a: &str, b: &str) -> f64 {
let sa: std::collections::BTreeSet<&str> = a.split_whitespace().collect();
let sb: std::collections::BTreeSet<&str> = b.split_whitespace().collect();
if sa.is_empty() && sb.is_empty() {
return 1.0;
}
let inter = sa.intersection(&sb).count() as f64;
let union = sa.union(&sb).count() as f64;
if union == 0.0 {
1.0
} else {
inter / union
}
}
/// Length-ratio of the longest common subsequence over two ancestor sequences.
fn lcs_ratio(a: &[String], b: &[String]) -> f64 {
if a.is_empty() && b.is_empty() {
return 1.0;
}
if a.is_empty() || b.is_empty() {
return 0.0;
}
let mut dp = vec![vec![0usize; b.len() + 1]; a.len() + 1];
for i in 0..a.len() {
for j in 0..b.len() {
dp[i + 1][j + 1] = if a[i] == b[j] {
dp[i][j] + 1
} else {
dp[i][j + 1].max(dp[i + 1][j])
};
}
}
let lcs = dp[a.len()][b.len()] as f64;
(2.0 * lcs) / (a.len() + b.len()) as f64
}
fn attr_score(base: &BTreeMap<String, String>, cand: &BTreeMap<String, String>) -> f64 {
let mut names: std::collections::BTreeSet<&str> = std::collections::BTreeSet::new();
names.extend(base.keys().map(|s| s.as_str()));
names.extend(cand.keys().map(|s| s.as_str()));
if names.is_empty() {
return 1.0; // no attributes on either side — neutral
}
let mut total = 0.0;
let mut got = 0.0;
for name in names {
let w = attr_weight(name);
total += w;
// present on only one side → no credit
if let (Some(a), Some(b)) = (base.get(name), cand.get(name)) {
if name == "class" {
got += w * token_jaccard(a, b);
} else if a == b {
got += w;
}
}
}
if total == 0.0 {
1.0
} else {
got / total
}
}
fn parent_sibling_score(base: &ElementFingerprint, cand: &ElementFingerprint) -> f64 {
// Split the 0.10 budget: parent tag 0.4, parent text 0.3, sibling pos 0.3.
let parent_tag = if base.parent_tag == cand.parent_tag {
1.0
} else {
0.0
};
let parent_text = string_similarity(&base.parent_text, &cand.parent_text);
let span = base.sibling_count.max(1) as f64;
let delta = (base.sibling_index as i64 - cand.sibling_index as i64).unsigned_abs() as f64;
let sibling = 1.0 - (delta / span).min(1.0);
0.4 * parent_tag + 0.3 * parent_text + 0.3 * sibling
}
/// Similarity score in 0..1 between a stored baseline and a candidate element.
pub fn score(base: &ElementFingerprint, cand: &ElementFingerprint) -> f64 {
let tag = if base.tag == cand.tag { 1.0 } else { 0.0 };
let text = string_similarity(&base.text, &cand.text);
let attrs = attr_score(&base.attrs, &cand.attrs);
let ancestors = lcs_ratio(&base.ancestors, &cand.ancestors);
let parent_sibling = parent_sibling_score(base, cand);
W_TAG * tag
+ W_TEXT * text
+ W_ATTRS * attrs
+ W_ANCESTORS * ancestors
+ W_PARENT_SIBLING * parent_sibling
}
/// Why a relocation was rejected.
#[derive(Debug, Clone, PartialEq)]
pub enum RejectReason {
/// No candidates to score.
NoCandidates,
/// Best score below [`ADAPTIVE_THRESHOLD`].
LowScore { best: f64 },
/// Best score too close to the runner-up (below [`ADAPTIVE_MARGIN`]).
Ambiguous { best: f64, second: f64 },
}
/// A successful relocation decision.
#[derive(Debug, Clone, PartialEq)]
pub struct Relocation {
/// Chosen candidate's backend node id.
pub backend_node_id: i64,
/// Winning score.
pub score: f64,
/// Runner-up score (0.0 when there was only one candidate).
pub second_score: f64,
}
/// Pick the best candidate, accepting only when confident. `candidates` is a
/// list of `(backend_node_id, fingerprint)` for the current page.
pub fn pick_best(
base: &ElementFingerprint,
candidates: &[(i64, ElementFingerprint)],
threshold: f64,
margin: f64,
) -> Result<Relocation, RejectReason> {
if candidates.is_empty() {
return Err(RejectReason::NoCandidates);
}
let mut scored: Vec<(i64, f64)> = candidates
.iter()
.map(|(id, fp)| (*id, score(base, fp)))
.collect();
// Highest score first; stable enough for deterministic ties.
scored.sort_by(|a, b| b.1.partial_cmp(&a.1).unwrap_or(std::cmp::Ordering::Equal));
let (best_id, best) = scored[0];
let second = scored.get(1).map(|(_, s)| *s).unwrap_or(0.0);
if best < threshold {
return Err(RejectReason::LowScore { best });
}
if best - second < margin {
return Err(RejectReason::Ambiguous { best, second });
}
Ok(Relocation {
backend_node_id: best_id,
score: best,
second_score: second,
})
}
#[cfg(test)]
mod tests {
use super::*;
fn fp(tag: &str, text: &str, attrs: &[(&str, &str)]) -> ElementFingerprint {
ElementFingerprint {
tag: tag.to_string(),
text: text.to_string(),
attrs: attrs
.iter()
.map(|(k, v)| (k.to_string(), v.to_string()))
.collect(),
..Default::default()
}
}
#[test]
fn identical_fingerprints_score_one() {
let a = fp("button", "Submit", &[("id", "go"), ("class", "btn primary")]);
assert!((score(&a, &a) - 1.0).abs() < 1e-9);
}
#[test]
fn different_tag_caps_score_below_threshold() {
let a = fp("button", "Submit", &[("id", "go")]);
let b = fp("a", "Submit", &[("id", "go")]);
// Same text + same attrs but different role: must lose the role weight
// (W_TAG = 0.20), landing around 0.80 and below a perfect match.
let s = score(&a, &b);
assert!(s < 0.85 && s > 0.75, "got {s}");
}
#[test]
fn string_similarity_basics() {
assert_eq!(string_similarity("abc", "abc"), 1.0);
assert_eq!(string_similarity("", ""), 1.0);
assert!(string_similarity("Submit", "Submit now") > 0.5);
assert!(string_similarity("Add post", "Post all") < 0.6);
}
#[test]
fn class_uses_token_overlap() {
let a = fp("div", "", &[("class", "card primary big")]);
let b = fp("div", "", &[("class", "card primary")]);
// partial class overlap should still score high (tag+text match, attrs partial)
let s = score(&a, &b);
assert!(s > 0.85, "got {s}");
}
#[test]
fn ancestors_lcs() {
let mut a = fp("button", "OK", &[]);
let mut b = fp("button", "OK", &[]);
a.ancestors = vec!["form#f".into(), "div.col".into(), "body".into()];
// b wrapped in an extra div — DOM path changed but mostly preserved
b.ancestors = vec!["form#f".into(), "div.wrap".into(), "div.col".into(), "body".into()];
let s = score(&a, &b);
assert!(s > 0.85, "got {s}");
}
#[test]
fn pick_best_accepts_clear_winner() {
let base = fp("button", "Submit", &[("id", "go")]);
let winner = fp("button", "Submit", &[("id", "go")]);
let other = fp("a", "Home", &[("href", "/")]);
let out = pick_best(
&base,
&[(10, other), (20, winner)],
ADAPTIVE_THRESHOLD,
ADAPTIVE_MARGIN,
)
.expect("should accept");
assert_eq!(out.backend_node_id, 20);
assert!(out.score > out.second_score);
}
#[test]
fn pick_best_rejects_ambiguous_twins() {
let base = fp("button", "Delete", &[("class", "btn danger")]);
// Two near-identical delete buttons — must refuse to guess.
let twin_a = fp("button", "Delete", &[("class", "btn danger")]);
let twin_b = fp("button", "Delete", &[("class", "btn danger")]);
let err = pick_best(
&base,
&[(1, twin_a), (2, twin_b)],
ADAPTIVE_THRESHOLD,
ADAPTIVE_MARGIN,
)
.unwrap_err();
assert!(matches!(err, RejectReason::Ambiguous { .. }), "got {err:?}");
}
#[test]
fn pick_best_rejects_low_score() {
let base = fp("button", "Submit order", &[("id", "checkout")]);
let junk = fp("span", "unrelated footer text", &[("class", "muted")]);
let err = pick_best(&base, &[(1, junk)], ADAPTIVE_THRESHOLD, ADAPTIVE_MARGIN).unwrap_err();
assert!(matches!(err, RejectReason::LowScore { .. }), "got {err:?}");
}
#[test]
fn pick_best_no_candidates() {
let base = fp("button", "x", &[]);
assert_eq!(
pick_best(&base, &[], ADAPTIVE_THRESHOLD, ADAPTIVE_MARGIN).unwrap_err(),
RejectReason::NoCandidates
);
}
}
+33 -6
View File
@@ -305,6 +305,22 @@ pub struct BrowserManager {
/// Origins visited during this session, used by save_state to collect cross-origin localStorage.
visited_origins: HashSet<String>,
next_tab_id: u32,
/// Whether to enable the CDP `Runtime` domain (console / error / exception capture).
/// OFF by default for stealth: a live `Runtime.enable` is a detectable CDP signal
/// (the patchright / rebrowser "runtime leak") — even when attached to the user's
/// real Chrome. Opt in via `AGENT_BROWSER_CAPTURE_CONSOLE=1` when you need the
/// `console` / `errors` commands to return page output.
pub capture_console: bool,
}
/// Whether console/error capture (and thus `Runtime.enable`) is opted into for this
/// daemon. Defaults to `false` so the common automation path leaves no Runtime-domain
/// fingerprint. Set `AGENT_BROWSER_CAPTURE_CONSOLE=1` (or `true`) to turn it on.
pub fn console_capture_enabled() -> bool {
std::env::var("AGENT_BROWSER_CAPTURE_CONSOLE")
.ok()
.map(|v| v == "1" || v.eq_ignore_ascii_case("true"))
.unwrap_or(false)
}
const LIGHTPANDA_CDP_CONNECT_TIMEOUT: Duration = Duration::from_secs(5);
@@ -413,6 +429,7 @@ impl BrowserManager {
ignore_https_errors,
visited_origins: HashSet::new(),
next_tab_id: 1,
capture_console: console_capture_enabled(),
};
manager.discover_and_attach_targets().await?;
manager
@@ -502,6 +519,7 @@ impl BrowserManager {
ignore_https_errors: false,
visited_origins: HashSet::new(),
next_tab_id: 1,
capture_console: console_capture_enabled(),
};
if direct_page {
@@ -629,9 +647,14 @@ impl BrowserManager {
self.client
.send_command_no_params("Page.enable", Some(session_id))
.await?;
self.client
.send_command_no_params("Runtime.enable", Some(session_id))
.await?;
// `Runtime.enable` leaves a detectable CDP signal (the patchright/rebrowser
// "runtime leak"), so only enable it when console/error capture is opted in.
// `Runtime.evaluate` / `Runtime.callFunctionOn` work fine without it.
if self.capture_console {
self.client
.send_command_no_params("Runtime.enable", Some(session_id))
.await?;
}
// Resume the target if it is paused waiting for the debugger.
// This is needed for real browser sessions (Chrome 144+) where targets
// are paused after attach until explicitly resumed. No-op otherwise.
@@ -665,9 +688,12 @@ impl BrowserManager {
self.client
.send_command_no_params("Page.enable", None)
.await?;
self.client
.send_command_no_params("Runtime.enable", None)
.await?;
// See `enable_domains`: `Runtime.enable` is a CDP fingerprint, gated on opt-in.
if self.capture_console {
self.client
.send_command_no_params("Runtime.enable", None)
.await?;
}
let _ = self
.client
.send_command_no_params("Runtime.runIfWaitingForDebugger", None)
@@ -1659,6 +1685,7 @@ async fn initialize_lightpanda_manager(
ignore_https_errors: false,
visited_origins: HashSet::new(),
next_tab_id: 1,
capture_console: console_capture_enabled(),
};
match discover_and_attach_lightpanda_targets(&mut manager, deadline).await {
+69
View File
@@ -146,6 +146,30 @@ struct ChromeArgs {
temp_user_data_dir: Option<PathBuf>,
}
/// Decide the `--force-webrtc-ip-handling-policy` value, if any, for a launched
/// Chrome. Returns `None` to leave WebRTC at Chrome's default behavior.
fn webrtc_ip_handling_policy(has_proxy: bool) -> Option<&'static str> {
let opt_in = std::env::var("AGENT_BROWSER_BLOCK_WEBRTC").ok();
let explicitly_off = opt_in
.as_deref()
.is_some_and(|v| v == "0" || v.eq_ignore_ascii_case("false"));
if explicitly_off {
return None;
}
let explicitly_on = opt_in
.as_deref()
.is_some_and(|v| v == "1" || v.eq_ignore_ascii_case("true"));
if has_proxy {
// Force all WebRTC UDP through the proxy so the real IP can't leak.
Some("disable_non_proxied_udp")
} else if explicitly_on {
// No proxy, but the user asked to hide the local network IP.
Some("default_public_interface_only")
} else {
None
}
}
fn build_chrome_args(options: &LaunchOptions) -> Result<ChromeArgs, String> {
let mut args = vec![
"--remote-debugging-port=0".to_string(),
@@ -204,6 +228,20 @@ fn build_chrome_args(options: &LaunchOptions) -> Result<ChromeArgs, String> {
args.push(format!("--proxy-bypass-list={}", bypass));
}
// WebRTC IP-leak handling. WebRTC enumerates ICE candidates that can expose
// the machine's real local/public IP even when HTTP traffic goes through a
// proxy — defeating the proxy. `--force-webrtc-ip-handling-policy` is a real
// Chrome privacy switch (no detectable JS lie), applied here for launched
// Chrome only (an attached real Chrome keeps the user's own flags).
// - proxy set -> `disable_non_proxied_udp`: force WebRTC through
// the proxy so the real IP can't leak.
// - AGENT_BROWSER_BLOCK_WEBRTC=1 (no proxy) -> `default_public_interface_only`:
// hide the local network IP (Brave/uBlock default).
// Opt out entirely with AGENT_BROWSER_BLOCK_WEBRTC=0.
if let Some(policy) = webrtc_ip_handling_policy(options.proxy.is_some()) {
args.push(format!("--force-webrtc-ip-handling-policy={}", policy));
}
let (user_data_dir, temp_user_data_dir) = if let Some(ref profile) = options.profile {
let expanded = expand_tilde(profile);
let dir = PathBuf::from(&expanded);
@@ -1343,6 +1381,37 @@ mod tests {
use super::*;
use crate::test_utils::EnvGuard;
#[test]
fn webrtc_policy_forces_proxy_when_proxy_set() {
let g = EnvGuard::new(&["AGENT_BROWSER_BLOCK_WEBRTC"]);
g.remove("AGENT_BROWSER_BLOCK_WEBRTC");
// Proxy set, no env: always force WebRTC through the proxy.
assert_eq!(
webrtc_ip_handling_policy(true),
Some("disable_non_proxied_udp")
);
// No proxy, no env: leave WebRTC at Chrome's default.
assert_eq!(webrtc_ip_handling_policy(false), None);
}
#[test]
fn webrtc_policy_opt_in_and_opt_out() {
let g = EnvGuard::new(&["AGENT_BROWSER_BLOCK_WEBRTC"]);
g.set("AGENT_BROWSER_BLOCK_WEBRTC", "1");
assert_eq!(
webrtc_ip_handling_policy(false),
Some("default_public_interface_only")
);
// Explicit opt-out wins even when a proxy is set.
g.set("AGENT_BROWSER_BLOCK_WEBRTC", "0");
assert_eq!(webrtc_ip_handling_policy(true), None);
assert_eq!(webrtc_ip_handling_policy(false), None);
g.remove("AGENT_BROWSER_BLOCK_WEBRTC");
}
#[cfg(unix)]
fn spawn_noop_child() -> Child {
Command::new("/bin/sh")
+107 -23
View File
@@ -2,6 +2,7 @@ use std::collections::HashMap;
use serde_json::Value;
use super::adaptive::{self, ElementFingerprint};
use super::cdp::client::CdpClient;
use super::cdp::types::*;
@@ -13,6 +14,9 @@ pub struct RefEntry {
pub nth: Option<usize>,
pub selector: Option<String>,
pub frame_id: Option<String>,
/// AX fingerprint captured at snapshot time, used by adaptive relocation when
/// the node is gone and the role/name/nth re-query also fails.
pub fingerprint: Option<ElementFingerprint>,
}
pub struct RefMap {
@@ -57,10 +61,19 @@ impl RefMap {
nth,
selector: None,
frame_id: frame_id.map(|s| s.to_string()),
fingerprint: None,
},
);
}
/// Attach an AX fingerprint to an existing ref (set during snapshot, used by
/// adaptive relocation). No-op if the ref is unknown.
pub fn set_fingerprint(&mut self, ref_id: &str, fingerprint: ElementFingerprint) {
if let Some(entry) = self.map.get_mut(ref_id) {
entry.fingerprint = Some(fingerprint);
}
}
pub fn add_selector(
&mut self,
ref_id: String,
@@ -78,6 +91,7 @@ impl RefMap {
nth,
selector: Some(selector),
frame_id: None,
fingerprint: None,
},
);
}
@@ -146,6 +160,46 @@ pub fn parse_ref(input: &str) -> Option<String> {
None
}
/// When a saved `@ref`'s node is gone and the role/name/nth re-query also failed,
/// try to relocate the element by AX fingerprint similarity. Returns the chosen
/// backend node id only when confident (high score + clear margin over the
/// runner-up). Opt out with `AGENT_BROWSER_ADAPTIVE_REF=0`.
async fn relocate_stale_ref(
client: &CdpClient,
ref_id: &str,
entry: &RefEntry,
session_id: &str,
iframe_sessions: &HashMap<String, String>,
) -> Option<i64> {
if std::env::var("AGENT_BROWSER_ADAPTIVE_REF").as_deref() == Ok("0") {
return None;
}
let baseline = entry.fingerprint.as_ref()?;
let candidates = super::snapshot::collect_current_fingerprints(
client,
session_id,
entry.frame_id.as_deref(),
iframe_sessions,
)
.await
.ok()?;
match adaptive::pick_best(
baseline,
&candidates,
adaptive::ADAPTIVE_THRESHOLD,
adaptive::ADAPTIVE_MARGIN,
) {
Ok(reloc) => {
eprintln!(
"[adaptive] relocated {ref_id} ({} \"{}\") score={:.2} second={:.2} -> backendNodeId {}",
entry.role, entry.name, reloc.score, reloc.second_score, reloc.backend_node_id
);
Some(reloc.backend_node_id)
}
Err(_) => None,
}
}
pub async fn resolve_element_center(
client: &CdpClient,
session_id: &str,
@@ -163,15 +217,19 @@ pub async fn resolve_element_center(
// Try cached backend_node_id first (fast path)
if let Some(backend_node_id) = entry.backend_node_id {
let mut active_id = backend_node_id;
// Identity check: React often re-uses the same DOM node when
// re-rendering — backendNodeId stays the same but accessibleName
// / role changes. Without this verification, `click @e20` (saved
// when the button said "Add post") happily clicks the *same*
// node that now says "Post all", silently submitting the thread.
//
// Set AGENT_BROWSER_VERIFY_REF=0 to skip (saves one CDP
// roundtrip per ref-based interaction; only safe if you know
// the page is static between snapshot and click).
// On mismatch, try adaptive fingerprint relocation before failing:
// a confident high-score/high-margin match is a stronger identity
// signal than role+name, and lets a moved+renamed element still
// resolve. If relocation isn't confident, surface the original
// identity error. Set AGENT_BROWSER_VERIFY_REF=0 to skip the check
// (and thus relocation) entirely.
if std::env::var("AGENT_BROWSER_VERIFY_REF").as_deref() != Ok("0") {
if let Err(e) = verify_ref_identity(
client,
@@ -183,7 +241,12 @@ pub async fn resolve_element_center(
)
.await
{
return Err(e);
match relocate_stale_ref(client, &ref_id, entry, session_id, iframe_sessions)
.await
{
Some(id) => active_id = id,
None => return Err(e),
}
}
}
@@ -191,7 +254,7 @@ pub async fn resolve_element_center(
.send_command_typed(
"DOM.getBoxModel",
&DomGetBoxModelParams {
backend_node_id: Some(backend_node_id),
backend_node_id: Some(active_id),
node_id: None,
object_id: None,
},
@@ -213,15 +276,9 @@ pub async fn resolve_element_center(
//
// Set AGENT_BROWSER_VERIFY_CLICK_TARGET=0 to skip.
if std::env::var("AGENT_BROWSER_VERIFY_CLICK_TARGET").as_deref() != Ok("0") {
if let Err(e) = verify_click_target(
client,
effective_session_id,
backend_node_id,
&ref_id,
x,
y,
)
.await
if let Err(e) =
verify_click_target(client, effective_session_id, active_id, &ref_id, x, y)
.await
{
return Err(e);
}
@@ -231,8 +288,9 @@ pub async fn resolve_element_center(
// backend_node_id is stale; re-query the accessibility tree below
}
// Fallback: re-query the accessibility tree to find a fresh node by role/name
let fresh_id = find_node_id_by_role_name(
// Fallback: re-query the accessibility tree to find a fresh node by role/name.
// If that fails, try adaptive fingerprint relocation before giving up.
let fresh_id = match find_node_id_by_role_name(
client,
session_id,
&entry.role,
@@ -241,7 +299,16 @@ pub async fn resolve_element_center(
entry.frame_id.as_deref(),
iframe_sessions,
)
.await?;
.await
{
Ok(id) => id,
Err(e) => match relocate_stale_ref(client, &ref_id, entry, session_id, iframe_sessions)
.await
{
Some(id) => id,
None => return Err(e),
},
};
let result: DomGetBoxModelResult = client
.send_command_typed(
"DOM.getBoxModel",
@@ -279,9 +346,11 @@ pub async fn resolve_element_object_id(
// Try cached backend_node_id first (fast path)
if let Some(backend_node_id) = entry.backend_node_id {
let mut active_id = backend_node_id;
// Same identity guard as resolve_element_center — see that
// function for why React DOM-node-reuse breaks ref-based
// interactions if we skip this.
// interactions if we skip this, and why a confident adaptive
// relocation is allowed to override an identity mismatch.
if std::env::var("AGENT_BROWSER_VERIFY_REF").as_deref() != Ok("0") {
if let Err(e) = verify_ref_identity(
client,
@@ -293,7 +362,12 @@ pub async fn resolve_element_object_id(
)
.await
{
return Err(e);
match relocate_stale_ref(client, &ref_id, entry, session_id, iframe_sessions)
.await
{
Some(id) => active_id = id,
None => return Err(e),
}
}
}
@@ -301,7 +375,7 @@ pub async fn resolve_element_object_id(
.send_command_typed(
"DOM.resolveNode",
&DomResolveNodeParams {
backend_node_id: Some(backend_node_id),
backend_node_id: Some(active_id),
node_id: None,
object_group: Some("agent-browser".to_string()),
},
@@ -317,8 +391,9 @@ pub async fn resolve_element_object_id(
// backend_node_id is stale; re-query the accessibility tree below
}
// Fallback: re-query the accessibility tree to find a fresh node by role/name
let fresh_id = find_node_id_by_role_name(
// Fallback: re-query the accessibility tree to find a fresh node by role/name.
// If that fails, try adaptive fingerprint relocation before giving up.
let fresh_id = match find_node_id_by_role_name(
client,
session_id,
&entry.role,
@@ -327,7 +402,16 @@ pub async fn resolve_element_object_id(
entry.frame_id.as_deref(),
iframe_sessions,
)
.await?;
.await
{
Ok(id) => id,
Err(e) => match relocate_stale_ref(client, &ref_id, entry, session_id, iframe_sessions)
.await
{
Some(id) => id,
None => return Err(e),
},
};
let result: DomResolveNodeResult = client
.send_command_typed(
"DOM.resolveNode",
+120 -2
View File
@@ -15,7 +15,111 @@ pub async fn click(
click_count: i32,
iframe_sessions: &HashMap<String, String>,
) -> Result<(), String> {
let (x, y, effective_session_id) = resolve_element_center(
// AGENT_BROWSER_CLICK_MODE: "" (default) = coordinate click with a DOM
// fallback; "coord" = strict coordinate only (no fallback); "dom" = always
// dispatch through the DOM.
let mode = std::env::var("AGENT_BROWSER_CLICK_MODE").unwrap_or_default();
// (A) Scroll the target into view first so the computed coordinates land
// inside the viewport. Without this, an element below the fold (or revealed
// after scroll/popup) yields off-viewport coordinates and the click lands on
// whatever currently occupies that point. Best-effort: ignore failures.
scroll_into_view_if_needed(client, session_id, ref_map, selector_or_ref, iframe_sessions).await;
if mode == "dom" {
return dom_click(client, session_id, ref_map, selector_or_ref, iframe_sessions).await;
}
let resolved = resolve_element_center(
client,
session_id,
ref_map,
selector_or_ref,
iframe_sessions,
)
.await;
match resolved {
Ok((x, y, effective_session_id)) => {
dispatch_click(client, &effective_session_id, x, y, button, click_count).await
}
Err(e) => {
// (B) The coordinate path failed — typically a persistent overlay
// failing the occlusion guard, or coordinates that won't resolve.
// Fall back to a DOM-dispatched `.click()` on the intended element,
// which targets the element directly instead of a screen point.
// Skipped for strict "coord" mode and for non-left / multi-clicks
// (a DOM `.click()` can't express right/middle/double semantics).
if mode == "coord" || button != "left" || click_count != 1 {
return Err(e);
}
eprintln!(
"[click] coordinate click failed ({e}); falling back to DOM dispatch \
(set AGENT_BROWSER_CLICK_MODE=coord to disable)"
);
dom_click(client, session_id, ref_map, selector_or_ref, iframe_sessions)
.await
.map_err(|dom_err| format!("{e}\n(DOM-dispatch fallback also failed: {dom_err})"))
}
}
}
/// Best-effort scroll-into-view before a coordinate click. Uses Chrome's
/// `scrollIntoViewIfNeeded` (only scrolls when not already fully visible),
/// falling back to centered `scrollIntoView`. Resolution failures are ignored —
/// the subsequent resolve will surface a real "not found" error.
async fn scroll_into_view_if_needed(
client: &CdpClient,
session_id: &str,
ref_map: &RefMap,
selector_or_ref: &str,
iframe_sessions: &HashMap<String, String>,
) {
let Ok((object_id, effective_session_id)) = resolve_element_object_id(
client,
session_id,
ref_map,
selector_or_ref,
iframe_sessions,
)
.await
else {
return;
};
let js = "function() { try { \
if (typeof this.scrollIntoViewIfNeeded === 'function') { this.scrollIntoViewIfNeeded(true); } \
else { this.scrollIntoView({ block: 'center', inline: 'center' }); } \
} catch (e) {} }";
let _ = client
.send_command_typed::<_, Value>(
"Runtime.callFunctionOn",
&CallFunctionOnParams {
function_declaration: js.to_string(),
object_id: Some(object_id),
arguments: None,
return_by_value: Some(true),
await_promise: Some(false),
},
Some(&effective_session_id),
)
.await;
// Let the scroll settle so the following getBoxModel sees final coordinates.
wait_for_paint_settled(client, &effective_session_id).await;
}
/// Dispatch a click through the DOM (`element.click()`) instead of via screen
/// coordinates. Targets the intended element directly, so it works when a
/// floating layer occludes the click point or the element sits in a portal that
/// confuses `elementFromPoint`. Used as the fallback for `click` and when
/// `AGENT_BROWSER_CLICK_MODE=dom`.
async fn dom_click(
client: &CdpClient,
session_id: &str,
ref_map: &RefMap,
selector_or_ref: &str,
iframe_sessions: &HashMap<String, String>,
) -> Result<(), String> {
let (object_id, effective_session_id) = resolve_element_object_id(
client,
session_id,
ref_map,
@@ -23,7 +127,21 @@ pub async fn click(
iframe_sessions,
)
.await?;
dispatch_click(client, &effective_session_id, x, y, button, click_count).await
client
.send_command_typed::<_, Value>(
"Runtime.callFunctionOn",
&CallFunctionOnParams {
function_declaration: "function() { this.click(); }".to_string(),
object_id: Some(object_id),
arguments: None,
return_by_value: Some(true),
await_promise: Some(false),
},
Some(&effective_session_id),
)
.await?;
wait_for_paint_settled(client, &effective_session_id).await;
Ok(())
}
pub async fn dblclick(
+4
View File
@@ -1,6 +1,8 @@
#[allow(dead_code)]
pub mod actions;
#[allow(dead_code)]
pub mod adaptive;
#[allow(dead_code)]
pub mod auth;
#[allow(dead_code)]
pub mod browser;
@@ -25,6 +27,8 @@ pub mod policy;
#[allow(dead_code)]
pub mod providers;
#[allow(dead_code)]
pub mod relay;
#[allow(dead_code)]
pub mod react;
#[allow(dead_code)]
pub mod recording;
+362
View File
@@ -0,0 +1,362 @@
//! Relay between the `ab-connect` browser extension and the daemon's `CdpClient`.
//!
//! The extension speaks a small CDP-over-WebSocket "envelope" protocol (adapted
//! from openclaw-browser-relay) and drives the user's real tabs via per-tab
//! `chrome.debugger`. The daemon's `CdpClient`, however, expects a **browser-
//! level** CDP endpoint (`Target.getTargets` / `Target.attachToTarget` → a
//! `sessionId`, then per-session commands). This relay bridges the two: it
//! tracks the targets the extension reports, answers the browser-level
//! `Target.*` discovery commands LOCALLY, and forwards everything else to the
//! extension as `forwardCDPCommand`. That keeps `CdpClient` and `browser.rs`
//! unchanged.
//!
//! This module is the pure translation core (no I/O) so the protocol can be
//! unit-tested; the tokio WebSocket server that drives it lives alongside.
use std::collections::HashMap;
use serde_json::{json, Value};
/// Protocol version advertised in the connect handshake (matches the extension).
pub const RELAY_PROTOCOL: i64 = 3;
/// One target (tab) the extension has attached, as the relay tracks it.
#[derive(Clone)]
struct TargetEntry {
session_id: String,
target_info: Value,
}
/// Relay translation state: the set of targets the extension currently exposes.
#[derive(Default)]
pub struct RelayState {
/// targetId -> entry
targets: HashMap<String, TargetEntry>,
}
/// What to do with a raw CDP command received from the `CdpClient`.
#[derive(Debug, PartialEq)]
pub enum ClientRoute {
/// Answer locally; the value is a raw CDP response `{id, result}`.
Local(Value),
/// Forward to the extension; the value is a `forwardCDPCommand` envelope.
Forward(Value),
}
/// An output the relay emits while handling an extension message.
#[derive(Debug, PartialEq)]
pub enum RelayOut {
/// Send this raw CDP message to the `CdpClient`.
ToClient(Value),
/// Send this envelope message back to the extension.
ToExt(Value),
}
impl RelayState {
pub fn new() -> Self {
Self::default()
}
/// The challenge the relay sends to the extension as soon as it connects,
/// kicking off the connect handshake.
pub fn connect_challenge(nonce: &str) -> Value {
json!({ "type": "event", "event": "connect.challenge", "payload": { "nonce": nonce } })
}
/// A keepalive ping for the extension.
pub fn ping() -> Value {
json!({ "method": "ping" })
}
/// Route a raw CDP command `{id, method, params?, sessionId?}` from the
/// `CdpClient`: answer browser-level `Target.*` discovery locally, forward
/// the rest to the extension.
pub fn route_client_command(&self, raw: &Value) -> ClientRoute {
let id = raw.get("id").cloned().unwrap_or(Value::Null);
let method = raw.get("method").and_then(|m| m.as_str()).unwrap_or("");
let params = raw.get("params").cloned().unwrap_or_else(|| json!({}));
let session_id = raw.get("sessionId").and_then(|s| s.as_str());
match method {
// Discovery is best-effort and event-driven in real CDP; abs only
// reads the getTargets result, so an empty ack is enough here.
"Target.setDiscoverTargets" | "Target.setAutoAttach" => {
ClientRoute::Local(json!({ "id": id, "result": {} }))
}
"Target.getTargets" => {
let infos: Vec<Value> =
self.targets.values().map(|t| t.target_info.clone()).collect();
ClientRoute::Local(json!({ "id": id, "result": { "targetInfos": infos } }))
}
"Target.attachToTarget" => {
let target_id = params.get("targetId").and_then(|t| t.as_str()).unwrap_or("");
match self.targets.get(target_id) {
Some(entry) => ClientRoute::Local(
json!({ "id": id, "result": { "sessionId": entry.session_id } }),
),
None => ClientRoute::Local(json!({
"id": id,
"error": { "code": -32602, "message": format!("No such target {target_id}") }
})),
}
}
// Everything else goes to the extension's chrome.debugger.
_ => ClientRoute::Forward(json!({
"id": id,
"method": "forwardCDPCommand",
"params": { "method": method, "params": params, "sessionId": session_id },
})),
}
}
/// Handle one decoded message from the extension. Updates target state and
/// returns the messages to emit (to the client and/or back to the
/// extension). `expected_token` is matched against the connect handshake.
pub fn handle_ext_message(&mut self, msg: &Value, expected_token: &str) -> Vec<RelayOut> {
// Connect handshake request from the extension.
if msg.get("type").and_then(|t| t.as_str()) == Some("req")
&& msg.get("method").and_then(|m| m.as_str()) == Some("connect")
{
let id = msg.get("id").cloned().unwrap_or(Value::Null);
let token = msg
.get("params")
.and_then(|p| p.get("auth"))
.and_then(|a| a.get("token"))
.and_then(|t| t.as_str())
.unwrap_or("");
let ok = !expected_token.is_empty() && token == expected_token;
let mut res = json!({ "type": "res", "id": id, "ok": ok });
if !ok {
res["error"] = json!({ "message": "invalid relay token" });
}
return vec![RelayOut::ToExt(res)];
}
// Keepalive.
if msg.get("method").and_then(|m| m.as_str()) == Some("pong") {
return vec![];
}
// Response to a forwardCDPCommand we sent → raw CDP response to client.
if msg.get("id").is_some()
&& (msg.get("result").is_some() || msg.get("error").is_some())
&& msg.get("method").is_none()
{
let mut out = json!({ "id": msg.get("id").cloned().unwrap_or(Value::Null) });
if let Some(r) = msg.get("result") {
out["result"] = r.clone();
}
if let Some(e) = msg.get("error") {
// CdpClient expects an error object; wrap a bare string.
out["error"] = match e {
Value::String(s) => json!({ "code": -32000, "message": s }),
other => other.clone(),
};
}
return vec![RelayOut::ToClient(out)];
}
// CDP event forwarded from a tab.
if msg.get("method").and_then(|m| m.as_str()) == Some("forwardCDPEvent") {
let p = msg.get("params").cloned().unwrap_or_else(|| json!({}));
let inner_method = p.get("method").and_then(|m| m.as_str()).unwrap_or("");
let inner_params = p.get("params").cloned().unwrap_or_else(|| json!({}));
let session_id = p.get("sessionId").and_then(|s| s.as_str());
// Learn/forget targets from the extension's synthesized Target events.
// We consume these to maintain state and do NOT forward them: abs
// discovers targets by pulling getTargets, and forwarding a second
// attachedToTarget would duplicate the one attachToTarget emits.
match inner_method {
"Target.attachedToTarget" => {
if let Some(info) = inner_params.get("targetInfo") {
if let Some(tid) = info.get("targetId").and_then(|t| t.as_str()) {
let sid = inner_params
.get("sessionId")
.and_then(|s| s.as_str())
.unwrap_or("")
.to_string();
self.targets.insert(
tid.to_string(),
TargetEntry { session_id: sid, target_info: info.clone() },
);
}
}
return vec![];
}
"Target.detachedFromTarget" => {
let gone = inner_params.get("sessionId").and_then(|s| s.as_str());
if let Some(gone) = gone {
self.targets.retain(|_, e| e.session_id != gone);
}
return vec![];
}
_ => {}
}
// Regular CDP event → deliver to the client with its sessionId.
let mut ev = json!({ "method": inner_method, "params": inner_params });
if let Some(sid) = session_id {
ev["sessionId"] = json!(sid);
}
return vec![RelayOut::ToClient(ev)];
}
vec![]
}
#[cfg(test)]
fn seed_target(&mut self, target_id: &str, session_id: &str) {
self.targets.insert(
target_id.to_string(),
TargetEntry {
session_id: session_id.to_string(),
target_info: json!({
"targetId": target_id,
"type": "page",
"title": "",
"url": "about:blank",
"attached": true,
}),
},
);
}
}
#[cfg(test)]
mod tests {
use super::*;
fn attached_event(target_id: &str, session_id: &str) -> Value {
json!({
"method": "forwardCDPEvent",
"params": {
"sessionId": session_id,
"method": "Target.attachedToTarget",
"params": {
"sessionId": session_id,
"targetInfo": { "targetId": target_id, "type": "page", "url": "https://x", "title": "X" }
}
}
})
}
#[test]
fn learns_target_from_attached_event_and_does_not_forward_it() {
let mut s = RelayState::new();
let out = s.handle_ext_message(&attached_event("T1", "cb-tab-1"), "tok");
assert!(out.is_empty(), "attachedToTarget should be consumed, not forwarded");
// Now getTargets must report it.
let route = s.route_client_command(&json!({ "id": 1, "method": "Target.getTargets" }));
match route {
ClientRoute::Local(v) => {
let infos = v["result"]["targetInfos"].as_array().unwrap();
assert_eq!(infos.len(), 1);
assert_eq!(infos[0]["targetId"], "T1");
}
_ => panic!("getTargets must be local"),
}
}
#[test]
fn attach_to_target_returns_known_session() {
let mut s = RelayState::new();
s.seed_target("T1", "cb-tab-1");
let route = s.route_client_command(
&json!({ "id": 5, "method": "Target.attachToTarget", "params": { "targetId": "T1", "flatten": true } }),
);
assert_eq!(
route,
ClientRoute::Local(json!({ "id": 5, "result": { "sessionId": "cb-tab-1" } }))
);
}
#[test]
fn attach_to_unknown_target_errors_locally() {
let s = RelayState::new();
let route = s.route_client_command(
&json!({ "id": 6, "method": "Target.attachToTarget", "params": { "targetId": "nope" } }),
);
match route {
ClientRoute::Local(v) => assert!(v.get("error").is_some()),
_ => panic!("should answer locally"),
}
}
#[test]
fn other_commands_forward_as_envelope() {
let s = RelayState::new();
let route = s.route_client_command(&json!({
"id": 9, "method": "Page.navigate",
"params": { "url": "https://x" }, "sessionId": "cb-tab-1"
}));
match route {
ClientRoute::Forward(v) => {
assert_eq!(v["method"], "forwardCDPCommand");
assert_eq!(v["id"], 9);
assert_eq!(v["params"]["method"], "Page.navigate");
assert_eq!(v["params"]["sessionId"], "cb-tab-1");
assert_eq!(v["params"]["params"]["url"], "https://x");
}
_ => panic!("Page.navigate must forward"),
}
}
#[test]
fn forward_command_response_becomes_client_response() {
let mut s = RelayState::new();
let out = s.handle_ext_message(&json!({ "id": 9, "result": { "frameId": "F1" } }), "tok");
assert_eq!(
out,
vec![RelayOut::ToClient(json!({ "id": 9, "result": { "frameId": "F1" } }))]
);
}
#[test]
fn forward_command_error_is_wrapped() {
let mut s = RelayState::new();
let out = s.handle_ext_message(&json!({ "id": 9, "error": "boom" }), "tok");
match &out[0] {
RelayOut::ToClient(v) => {
assert_eq!(v["id"], 9);
assert_eq!(v["error"]["message"], "boom");
}
_ => panic!("expected ToClient"),
}
}
#[test]
fn regular_event_is_forwarded_with_session() {
let mut s = RelayState::new();
let ev = json!({
"method": "forwardCDPEvent",
"params": { "sessionId": "cb-tab-1", "method": "Page.loadEventFired", "params": { "timestamp": 1.0 } }
});
let out = s.handle_ext_message(&ev, "tok");
assert_eq!(
out,
vec![RelayOut::ToClient(json!({
"method": "Page.loadEventFired",
"params": { "timestamp": 1.0 },
"sessionId": "cb-tab-1"
}))]
);
}
#[test]
fn connect_handshake_validates_token() {
let mut s = RelayState::new();
let req = json!({ "type": "req", "id": "c1", "method": "connect", "params": { "auth": { "token": "good" } } });
let ok = s.handle_ext_message(&req, "good");
assert_eq!(ok, vec![RelayOut::ToExt(json!({ "type": "res", "id": "c1", "ok": true }))]);
let bad = s.handle_ext_message(&req, "different");
match &bad[0] {
RelayOut::ToExt(v) => {
assert_eq!(v["ok"], false);
assert!(v.get("error").is_some());
}
_ => panic!("expected ToExt"),
}
}
}
+118
View File
@@ -6,6 +6,7 @@ use super::cdp::client::CdpClient;
use super::cdp::types::{
AXNode, AXProperty, AXValue, EvaluateParams, EvaluateResult, GetFullAXTreeResult,
};
use super::adaptive::ElementFingerprint;
use super::element::{resolve_ax_session, RefMap};
const INTERACTIVE_ROLES: &[&str] = &[
@@ -148,6 +149,122 @@ impl TreeNode {
}
}
/// Build an AX fingerprint for a tree node, used by adaptive @ref relocation.
/// Pulls only data already in the AX tree (no extra CDP calls): role as `tag`,
/// accessible name as `text`, a few discriminating AX properties as `attrs`, and
/// the ancestor/parent/sibling structure from the tree links.
fn build_ax_fingerprint(tree_nodes: &[TreeNode], idx: usize) -> ElementFingerprint {
let node = &tree_nodes[idx];
let mut attrs = std::collections::BTreeMap::new();
if let Some(v) = &node.value_text {
if !v.is_empty() {
attrs.insert("value".to_string(), v.clone());
}
}
if let Some(u) = &node.url {
if !u.is_empty() {
attrs.insert("url".to_string(), u.clone());
}
}
if let Some(l) = node.level {
attrs.insert("level".to_string(), l.to_string());
}
if let Some(c) = &node.checked {
attrs.insert("checked".to_string(), c.clone());
}
// Ancestor roles, nearest first, capped to keep the signature stable.
let mut ancestors = Vec::new();
let mut cur = node.parent_idx;
while let Some(pidx) = cur {
if ancestors.len() >= 6 {
break;
}
let role = tree_nodes[pidx].role.clone();
if !role.is_empty() {
ancestors.push(role);
}
cur = tree_nodes[pidx].parent_idx;
}
let (parent_tag, parent_text) = node
.parent_idx
.map(|pidx| (tree_nodes[pidx].role.clone(), tree_nodes[pidx].name.clone()))
.unwrap_or_default();
// Position among same-role siblings under the same parent.
let (sibling_index, sibling_count) = match node.parent_idx {
Some(pidx) => {
let mut count = 0u32;
let mut index = 0u32;
for &child in &tree_nodes[pidx].children {
if tree_nodes[child].role == node.role {
if child == idx {
index = count;
}
count += 1;
}
}
(index, count)
}
None => (0, 0),
};
ElementFingerprint {
tag: node.role.clone(),
text: node.name.clone(),
attrs,
ancestors,
parent_tag,
parent_text,
sibling_index,
sibling_count,
}
}
/// Collect AX fingerprints for every node that has a backend node id, used as the
/// candidate set when relocating a stale @ref. Reuses the same extraction as the
/// baseline so the two are scored in the same space.
fn collect_fingerprints(tree_nodes: &[TreeNode]) -> Vec<(i64, ElementFingerprint)> {
tree_nodes
.iter()
.enumerate()
.filter_map(|(idx, n)| {
n.backend_node_id
.map(|bid| (bid, build_ax_fingerprint(tree_nodes, idx)))
})
.collect()
}
/// Fetch a fresh AX tree for the given frame and return `(backend_node_id,
/// fingerprint)` for every node — the candidate set for adaptive @ref
/// relocation. One `getFullAXTree` call, no per-element work.
pub(super) async fn collect_current_fingerprints(
client: &CdpClient,
session_id: &str,
frame_id: Option<&str>,
iframe_sessions: &HashMap<String, String>,
) -> Result<Vec<(i64, ElementFingerprint)>, String> {
let (ax_params, effective_session_id) =
resolve_ax_session(frame_id, session_id, iframe_sessions);
let _ = client
.send_command_no_params("DOM.enable", Some(effective_session_id))
.await;
let _ = client
.send_command_no_params("Accessibility.enable", Some(effective_session_id))
.await;
let ax_tree: GetFullAXTreeResult = client
.send_command_typed(
"Accessibility.getFullAXTree",
&ax_params,
Some(effective_session_id),
)
.await?;
let (tree_nodes, _roots) = build_tree(&ax_tree.nodes);
Ok(collect_fingerprints(&tree_nodes))
}
/// The type of a hidden form input found inside a cursor-interactive element.
#[derive(Clone, Copy)]
enum HiddenInputKind {
@@ -397,6 +514,7 @@ pub async fn take_snapshot(
actual_nth,
frame_id,
);
ref_map.set_fingerprint(&ref_id, build_ax_fingerprint(&tree_nodes, *idx));
tree_nodes[*idx].has_ref = true;
tree_nodes[*idx].ref_id = Some(ref_id);
+139 -2
View File
@@ -51,13 +51,18 @@ pub fn build_stealth_script(mode: StealthMode, locale: Option<&str>) -> String {
vec![locale, base_lang]
};
let config_line = format!(
r#"const __abStealth = {{ locale: "{}", languages: {}, allowWebGLContextFallback: false }};"#,
r#"const __abStealth = {{ locale: "{}", languages: {}, allowWebGLContextFallback: false, hideCanvas: {}, canvasSeed: {} }};"#,
locale,
serde_json::to_string(&languages).unwrap_or_else(|_| r#"["en-US","en"]"#.to_string()),
hide_canvas_enabled(),
canvas_noise_seed(),
);
// NB: this prefix MUST match the first line of stealth_scripts.js verbatim,
// otherwise the fallback below prepends a SECOND `const __abStealth`
// declaration and the whole script dies with a redeclaration SyntaxError.
if let Some(rest) = STEALTH_SCRIPTS_RAW.strip_prefix(
r#"const __abStealth = { locale: "en-US", languages: ["en-US", "en"], allowWebGLContextFallback: false };"#,
r#"const __abStealth = { locale: "en-US", languages: ["en-US", "en"], allowWebGLContextFallback: false, hideCanvas: false, canvasSeed: 0 };"#,
) {
format!("{}{}", config_line, rest)
} else {
@@ -65,6 +70,35 @@ pub fn build_stealth_script(mode: StealthMode, locale: Option<&str>) -> String {
}
}
/// Whether canvas/audio fingerprint noise is opted into (FullLaunch only).
/// OFF by default: injecting noise is a deliberate "lie" that can itself be a
/// tell, so it's reserved for users who explicitly want it via
/// `AGENT_BROWSER_HIDE_CANVAS=1`.
fn hide_canvas_enabled() -> bool {
std::env::var("AGENT_BROWSER_HIDE_CANVAS")
.ok()
.map(|v| v == "1" || v.eq_ignore_ascii_case("true"))
.unwrap_or(false)
}
/// A per-process seed so canvas/audio noise is STABLE within a session (a real
/// device returns the same hash on repeated reads) but differs from the
/// headless-stable default. 0 is avoided so the JS can treat it as "unset".
fn canvas_noise_seed() -> u32 {
use std::sync::OnceLock;
static SEED: OnceLock<u32> = OnceLock::new();
*SEED.get_or_init(|| {
use std::time::{SystemTime, UNIX_EPOCH};
let nanos = SystemTime::now()
.duration_since(UNIX_EPOCH)
.map(|d| d.subsec_nanos())
.unwrap_or(0x9e3779b9);
// mix the bits a little, then force non-zero
let mixed = nanos ^ nanos.rotate_left(13).wrapping_mul(2654435761);
mixed | 1
})
}
/// Apply stealth patches to a browser session.
///
/// In `CdpAttach` mode (user's real Chrome): only removes `navigator.webdriver`.
@@ -118,11 +152,74 @@ pub async fn apply_stealth(
.await?;
}
}
// Align the timezone for fresh launches when explicitly requested.
// Headless/launched Chrome often reports UTC (or the host's zone), which
// can contradict a proxy's geolocation or a spoofed locale.
// `Emulation.setTimezoneOverride` is a NATIVE override — Intl.DateTimeFormat
// and Date both follow it with no detectable JS lie. Opt-in only:
// AGENT_BROWSER_TIMEZONE=<IANA id> -> use that zone (e.g. align to proxy)
// AGENT_BROWSER_TIMEZONE=auto -> derive a default from the locale
// (unset) -> leave the real timezone untouched
if let Some(tz) = resolve_timezone(locale) {
let _ = client
.send_command(
"Emulation.setTimezoneOverride",
Some(json!({ "timezoneId": tz })),
Some(session_id),
)
.await;
}
}
Ok(())
}
/// Resolve the timezone to emulate for a fresh-launch session, if any.
/// Controlled by `AGENT_BROWSER_TIMEZONE`: an explicit IANA id, or `auto` to
/// derive a sensible default from the locale. Returns `None` (leave the real
/// timezone) when unset, empty, or when `auto` can't map the locale.
fn resolve_timezone(locale: Option<&str>) -> Option<String> {
let raw = std::env::var("AGENT_BROWSER_TIMEZONE").ok()?;
let raw = raw.trim();
if raw.is_empty() {
return None;
}
if raw.eq_ignore_ascii_case("auto") {
return locale
.and_then(locale_default_timezone)
.map(str::to_string);
}
Some(raw.to_string())
}
/// Best-effort IANA timezone for a locale. Used only for
/// `AGENT_BROWSER_TIMEZONE=auto`; unknown locales return `None` so the real
/// timezone is left untouched rather than guessing a wrong one.
fn locale_default_timezone(locale: &str) -> Option<&'static str> {
let tz = match locale.to_ascii_lowercase().as_str() {
"en-us" => "America/New_York",
"en-ca" => "America/Toronto",
"en-gb" => "Europe/London",
"en-au" => "Australia/Sydney",
"ja" | "ja-jp" => "Asia/Tokyo",
"ko" | "ko-kr" => "Asia/Seoul",
"zh-cn" | "zh-hans" | "zh-hans-cn" => "Asia/Shanghai",
"zh-tw" | "zh-hant" | "zh-hant-tw" => "Asia/Taipei",
"zh-hk" => "Asia/Hong_Kong",
"de" | "de-de" => "Europe/Berlin",
"fr" | "fr-fr" => "Europe/Paris",
"es" | "es-es" => "Europe/Madrid",
"it" | "it-it" => "Europe/Rome",
"nl" | "nl-nl" => "Europe/Amsterdam",
"pt-br" => "America/Sao_Paulo",
"pt" | "pt-pt" => "Europe/Lisbon",
"ru" | "ru-ru" => "Europe/Moscow",
_ => return None,
};
Some(tz)
}
/// Get the browser's User-Agent string via CDP.
async fn get_browser_user_agent(client: &CdpClient, session_id: &str) -> Option<String> {
let result = client
@@ -235,3 +332,43 @@ fn build_ua_metadata(ua: &str, locale: Option<&str>) -> serde_json::Value {
"wow64": false,
})
}
#[cfg(test)]
mod timezone_tests {
use super::{locale_default_timezone, resolve_timezone};
#[test]
fn maps_common_locales_case_insensitively() {
assert_eq!(locale_default_timezone("en-US"), Some("America/New_York"));
assert_eq!(locale_default_timezone("ja-JP"), Some("Asia/Tokyo"));
assert_eq!(locale_default_timezone("zh-CN"), Some("Asia/Shanghai"));
assert_eq!(locale_default_timezone("ZH-TW"), Some("Asia/Taipei"));
assert_eq!(locale_default_timezone("ja"), Some("Asia/Tokyo"));
}
#[test]
fn unknown_locale_returns_none() {
assert_eq!(locale_default_timezone("xx-YY"), None);
assert_eq!(locale_default_timezone(""), None);
}
#[test]
fn resolve_timezone_honors_env() {
// Serialized via a single test to avoid cross-test env races on this key.
std::env::remove_var("AGENT_BROWSER_TIMEZONE");
assert_eq!(resolve_timezone(Some("en-US")), None);
std::env::set_var("AGENT_BROWSER_TIMEZONE", "Europe/Berlin");
assert_eq!(resolve_timezone(None), Some("Europe/Berlin".to_string()));
std::env::set_var("AGENT_BROWSER_TIMEZONE", " ");
assert_eq!(resolve_timezone(Some("en-US")), None);
std::env::set_var("AGENT_BROWSER_TIMEZONE", "auto");
assert_eq!(resolve_timezone(Some("ja-JP")), Some("Asia/Tokyo".to_string()));
assert_eq!(resolve_timezone(Some("xx-YY")), None);
assert_eq!(resolve_timezone(None), None);
std::env::remove_var("AGENT_BROWSER_TIMEZONE");
}
}
+124 -1
View File
@@ -1,4 +1,4 @@
const __abStealth = { locale: "en-US", languages: ["en-US", "en"], allowWebGLContextFallback: false };
const __abStealth = { locale: "en-US", languages: ["en-US", "en"], allowWebGLContextFallback: false, hideCanvas: false, canvasSeed: 0 };
(function(){
// Prefer the CDP-level automation override (Emulation.setAutomationOverride),
// which makes navigator.webdriver report `false` NATIVELY — undetectable by
@@ -1275,3 +1275,126 @@ const __abStealth = { locale: "en-US", languages: ["en-US", "en"], allowWebGLCon
}
}
})();
// Canvas + audio fingerprint noise (OPT-IN, full-launch only).
// Headless Chrome produces a stable canvas/audio hash that trackers use as a
// device id. When __abStealth.hideCanvas is on we perturb readback APIs with a
// SESSION-STABLE, sub-perceptual amount of noise: repeated reads on this page
// return the same noised result (a real device is consistent too), but the
// hash differs from the headless default. Off by default — noise is itself a
// "lie", so it's reserved for users who explicitly enable it.
(function(){
if (!__abStealth || __abStealth.hideCanvas !== true) return;
// Deterministic PRNG keyed by the per-session seed plus a position, so the
// same pixel/sample is perturbed identically every read within the session.
const baseSeed = (__abStealth.canvasSeed >>> 0) || 0x9e3779b9;
const noiseAt = (n) => {
let t = (baseSeed ^ Math.imul(n | 0, 0x6d2b79f5)) >>> 0;
t = Math.imul(t ^ (t >>> 15), t | 1) >>> 0;
t ^= t + Math.imul(t ^ (t >>> 7), t | 61);
return ((t ^ (t >>> 14)) >>> 0) / 4294967296;
};
// Make a wrapped function masquerade as the native one (toString + name).
const mask = (wrapped, native) => {
try {
Object.defineProperty(wrapped, 'name', {
value: native.name,
configurable: true,
});
Object.defineProperty(wrapped, 'toString', {
value: () => native.toString(),
configurable: true,
writable: true,
});
} catch {}
return wrapped;
};
// ---- Canvas 2D readback ---------------------------------------------------
const perturbImageData = (imageData) => {
const data = imageData && imageData.data;
if (!data || !data.length) return imageData;
for (let i = 0; i < data.length; i += 4) {
// Touch ~5% of pixels by +/-1 on each RGB channel; leave alpha alone.
if (noiseAt(i) < 0.05) {
const delta = noiseAt(i + 1) < 0.5 ? -1 : 1;
data[i] = Math.max(0, Math.min(255, data[i] + delta));
data[i + 1] = Math.max(0, Math.min(255, data[i + 1] + delta));
data[i + 2] = Math.max(0, Math.min(255, data[i + 2] + delta));
}
}
return imageData;
};
try {
const ctxProto = (typeof CanvasRenderingContext2D !== 'undefined')
? CanvasRenderingContext2D.prototype : null;
if (ctxProto && typeof ctxProto.getImageData === 'function') {
const nativeGetImageData = ctxProto.getImageData;
ctxProto.getImageData = mask(function(...args) {
return perturbImageData(nativeGetImageData.apply(this, args));
}, nativeGetImageData);
}
} catch {}
// For toDataURL/toBlob, draw the (already-rendered) canvas onto a scratch
// canvas, perturb its pixels, then encode that — so the export hash shifts
// without disturbing what the page sees on screen.
const exportNoised = (canvas) => {
try {
const w = canvas.width, h = canvas.height;
if (!w || !h) return null;
const scratch = document.createElement('canvas');
scratch.width = w; scratch.height = h;
const sctx = scratch.getContext('2d');
if (!sctx) return null;
sctx.drawImage(canvas, 0, 0);
const img = sctx.getImageData(0, 0, w, h);
perturbImageData(img);
sctx.putImageData(img, 0, 0);
return scratch;
} catch { return null; }
};
try {
const canvasProto = (typeof HTMLCanvasElement !== 'undefined')
? HTMLCanvasElement.prototype : null;
if (canvasProto && typeof canvasProto.toDataURL === 'function') {
const nativeToDataURL = canvasProto.toDataURL;
canvasProto.toDataURL = mask(function(...args) {
const scratch = exportNoised(this);
return nativeToDataURL.apply(scratch || this, args);
}, nativeToDataURL);
}
if (canvasProto && typeof canvasProto.toBlob === 'function') {
const nativeToBlob = canvasProto.toBlob;
canvasProto.toBlob = mask(function(cb, ...rest) {
const scratch = exportNoised(this);
return nativeToBlob.call(scratch || this, cb, ...rest);
}, nativeToBlob);
}
} catch {}
// ---- AudioBuffer readback -------------------------------------------------
// Perturb time-domain samples by a tiny, seed-stable amount so the audio
// fingerprint (sum/hash of channel data) shifts without audible effect.
try {
const audioProto = (typeof AudioBuffer !== 'undefined') ? AudioBuffer.prototype : null;
if (audioProto && typeof audioProto.getChannelData === 'function') {
const nativeGetChannelData = audioProto.getChannelData;
const seen = new WeakSet();
audioProto.getChannelData = mask(function(...args) {
const channel = nativeGetChannelData.apply(this, args);
// Only perturb once per buffer to keep reads consistent.
if (channel && !seen.has(channel)) {
seen.add(channel);
for (let i = 0; i < channel.length; i += 100) {
channel[i] = channel[i] + (noiseAt(i) - 0.5) * 1e-7;
}
}
return channel;
}, nativeGetChannelData);
}
} catch {}
})();
+11
View File
@@ -0,0 +1,11 @@
# Attribution
The chrome.debugger attach + CDP Target handling in `background.js` is adapted
from **openclaw-browser-relay** by chengyixu
(https://github.com/chengyixu/openclaw-browser-relay, MIT per its README).
Changes for agent-browser-stealth: rebranded to "agent-browser connect"; the
transport is rewritten from a localhost WebSocket + shared token to Chrome
**native messaging** (host `com.agent_browser.connect`) — no port, no token,
Chrome authenticates the extension to the host by id. WebSocket/token/options
code removed.
+318
View File
@@ -0,0 +1,318 @@
// agent-browser connect — MV3 service worker.
//
// Bridges the user's real Chrome tabs to the local agent-browser daemon over a
// Chrome **native messaging** channel (no localhost port, no token: Chrome
// authenticates this extension to the host by id). It attaches chrome.debugger
// to eligible tabs and relays CDP both ways via a tiny envelope:
// host → ext : {id, method:"forwardCDPCommand", params:{method,params,sessionId}}
// ext → host : {id, result|error} (command reply)
// ext → host : {method:"forwardCDPEvent", params:{sessionId,method,params}}
//
// Target/discovery semantics (getTargets/attachToTarget) are emulated on the
// daemon side; here we just attach tabs and announce them as
// Target.attachedToTarget so the daemon's CDP client sees them appear.
//
// Adapted from openclaw-browser-relay (MIT, chengyixu) — the chrome.debugger
// attach + Target handling; the transport is rewritten from WebSocket+token to
// native messaging.
const HOST_NAME = 'com.agent_browser.connect'
const SKIP_URL = /^(chrome|chrome-extension|devtools|chrome-untrusted|edge|about):/i
/** @type {chrome.runtime.Port|null} */
let port = null
let nextSession = 1
/** tabId -> { sessionId, targetId } */
const tabs = new Map()
/** sessionId -> tabId (main session per tab) */
const sessionToTab = new Map()
/** child (OOPIF/worker) sessionId -> tabId */
const childSessionToTab = new Map()
function postToHost(msg) {
try {
if (port) port.postMessage(msg)
} catch (e) {
// port died; onDisconnect will reconnect.
}
}
function setBadge(tabId, kind) {
const map = { on: '', connecting: '…', error: '!' }
const colors = { on: '#16a34a', connecting: '#d97706', error: '#b91c1c' }
try {
chrome.action.setBadgeText({ tabId, text: map[kind] ?? '' })
if (colors[kind]) chrome.action.setBadgeBackgroundColor({ tabId, color: colors[kind] })
} catch {}
}
// ---- native messaging transport ------------------------------------------
function connectHost() {
if (port) return
try {
port = chrome.runtime.connectNative(HOST_NAME)
} catch (e) {
port = null
return
}
port.onMessage.addListener((msg) => void whenReady(() => onHostMessage(msg)))
port.onDisconnect.addListener(() => {
port = null
// Sessions are stale once the host is gone; the daemon re-discovers on
// reconnect. Keep chrome.debugger attached so reconnect is cheap.
for (const tabId of tabs.keys()) setBadge(tabId, 'connecting')
})
// Tell the daemon about everything we already have attached, then attach
// anything new.
reannounceAttachedTabs()
void attachAllTabs()
}
async function onHostMessage(msg) {
if (!msg || typeof msg !== 'object') return
// Optional keepalive.
if (msg.method === 'ping') {
postToHost({ method: 'pong' })
return
}
// Daemon (re)connected — (re)attach and announce every tab so it discovers
// the user's existing tabs rather than racing an empty target list.
if (msg.method === 'attachAll') {
reannounceAttachedTabs()
await attachAllTabs()
return
}
if (typeof msg.id !== 'undefined' && msg.method === 'forwardCDPCommand') {
try {
const result = await handleForwardCdpCommand(msg)
postToHost({ id: msg.id, result })
} catch (err) {
postToHost({ id: msg.id, error: err instanceof Error ? err.message : String(err) })
}
}
}
// ---- CDP command dispatch -------------------------------------------------
function tabForSession(sessionId) {
return sessionToTab.get(sessionId) ?? childSessionToTab.get(sessionId) ?? null
}
function tabForTarget(targetId) {
for (const [tabId, t] of tabs.entries()) if (t.targetId === targetId) return tabId
return null
}
function anyConnectedTab() {
const it = tabs.keys().next()
return it.done ? null : it.value
}
async function handleForwardCdpCommand(msg) {
const method = String(msg?.params?.method || '')
const params = msg?.params?.params || undefined
const sessionId = typeof msg?.params?.sessionId === 'string' ? msg.params.sessionId : undefined
// Browser-level Target methods that map onto chrome.tabs.
if (method === 'Target.createTarget') {
const url = typeof params?.url === 'string' && params.url ? params.url : 'about:blank'
const tab = await chrome.tabs.create({ url, active: false })
if (!tab.id) throw new Error('createTarget: no tab id')
await new Promise((r) => setTimeout(r, 100))
const t = await attachTab(tab.id)
return { targetId: t.targetId }
}
if (method === 'Target.closeTarget') {
const tid = typeof params?.targetId === 'string' ? params.targetId : ''
const tabId = tid ? tabForTarget(tid) : null
if (!tabId) return { success: false }
try {
await chrome.tabs.remove(tabId)
} catch {
return { success: false }
}
return { success: true }
}
if (method === 'Target.activateTarget') {
const tid = typeof params?.targetId === 'string' ? params.targetId : ''
const tabId = tid ? tabForTarget(tid) : null
if (tabId) {
const tab = await chrome.tabs.get(tabId).catch(() => null)
if (tab?.windowId) await chrome.windows.update(tab.windowId, { focused: true }).catch(() => {})
await chrome.tabs.update(tabId, { active: true }).catch(() => {})
}
return {}
}
// Everything else → chrome.debugger on the resolved tab.
const tabId =
(sessionId ? tabForSession(sessionId) : null) ??
(typeof params?.targetId === 'string' ? tabForTarget(params.targetId) : null) ??
anyConnectedTab()
if (!tabId) throw new Error(`no attached tab for ${method}`)
const dbg = { tabId }
// Re-enabling Runtime can leave a stale state; bounce it (matches upstream).
if (method === 'Runtime.enable') {
try {
await chrome.debugger.sendCommand(dbg, 'Runtime.disable')
await new Promise((r) => setTimeout(r, 30))
} catch {}
return await chrome.debugger.sendCommand(dbg, 'Runtime.enable', params)
}
return await chrome.debugger.sendCommand(dbg, method, params)
}
// ---- attach / detach ------------------------------------------------------
async function attachTab(tabId) {
const existing = tabs.get(tabId)
if (existing) return existing
const dbg = { tabId }
try {
await chrome.debugger.attach(dbg, '1.3')
} catch (e) {
// After a service-worker restart, chrome.debugger may still be bound to
// this tab from the previous instance — "Another debugger is already
// attached". The tab is still controllable via {tabId}, so don't skip it
// (skipping is why existing tabs went un-announced and the daemon opened a
// blank tab instead). Re-announce it. Any other error (restricted page) is
// surfaced and the caller skips this tab.
const msg = String((e && e.message) || e)
if (!/already attached|already being debugged/i.test(msg)) throw e
}
await chrome.debugger.sendCommand(dbg, 'Page.enable').catch(() => {})
const info = /** @type {any} */ (await chrome.debugger.sendCommand(dbg, 'Target.getTargetInfo'))
const targetInfo = info?.targetInfo
const targetId = String(targetInfo?.targetId || '')
if (!targetId) throw new Error('attachTab: no targetId')
const sessionId = `cb-tab-${nextSession++}`
const entry = { sessionId, targetId }
tabs.set(tabId, entry)
sessionToTab.set(sessionId, tabId)
setBadge(tabId, port ? 'on' : 'connecting')
postToHost({
method: 'forwardCDPEvent',
params: {
sessionId,
method: 'Target.attachedToTarget',
params: { sessionId, targetInfo: { ...targetInfo, attached: true } },
},
})
return entry
}
function detachTab(tabId, notify) {
const entry = tabs.get(tabId)
if (!entry) return
tabs.delete(tabId)
sessionToTab.delete(entry.sessionId)
for (const [sid, tid] of childSessionToTab.entries()) if (tid === tabId) childSessionToTab.delete(sid)
if (notify) {
postToHost({
method: 'forwardCDPEvent',
params: { sessionId: entry.sessionId, method: 'Target.detachedFromTarget', params: { sessionId: entry.sessionId } },
})
}
}
function eligible(tab) {
return !!tab && !!tab.id && typeof tab.url === 'string' && !SKIP_URL.test(tab.url)
}
async function attachAllTabs() {
let all = []
try {
all = await chrome.tabs.query({})
} catch {
return
}
for (const tab of all) {
if (eligible(tab) && !tabs.has(tab.id)) {
try {
await attachTab(tab.id)
} catch {
// Tab may be a restricted page or already attached elsewhere.
}
}
}
}
function reannounceAttachedTabs() {
for (const [, entry] of tabs.entries()) {
postToHost({
method: 'forwardCDPEvent',
params: {
sessionId: entry.sessionId,
method: 'Target.attachedToTarget',
params: { sessionId: entry.sessionId, targetInfo: { targetId: entry.targetId, type: 'page', attached: true } },
},
})
}
}
// ---- chrome.debugger events ----------------------------------------------
chrome.debugger.onEvent.addListener((source, method, params) =>
void whenReady(() => {
const tabId = source.tabId
if (!tabId) return
const entry = tabs.get(tabId)
if (!entry) return
if (method === 'Target.attachedToTarget' && params?.sessionId) {
childSessionToTab.set(String(params.sessionId), tabId)
}
if (method === 'Target.detachedFromTarget' && params?.sessionId) {
childSessionToTab.delete(String(params.sessionId))
}
postToHost({
method: 'forwardCDPEvent',
params: { sessionId: source.sessionId || entry.sessionId, method, params },
})
}),
)
chrome.debugger.onDetach.addListener((source) =>
void whenReady(() => {
if (source.tabId) detachTab(source.tabId, true)
}),
)
// ---- tab lifecycle --------------------------------------------------------
chrome.tabs.onUpdated.addListener((tabId, changeInfo, tab) =>
void whenReady(async () => {
if (changeInfo.status === 'complete' && eligible(tab) && !tabs.has(tabId) && port) {
try {
await attachTab(tabId)
} catch {}
}
}),
)
chrome.tabs.onRemoved.addListener((tabId) => void whenReady(() => detachTab(tabId, true)))
// ---- bootstrap + keepalive ------------------------------------------------
chrome.runtime.onInstalled.addListener(() => void whenReady(connectHost))
chrome.runtime.onStartup.addListener(() => void whenReady(connectHost))
chrome.action.onClicked.addListener(() => void whenReady(connectHost))
// MV3 service workers get suspended; an alarm wakes us to keep the host link
// and badges fresh.
chrome.alarms.create('keepalive', { periodInMinutes: 0.4 })
chrome.alarms.onAlarm.addListener((a) => {
if (a.name !== 'keepalive') return
void whenReady(() => {
if (!port) connectHost()
else void attachAllTabs()
})
})
// Gate placeholder so future async state-rehydration can hook in.
async function whenReady(fn) {
return fn()
}
// Kick a connection attempt as soon as the worker starts.
connectHost()
+16
View File
@@ -0,0 +1,16 @@
{
"manifest_version": 3,
"name": "agent-browser connect",
"version": "0.2.0",
"description": "Let agent-browser drive your logged-in Chrome — install once, no token, no per-use confirmation.",
"key": "MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAvPGcSmMx7dSfq9gRBDbQuAgx/+TEavrDxP/4jLa2+Ycexf/FEmq1MN8gAHoTRjSyp66YKD+1qI1CF6bk0rH5ZtxpRO7DYRTUPcsA1IpHbgEn5mppx3YxNGZilfkEZyrxdhBqUIzq3J74+/kpZzEVsO+DQTbAZSsFfdUkoCb5mbJid2VQYeqeBnYGAhbpGvN1P99jdT9EA1nKINb3ji6tLobCpyQ1fjf2uWm4mUirWkkF/nbUFVFEAh33Q/IYZmtUHgDYea5LsM9xH4KAG2kxMvFGj6vHR39sZd5/+gnvwScTcItUWQ9lFIyWYiwrSB25Lu0FshfllevXUFrG5vrvRwIDAQAB",
"icons": {
"16": "icons/icon16.png",
"32": "icons/icon32.png",
"48": "icons/icon48.png",
"128": "icons/icon128.png"
},
"permissions": ["debugger", "tabs", "nativeMessaging", "storage", "alarms", "webNavigation"],
"background": { "service_worker": "background.js", "type": "module" },
"action": { "default_title": "agent-browser connect" }
}
+7 -3
View File
@@ -95,10 +95,14 @@ fi
mkdir -p "$bindir"
mv "$tmp/${BIN_NAME}" "$bindir/${BIN_NAME}"
# convenience aliases: `abs` (short) -> agent-browser
ln -sf "$bindir/${BIN_NAME}" "$bindir/abs" 2>/dev/null || true
# Aliases pointing at the same binary: `abs` (short) and `agent-browser-stealth`
# (the fork's package name). All three names work, and an upgrade refreshes
# whichever name you actually run.
for alias_name in abs agent-browser-stealth; do
ln -sf "$bindir/${BIN_NAME}" "$bindir/${alias_name}" 2>/dev/null || true
done
info "installed ${BIN_NAME} -> ${bindir}/${BIN_NAME}"
info "installed -> ${bindir}/ (agent-browser, agent-browser-stealth, abs)"
"$bindir/${BIN_NAME}" --version 2>/dev/null || true
case ":$PATH:" in
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "agent-browser-stealth",
"version": "0.27.0-fork.14",
"version": "0.27.0-fork.18",
"description": "Browser automation CLI for AI agents — stealth fork with anti-detection",
"type": "module",
"packageManager": "pnpm@11.1.3",
+100 -3
View File
@@ -29,6 +29,57 @@ Refs (`@e1`, `@e2`, ...) are assigned fresh on every snapshot. They become
submits, dynamic re-renders, dialog opens. Always re-snapshot before your
next ref interaction.
## Before you automate: pick the cheapest tool
Driving a browser is the heavy option. agent-browser earns its keep when you
need a **real, logged-in browser** — not for reading text off a public page.
| You need | Use |
|---|---|
| Discover what exists / find sources | `WebSearch` |
| Specific facts from a static or public page | `WebFetch` or `curl` (no browser) |
| Login state, interaction, JS-rendered or anti-bot pages | **agent-browser** (this skill) |
| A page the user saved before / an internal system | `agent-browser find-url <keywords>` (their bookmarks), then open it |
| The user's **own already-open, logged-in** Chrome window | the **extension connect** flow (below) |
Don't hand-build deep URLs with query params — links discovered by *interacting*
with the site carry the right hidden context and dodge anti-bot checks; a
hand-constructed URL often doesn't.
### Driving the user's real, already-open Chrome (extension)
When the task needs the user's *live* logged-in window (their real session, the
window they're looking at — not a fresh browser), use the extension connect flow:
`agent-browser extension install` once, load `extensions/ab-connect` in
`chrome://extensions` once (a GUI step you can perform with a **computer-use /
GUI-automation tool** like the `cua-driver` skill — see
`references/commands.md` → "Drive your real, logged-in Chrome"), then
`agent-browser extension connect`. After that it's zero-confirmation, zero-token
CLI. Use `--launch` instead when a fresh, isolated browser is fine.
## Two ways to drive a page — and when to drop to `eval`
You have a **real Chrome with the user's DOM**. Two layers, mix them freely:
1. **Structured** (`snapshot` + `@ref`, `find`, typed actions) — convenient and
readable; best for straightforward forms and navigation. But the a11y view is
*lossy and fragile*: refs go stale on any change, hidden inputs never show up,
overlays can block coordinate clicks.
2. **eval-first** (`agent-browser eval "<js>"`) — your eyes and hands on the real
DOM: read hidden inputs, reach into Shadow DOM / iframes, inspect
`form.elements` and `.validity`, extract the exact shape you want, or call
`el.click()` directly. **The moment the structured path fights you, drop to
`eval` instead of retrying it** — it's the fast way to find *why* something
failed (e.g. a hidden `point_choice=none` the UI never exposes).
```bash
# "what's actually in this form / why won't it submit?"
agent-browser eval "[...document.forms[0].elements].map(e=>[e.name,e.type,e.value,e.checked])"
agent-browser eval "document.querySelector('[name=point_choice]')?.value"
agent-browser eval "[...document.forms[0].elements].filter(e=>!e.validity.valid).map(e=>e.name+': '+e.validationMessage)"
agent-browser eval "document.querySelector('#stubborn').click()" # direct DOM click, bypasses overlays
```
## Quickstart
```bash
@@ -137,9 +188,17 @@ agent-browser fill "input[name=email]" "user@test.com"
agent-browser click "button.primary"
```
Rule of thumb: snapshot + `@eN` refs are fastest and most reliable for
AI agents. `find role/text/label` is next best and doesn't require a prior
snapshot. Raw CSS is a fallback when the others fail.
Escalation ladder: snapshot + `@eN` refs are quickest for straightforward
pages → `find role/text/label` when you'd rather skip the snapshot → raw CSS
**`eval` the moment any of those fight you** (stale refs, hidden state,
occluded clicks). Don't retry a flaky structured locator three times; drop to
`eval` and act on the DOM directly.
`click` auto-scrolls into view and, if the coordinate click is occluded, falls
back to a DOM `.click()`. If a click *reports success but nothing happened*
classic for an autocomplete/menu `<li>` that closes on the input's blur — retry
that one with `AGENT_BROWSER_CLICK_MODE=dom agent-browser click ...`, or just
`agent-browser eval "<select the item via JS>"`.
## Waiting (read this)
@@ -209,6 +268,44 @@ AGENT_BROWSER_SESSION_NAME=my-app agent-browser open https://app.example.com
# State is auto-saved and restored on subsequent runs with the same name.
```
### Remember a site's quirks (site notes)
A site behaves the same every time you visit it. When you work out something
durable — a working selector, a URL pattern, a hidden field a form needs, an
anti-bot trap, what requires login — **write it down so the next run doesn't
re-discover it.** Keep one markdown file per domain (these are your own notes,
not shipped with the skill):
```
~/.agent-browser/site-patterns/<domain>.md
```
**Before** working on a domain, read its file if it exists (use your normal file
tools — this is plain markdown you own). Treat it as *hints, not guarantees*
sites change; verify before relying. **After** a successful session that taught
you something durable, create or update it. Suggested shape:
```markdown
---
domain: app.example.com
updated: 2026-06-05
---
## Platform traits
SPA; form renders ~1s after load (wait --text). Cloudflare on /login.
## Working patterns
- Address pick: the `<li>` closes on blur — select with CLICK_MODE=dom.
- Submit needs hidden `point_choice` set (eval), the UI never exposes it.
- Stable selector for "Continue": button[data-testid=submit]
## Known traps (date them)
- 2026-06-05: @ref to the basket button goes stale after the mini-cart opens;
re-snapshot or use `find role button --name "Checkout"`.
```
This is how repeat visits get fast and reliable instead of re-solving the same
page every time.
### Extract data
```bash
+114 -2
View File
@@ -318,15 +318,58 @@ agent-browser --version # Show version (-V)
agent-browser <command> --help # Show detailed help for a command
```
## Drive your real, logged-in Chrome (extension — zero confirmation)
Chrome 136 blocked `--remote-debugging-port` on the default profile, so to drive
the user's *existing* logged-in window, agent-browser uses a Chrome **extension**
over native messaging — no port, no token, no per-use confirmation (the
codex/claude approach).
One-time setup:
```bash
agent-browser extension install # writes the native-messaging host manifest
```
Then load the extension **once** — this is a GUI step (Chrome's `chrome://extensions`
is privileged; the CLI can't load an unpacked extension):
> chrome://extensions → enable **Developer mode** (top-right) → **Load unpacked**
> select `<repo>/extensions/ab-connect`
**You can do this load step yourself with a computer-use / GUI-automation tool**
(e.g. the `cua-driver` skill) — drive `chrome://extensions`, toggle Developer
mode, click *Load unpacked*, pick the folder in the Open dialog. If the extension
is already loaded, clicking its **Reload** (↻) button after a code change is
enough. Notes from doing this live: tools that send *synthetic keystrokes* (e.g.
peekaboo) often don't reach Chrome — **`cua-driver` works** because it reads
Chrome's accessibility tree and clicks real elements. The native "Open" file
dialog is the fiddly part; if keystroke entry there fails, ask the user to pick
the folder (one click). After it loads, Chrome assigns the extension a fixed id
(pinned in its manifest) and auto-connects the host.
Then, any time (pure CLI, zero confirmation):
```bash
agent-browser extension connect # auto-attaches to the live, logged-in tabs
agent-browser tab # list the real tabs it now controls
agent-browser tab t3 # switch the session to one of them
agent-browser snapshot -i / eval / click ... # drive it like any session
agent-browser extension status # is the host installed?
agent-browser extension uninstall # remove the host manifest
```
Security: the extension↔host link is authenticated by Chrome (extension id); the
host↔agent-browser CDP link uses an unguessable URL in a 0600 file. Use this when
you need the user's real cookies/login on their actual machine. (`--extension
<path>` is unrelated — that loads an extension into a *launched* browser.)
## Debugging
```bash
agent-browser --headed open example.com # Show browser window
agent-browser --cdp 9222 snapshot # Connect via CDP port
agent-browser connect 9222 # Alternative: connect command
agent-browser console # View console messages
agent-browser console # View console messages (needs AGENT_BROWSER_CAPTURE_CONSOLE=1)
agent-browser console --clear # Clear console
agent-browser errors # View page errors
agent-browser errors # View page errors (needs AGENT_BROWSER_CAPTURE_CONSOLE=1)
agent-browser errors --clear # Clear errors
agent-browser highlight @e1 # Highlight element
agent-browser inspect # Open Chrome DevTools for this session
@@ -336,6 +379,41 @@ agent-browser profiler start # Start Chrome DevTools profiling
agent-browser profiler stop trace.json # Stop and save profile
```
### Finding a page the user saved (`find-url`)
Search the user's local Chrome/Edge **bookmarks** by keyword — for internal
systems or previously-saved pages that public search can't reach. Local read, no
browser/daemon needed.
```bash
agent-browser find-url jira board # all keywords must match (name or url)
agent-browser find-url --limit 10 invoices
agent-browser find-url --browser edge --profile "Profile 1" wiki
agent-browser find-url grafana --json # {results:[{name,url,folder}], count}
```
Results are most-recently-added first. `javascript:`/`data:` bookmarklets are
skipped. (Visited-history search isn't included yet — bookmarks only.)
### Debugging forms / hidden state with `eval`
The a11y `snapshot` shows visible, interactive elements — it does **not** show
hidden inputs or a control's actual submitted value. When a form "looks filled"
but submit-validation rejects it, go straight to the DOM with `eval` instead of
guessing from the snapshot. This is usually the fastest way to find the real
problem (e.g. a hidden `point_choice=none` that the visible UI never exposes):
```bash
# Dump every field's name → value, including hidden inputs and unchecked radios
agent-browser eval "JSON.stringify([...document.forms[0].elements].map(e=>({name:e.name,type:e.type,value:e.value,checked:e.checked})).filter(e=>e.name))"
# Inspect one hidden field directly
agent-browser eval "document.querySelector('[name=point_choice]')?.value"
# Why won't it submit? Ask the browser's own validity API
agent-browser eval "[...document.forms[0].elements].filter(e=>!e.validity?.valid).map(e=>e.name+': '+e.validationMessage)"
```
## React / Web Vitals
Requires `--enable react-devtools` at launch for the `react ...` commands.
@@ -391,4 +469,38 @@ AGENT_BROWSER_HIDE_SCROLLBARS="false" # Keep native scrollbars visible in
AGENT_BROWSER_PROVIDER="browserbase" # Cloud browser provider
AGENT_BROWSER_STREAM_PORT="9223" # Override WebSocket streaming port (default: OS-assigned)
AGENT_BROWSER_HOME="/path/to/agent-browser" # Custom install location
AGENT_BROWSER_CLICK_MODE="dom" # Click strategy: "" (default: scroll-in + coordinate
# click, DOM-dispatch fallback), "coord" (strict
# coordinate only), "dom" (always element.click())
```
### Click reliability
`click` auto-scrolls the target into view first, then dispatches a coordinate
click. If that fails (a floating layer fails the occlusion guard, or the point
won't resolve) it falls back to a DOM-dispatched `.click()` on the intended
element. If a click *reports success but the page didn't react* — common for
autocomplete/menu `<li>` items that close on the input's blur — retry that one
with `AGENT_BROWSER_CLICK_MODE=dom` (a DOM dispatch doesn't move focus the way a
real pointer press does, so the item still selects). `=coord` disables the
fallback when you specifically want a hard failure on occlusion.
### Stealth / anti-detection knobs (fork)
```bash
AGENT_BROWSER_CAPTURE_CONSOLE="1" # Enable `console`/`errors` capture. OFF by default:
# a live CDP Runtime domain is a detectable bot signal,
# so console/errors return empty (with a hint) until set.
AGENT_BROWSER_TIMEZONE="Asia/Tokyo" # --launch only. Native timezone override (IANA id, or
# "auto" to derive from locale). Aligns Intl+Date to a proxy.
AGENT_BROWSER_BLOCK_WEBRTC="1" # --launch only. Hide local IP via WebRTC. Auto-forces WebRTC
# through the proxy when one is set; "0" opts out.
AGENT_BROWSER_HIDE_CANVAS="1" # --launch only. Session-stable canvas/audio fingerprint noise.
AGENT_BROWSER_ADAPTIVE_REF="0" # Disable adaptive @ref relocation (on by default; relocates a
# moved element by fingerprint when role/name re-query fails).
```
> **Heads-up for `console` / `errors`:** capture is **off by default** in this stealth
> fork. Both commands return `{"messages":[]}` / `{"errors":[]}` plus a `hint` until you
> launch the session with `AGENT_BROWSER_CAPTURE_CONSOLE=1`. This keeps the CDP `Runtime`
> domain disabled (a known bot signal) for the common automation path.
+1 -1
View File
@@ -96,7 +96,7 @@ Read [references/issue-taxonomy.md](references/issue-taxonomy.md) for the full l
- Within each section, test interactive elements: click buttons, fill forms, open dropdowns/modals.
- Check edge cases: empty states, error handling, boundary inputs.
- Try realistic end-to-end workflows (create, edit, delete flows).
- Check the browser console for errors periodically.
- Check the browser console for errors periodically. **Console/error capture is off by default in this stealth fork** — start the dogfood session with `AGENT_BROWSER_CAPTURE_CONSOLE=1` (e.g. `AGENT_BROWSER_CAPTURE_CONSOLE=1 agent-browser --session {SESSION} open <url>`) or `console`/`errors` will return empty.
**At each page:**
+3
View File
@@ -230,6 +230,9 @@ agent-browser snapshot -i | grep -c "treeitem"
### Check console for errors
Console/error capture is off by default in this stealth fork — launch the session with
`AGENT_BROWSER_CAPTURE_CONSOLE=1` first, or these return empty.
```bash
agent-browser console
agent-browser errors