Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
d81bc01645 | ||
|
|
c667e0e704 | ||
|
|
08cb8dbeb9 | ||
|
|
fd2cdcde77 | ||
|
|
8e001e3d88 | ||
|
|
eb2bc343a0 | ||
|
|
d86c9c4be2 | ||
|
|
32c25a6627 | ||
|
|
a8ce3dd3f8 | ||
|
|
997373fd57 | ||
|
|
5a858af93f |
@@ -205,6 +205,34 @@ chrome-use --launch --profile auto open https://x.com/home
|
||||
|
||||
In CI environments, standalone mode is used automatically.
|
||||
|
||||
## Site adapters — turn a website into a structured-data CLI
|
||||
|
||||
Most "read GitHub issues" / "search Reddit" / "get my Bilibili feed" tasks don't
|
||||
need clicking and screenshotting at all — the site already has a JSON API behind
|
||||
its own login. A **site adapter** is a tiny JS function that calls that API *from
|
||||
inside your logged-in tab* (your cookies, same-origin `fetch`, the site's own
|
||||
modules) and returns clean JSON. The site can't tell it apart from you, because it
|
||||
*is* you.
|
||||
|
||||
chrome-use ships none of these adapters — `site update` fetches the community
|
||||
[**bb-sites**](https://github.com/epiral/bb-sites) pack at runtime (like a package
|
||||
manager pulling a dependency), then runs them over chrome-use's stealth transport:
|
||||
|
||||
```bash
|
||||
chrome-use site update # fetch the adapter pack (~145 commands)
|
||||
chrome-use site list # github/issues, reddit/search, bilibili/feed, …
|
||||
chrome-use site info github/issues # see an adapter's args + domain
|
||||
|
||||
# Run one — navigates to the site (reusing the tab if you're already there) and returns JSON
|
||||
chrome-use site github/issues epiral/bb-browser --json
|
||||
chrome-use site reddit/search "rust async" --json
|
||||
chrome-use site bilibili/feed --json # works because it's your logged-in session
|
||||
```
|
||||
|
||||
Positional args fill the adapter's declared args in order; `--key value` overrides
|
||||
by name. Adapters are authored by the bb-sites community and remain their authors'
|
||||
property — chrome-use just runs them.
|
||||
|
||||
## Automated testing (`chrome-use test`)
|
||||
|
||||
Turn the repetitive "open it, click around, check it's right" work into a
|
||||
@@ -326,3 +354,6 @@ We deliberately **don't ship our own bot detector** — the strongest, most hone
|
||||
## License
|
||||
|
||||
Apache-2.0
|
||||
---
|
||||
|
||||
> Built by **leeguooooo** — field notes on AI agents, reverse engineering & Cloudflare Workers at **[blog.misonote.com](https://blog.misonote.com)** · follow on **[X @leeguooooo](https://x.com/leeguooooo)**
|
||||
|
||||
+124
-1
@@ -63,6 +63,25 @@ chrome-use 让**任意** agent(Claude Code、Cursor、Codex、你自己的脚
|
||||
|
||||
每个 `--session` 拿到**自己的彩色标签组**,多个 agent 共用同一个真实浏览器、互不干扰,也不动你自己的标签页。
|
||||
|
||||
## 为什么用扩展(而非裸调试端口)
|
||||
|
||||
其他本地工具走裸 `--remote-debugging-port`(CDP)驱动 Chrome。从 **Chrome 136** 起,每次这样连接都会弹出一个阻塞式的 **"Allow remote debugging?"** 同意框 —— 而且端口得提前开好。我们的扩展改用原生消息:**装一次,之后零确认。**
|
||||
|
||||
| | **chrome-use**(本扩展) | web-access(裸 CDP 端口) | Claude in Chrome(chrome.debugger) |
|
||||
|---|---|---|---|
|
||||
| 连接方式 | 原生消息 —— 无端口、无 token | `--remote-debugging-port` | `chrome.debugger` |
|
||||
| **"Allow remote debugging?" 弹框** | **从不** ✅ | **每次连都弹** 🔴 | 无 |
|
||||
| 复用你的真实登录 | 是 | 是 | 是 |
|
||||
| `Runtime.enable`(CDP)泄漏¹ | **默认关闭 → 干净** ✅ | 域已启用 | 不适用 |
|
||||
| CreepJS 隐身分² | **0% stealth · 0% headless** ✅ | 真实 Chrome | 真实 Chrome |
|
||||
| 每会话标签组 / 并发 agent | **支持** ✅ | 无 | 无 |
|
||||
| 为 chrome-use CLI 打造 | 是 | 独立代理 | 单 app 助手 |
|
||||
|
||||
> ¹ 对 [rebrowser-bot-detector](https://bot-detector.rebrowser.net/) 实测:我们的中继报 `runtimeEnableLeak: 🟢 No leak`、`navigatorWebdriver: 🟢`。
|
||||
> ² 对 [CreepJS](https://abrahamjuliot.github.io/creepjs/) 在「连接真实 Chrome」路径上实测 —— 见 [反检测](#反检测)。
|
||||
>
|
||||
> 同意框不是假想:裸端口工具**每次** attach 都会弹(Chrome 136+ 安全策略)。扩展路径从不弹。
|
||||
|
||||
## 安装
|
||||
|
||||
```bash
|
||||
@@ -71,6 +90,15 @@ curl -fsSL https://raw.githubusercontent.com/leeguooooo/chrome-use/main/install.
|
||||
|
||||
从最新的 [GitHub Release](https://github.com/leeguooooo/chrome-use/releases) 下载对应平台的预编译二进制,安装 `chrome-use`(以及 `abs` 别名)。无需 npm,无需 token。
|
||||
|
||||
<details>
|
||||
<summary>其他安装方式</summary>
|
||||
|
||||
- **锁定版本:** `AGENT_BROWSER_VERSION=v0.27.0-fork.12 curl -fsSL https://raw.githubusercontent.com/leeguooooo/chrome-use/main/install.sh | sh`
|
||||
- **自定义路径:** `AGENT_BROWSER_BIN_DIR=$HOME/bin curl -fsSL … | sh`
|
||||
- **Windows:** 从 [Releases 页](https://github.com/leeguooooo/chrome-use/releases) 下载 `chrome-use-win32-x64.tar.gz`,把 `chrome-use.exe` 放进 PATH。
|
||||
- **npm(旧渠道):** `npm install -g chrome-use` —— 仍在发布,但 GitHub Releases 现在是主渠道。
|
||||
</details>
|
||||
|
||||
### 安装 AI agent skills
|
||||
|
||||
```bash
|
||||
@@ -79,6 +107,10 @@ npx skills add leeguooooo/chrome-use
|
||||
|
||||
把 `skills/chrome-use` 拉进当前项目,让你的 AI agent 拿到正确的用法和预授权的 bash 权限。
|
||||
|
||||
## 命令名
|
||||
|
||||
`chrome-use`、`chrome-use`、`abs` 是**同一个二进制** —— `abs` 只是短别名。没有单独的「隐身可执行文件」;隐身是**运行时行为**(见下方 [反检测](#反检测)),根据你是连接真实 Chrome 还是 `--launch` 全新实例自动启用。
|
||||
|
||||
## 连接你的 Chrome
|
||||
|
||||
**推荐 —— 浏览器扩展(一键,无弹窗)。** 从 Chrome 应用商店安装 [**chrome-use** 扩展](https://chromewebstore.google.com/detail/chrome-use/knfcmbamhjmaonkfnjhldjedeobeafmk),再注册一次本地桥:
|
||||
@@ -128,8 +160,69 @@ chrome-use --launch open https://example.com
|
||||
|
||||
# 保留登录:用你真实的 Chrome profile 启动
|
||||
chrome-use --launch --profile auto open https://x.com/home
|
||||
# 或显式指定:--profile Default / --profile "Profile 1"
|
||||
```
|
||||
|
||||
## 站点适配器 —— 把一个网站变成「结构化数据 CLI」
|
||||
|
||||
大多数「读 GitHub issue」「搜 Reddit」「拉我的 B 站动态」这类任务,根本不需要点击 +
|
||||
截图 —— 网站登录态背后本来就有 JSON 接口。**站点适配器**就是一小段 JS 函数,它在你
|
||||
**已登录的标签页内**调用那个接口(用你的 cookie、同源 `fetch`、网站自己的模块),返回
|
||||
干净的 JSON。网站分辨不出它和你的区别,因为它**就是你**。
|
||||
|
||||
chrome-use 本身不附带任何适配器 —— `site update` 会在运行时拉取社区的
|
||||
[**bb-sites**](https://github.com/epiral/bb-sites) 适配器包(就像包管理器拉依赖),
|
||||
然后在 chrome-use 的隐身通道上运行它们:
|
||||
|
||||
```bash
|
||||
chrome-use site update # 拉取适配器包(约 145 条命令)
|
||||
chrome-use site list # github/issues、reddit/search、bilibili/feed…
|
||||
chrome-use site info github/issues # 查看某个适配器的参数 + 域名
|
||||
|
||||
# 运行一个 —— 会导航到对应站点(已在该站点则复用当前标签页)并返回 JSON
|
||||
chrome-use site github/issues epiral/bb-browser --json
|
||||
chrome-use site reddit/search "rust async" --json
|
||||
chrome-use site bilibili/feed --json # 能用,因为走的是你的登录态
|
||||
```
|
||||
|
||||
位置参数按适配器声明的参数顺序填入;`--key value` 按名覆盖。适配器由 bb-sites 社区编写、
|
||||
版权归各自作者所有 —— chrome-use 只负责运行它们。
|
||||
|
||||
## 自动化测试(`chrome-use test`)
|
||||
|
||||
把反复的「打开它、点一圈、看对不对」变成**可重跑的测试套件** —— 前端的单元测试。用 YAML 写用例;步骤复用 chrome-use 自己的命令,断言编译成一次检查:
|
||||
|
||||
```yaml
|
||||
# smoke.yaml
|
||||
suite: chatgpt smoke
|
||||
setup:
|
||||
- account: chatgpt/huayue # 注入一个 cookie-use 登录(可选)
|
||||
cases:
|
||||
- name: home loads logged in
|
||||
steps:
|
||||
- open: https://chatgpt.com/
|
||||
- wait: { load: networkidle }
|
||||
assert:
|
||||
- url: { contains: chatgpt.com }
|
||||
- visible: "#prompt-textarea"
|
||||
```
|
||||
|
||||
```bash
|
||||
chrome-use test smoke.yaml # 启动隔离浏览器,跑用例
|
||||
chrome-use test smoke.yaml --session default # …或对你已连接的 Chrome 跑
|
||||
```
|
||||
|
||||
```
|
||||
suite: chatgpt smoke (session cu-test)
|
||||
✓ home loads logged in 1.2s
|
||||
✗ composer takes text 0.8s
|
||||
assert text "#prompt-textarea" contains "hi" → got ""
|
||||
↳ cu-test-artifacts/composer-takes-text.png
|
||||
2 cases · 1 passed · 1 failed
|
||||
```
|
||||
|
||||
任一用例失败时退出码非零(可直接丢进 CI),失败用例会存截图。断言:`url` · `visible` · `hidden` · `text` · `count` · `eval`。步骤:`open` · `click` · `fill` · `type` · `press` · `wait` · `scroll` · `eval`。完整指南:`chrome-use skills get test`。发现回归?加个用例 —— 用得越多,套件越值钱。
|
||||
|
||||
## 反检测
|
||||
|
||||
连接你真实 Chrome 时,我们**零** JS 注入 —— 浏览器指纹完全是真的。指导原则是 **native CDP/Chrome 覆盖优先于 JS 谎言**:被重定义的 getter 本身可被检测,原生覆盖则不会。
|
||||
@@ -146,7 +239,9 @@ chrome-use --launch --profile auto open https://x.com/home
|
||||
| [rebrowser-bot-detector](https://bot-detector.rebrowser.net/) | `runtimeEnableLeak` 🟢 · `pwInitScripts` 🟢 |
|
||||
| [bot.sannysoft.com](https://bot.sannysoft.com) | 全绿 |
|
||||
|
||||
`--launch` 独立模式下会改用一整套隐身补丁,同样过上述检测。
|
||||
CreepJS 上的 `0% stealth` 是关键数字:因为连接路径**什么都不打补丁**,根本没有可供说谎检测器抓的 override。(读 `navigator.languages` 顺序或 IP 地理位置的面板可能给个软性的「navigator」/「location」标记 —— 那反映的是*你真实 Chrome* 的语言列表和网络,不是自动化破绽。)
|
||||
|
||||
`--launch` 独立模式(全新浏览器)会改用一整套隐身补丁,也能过上述检测 —— 唯一例外:CreepJS 报 **~20% stealth**,因为 srcdoc-iframe 的 `contentWindow` 补丁触发了它的 `hasIframeProxy` 探测(用来藏自动化的 proxy 本身成了破绽)。其余全干净(`0% headless`、sannysoft/browserscan 全绿、Cloudflare 通过)。设 **`AGENT_BROWSER_DISABLE_IFRAME_PROXY=1`** 去掉那个补丁即可拿到干净的 **0% stealth**(代价是放弃小众的 srcdoc-iframe 遮蔽)。**扩展连接路径**(你的真实 Chrome)零 JS 注入、不受影响 —— 它才是货真价实的 0% 路径。
|
||||
|
||||
### 类人输入(行为隐身)
|
||||
|
||||
@@ -167,6 +262,30 @@ chrome-use --launch --profile auto open https://x.com/home
|
||||
|
||||
操作你的真实 Chrome 不该打断你的工作。agent **全程在后台操作**:新标签后台打开(在自己的彩色会话标签组里),**从不强制把标签拽到前台**,并用 `Emulation.setFocusEmulationEnabled` 让每个 agent 标签照常渲染、`document.hasFocus()` / `visibilityState` 仍报 `visible`。于是截图正常、页面不被降频,"标签全程隐藏"也不会变成新的机器人信号。你在自己的标签里照常工作,agent 在旁边默默干活。(想置顶某个标签仍可显式调用命令。)
|
||||
|
||||
### 自己验证
|
||||
|
||||
别光听我们说 —— 把你连接的 Chrome 指向最硬的公开检测器,自己对比:
|
||||
|
||||
- **[CreepJS](https://abrahamjuliot.github.io/creepjs/)** —— 最全面的指纹 / 说谎检测器
|
||||
- **[bot.incolumitas.com](https://bot.incolumitas.com/)** —— 行为 + 指纹打分,方法公开
|
||||
- **[BrowserScan](https://www.browserscan.net/bot-detection)** —— Webdriver / User-Agent / CDP / Navigator
|
||||
- **[bot.sannysoft.com](https://bot.sannysoft.com)** —— 经典自动化特征清单
|
||||
- **[pixelscan.net](https://pixelscan.net/)** · **[iphey.com](https://iphey.com/)** —— 一致性与身份
|
||||
|
||||
我们故意**不自带 bot 检测器** —— 最强、最诚实的基准,就是拿市面上最好的检测器去测你的真实浏览器。
|
||||
|
||||
### 调参(环境变量)
|
||||
|
||||
| 变量 | 默认 | 作用 |
|
||||
|---|---|---|
|
||||
| `AGENT_BROWSER_CAPTURE_CONSOLE` | 关 | 启用 `Runtime` 域,让 `console` / `errors` 捕获页面输出。关闭可保持最隐身的画像。 |
|
||||
| `AGENT_BROWSER_HUMANIZE` | 关 | 类人输入动作:`off`(瞬时)、`fast`(轻量缓动轨迹)、`human`(全套曲线轨迹 + 落点抖动 + 击键节奏 + 缓动滚动/拖拽)。也可用 `--humanize`。默认 `off`;自适应检测器会把 Akamai/PerimeterX/DataDome 守护的页面自动升到 `human`。 |
|
||||
| `AGENT_BROWSER_TIMEZONE` | 未设 | 仅 `--launch`。IANA id(如 `Asia/Tokyo`)原生设置时区(Intl + Date 跟随,无 JS 谎言)以匹配代理;`auto` 按 locale 推导。 |
|
||||
| `AGENT_BROWSER_BLOCK_WEBRTC` | auto | 仅 `--launch`。设了代理时自动强制 WebRTC 走代理(不泄漏真实 IP)。`1` 无代理时也隐藏本地 IP;`0` 退出。 |
|
||||
| `AGENT_BROWSER_HIDE_CANVAS` | 关 | 仅 `--launch`。加入会话稳定的 canvas/audio 指纹噪声。默认关(噪声本身就是一种「谎言」)。 |
|
||||
| `AGENT_BROWSER_ADAPTIVE_REF` | 开 | 当保存的 `@ref` 移动且 role/name 重查失败时,按指纹相似度重定位(需高分 + 明显领先,否则明确报错)。`0` 关闭。 |
|
||||
| `AGENT_BROWSER_CLICK_MODE` | _(auto)_ | 点击策略。默认先滚动入视、派发坐标点击,若被浮层遮挡则回退 DOM `.click()`。`dom` 始终用 `.click()`(适合 blur 即关的自动补全/菜单项);`coord` 严格只用坐标(遮挡时硬失败)。 |
|
||||
|
||||
## chrome-use 的独特之处
|
||||
|
||||
- **默认 auto-connect** —— `chrome-use open` 连你现有的 Chrome 而非启新的
|
||||
@@ -181,3 +300,7 @@ chrome-use --launch --profile auto open https://x.com/home
|
||||
## License
|
||||
|
||||
Apache-2.0
|
||||
|
||||
---
|
||||
|
||||
> 由 **leeguooooo** 打造 —— AI agent、逆向工程与 Cloudflare Workers 的实战笔记见 **[blog.misonote.com](https://blog.misonote.com)** · 关注 **[X @leeguooooo](https://x.com/leeguooooo)**
|
||||
|
||||
Generated
+1
-1
@@ -290,7 +290,7 @@ checksum = "613afe47fcd5fac7ccf1db93babcb082c5994d996f20b8b159f2ad1658eb5724"
|
||||
|
||||
[[package]]
|
||||
name = "chrome-use"
|
||||
version = "1.5.14"
|
||||
version = "1.5.19"
|
||||
dependencies = [
|
||||
"aes",
|
||||
"aes-gcm",
|
||||
|
||||
+1
-1
@@ -1,6 +1,6 @@
|
||||
[package]
|
||||
name = "chrome-use"
|
||||
version = "1.5.14"
|
||||
version = "1.5.19"
|
||||
edition = "2021"
|
||||
description = "Fast browser automation CLI for AI agents"
|
||||
license = "Apache-2.0"
|
||||
|
||||
@@ -78,6 +78,7 @@ const KNOWN_COMMANDS: &[&str] = &[
|
||||
"drag",
|
||||
"dialog",
|
||||
"upload",
|
||||
"site",
|
||||
];
|
||||
|
||||
/// Levenshtein distance, capped — small inputs only (command names).
|
||||
@@ -1193,6 +1194,47 @@ fn parse_command_inner(args: &[String], flags: &Flags) -> Result<Value, ParseErr
|
||||
Ok(json!({ "id": id, "action": "evaluate", "script": script }))
|
||||
}
|
||||
|
||||
"site" => {
|
||||
// `site <name>/<command> [positional...] [--key value]`. The
|
||||
// `update`/`list`/`info` subcommands are handled CLI-side (main.rs)
|
||||
// and never reach here — by this point `rest[0]` is a `name/cmd`
|
||||
// adapter spec. Load it, map the args onto the adapter's declared
|
||||
// `args`, and emit a `site` action: the daemon navigates to the
|
||||
// adapter's @meta.domain (reusing the tab if already there) and evals
|
||||
// the adapter function in the site's own logged-in page.
|
||||
let spec = rest.first().ok_or(ParseError::InvalidValue {
|
||||
message: "site requires <name>/<command> (run `chrome-use site list`)".to_string(),
|
||||
usage: "site <name>/<command> [args]",
|
||||
})?;
|
||||
let adapter =
|
||||
crate::site::load_adapter(spec).map_err(|e| ParseError::InvalidValue {
|
||||
message: e,
|
||||
usage: "site <name>/<command> [args]",
|
||||
})?;
|
||||
let domain = adapter
|
||||
.domain()
|
||||
.ok_or(ParseError::InvalidValue {
|
||||
message: format!("site: adapter `{spec}` @meta is missing a \"domain\""),
|
||||
usage: "site <name>/<command>",
|
||||
})?
|
||||
.to_string();
|
||||
// Split remaining args: `--key value` → named, everything else → positional.
|
||||
let mut positional: Vec<String> = Vec::new();
|
||||
let mut named: Vec<(String, String)> = Vec::new();
|
||||
let mut it = rest[1..].iter();
|
||||
while let Some(a) = it.next() {
|
||||
if let Some(key) = a.strip_prefix("--") {
|
||||
let val = it.next().map(|s| s.to_string()).unwrap_or_default();
|
||||
named.push((key.to_string(), val));
|
||||
} else {
|
||||
positional.push(a.to_string());
|
||||
}
|
||||
}
|
||||
let mapped = crate::site::map_args(&adapter, &positional, &named);
|
||||
let script = crate::site::build_eval(&adapter, &mapped);
|
||||
Ok(json!({ "id": id, "action": "site", "domain": domain, "script": script }))
|
||||
}
|
||||
|
||||
// === Stealth self-check ===
|
||||
"stealth" => {
|
||||
// `stealth [status]` — local stealth self-check: mode, live probes
|
||||
|
||||
@@ -10,6 +10,7 @@ mod flags;
|
||||
mod install;
|
||||
mod native;
|
||||
mod output;
|
||||
mod site;
|
||||
mod skills;
|
||||
mod test_runner;
|
||||
#[cfg(test)]
|
||||
@@ -834,6 +835,84 @@ fn main() {
|
||||
exit(test_runner::run_test(suite, &flags));
|
||||
}
|
||||
|
||||
// Handle `site`: site adapters — turn a website into a structured-data CLI by
|
||||
// running a per-command JS adapter inside your logged-in tab. `update`/`list`/
|
||||
// `info` are CLI-side (download/filesystem); `site <name>/<cmd> [args]` falls
|
||||
// through to the daemon dispatch below (navigate to the adapter's domain + eval).
|
||||
if clean.first().map(|s| s.as_str()) == Some("site") {
|
||||
match clean.get(1).map(|s| s.as_str()) {
|
||||
Some("update") => {
|
||||
let rt = tokio::runtime::Runtime::new().expect("Failed to create tokio runtime");
|
||||
match rt.block_on(site::update()) {
|
||||
Ok(n) if flags.json => {
|
||||
println!("{}", json!({ "success": true, "adapters": n }))
|
||||
}
|
||||
Ok(n) => println!(
|
||||
"{} synced {} site adapters → ~/.chrome-use/sites (run `chrome-use site list`)",
|
||||
color::success_indicator(),
|
||||
n
|
||||
),
|
||||
Err(e) => {
|
||||
eprintln!("{} {}", color::error_indicator(), e);
|
||||
exit(1);
|
||||
}
|
||||
}
|
||||
return;
|
||||
}
|
||||
Some("list") => {
|
||||
match site::list_adapters() {
|
||||
Ok(list) if flags.json => {
|
||||
println!("{}", json!({ "success": true, "adapters": list }))
|
||||
}
|
||||
Ok(list) if list.is_empty() => {
|
||||
println!("no site adapters installed — run `chrome-use site update`")
|
||||
}
|
||||
Ok(list) => {
|
||||
for a in &list {
|
||||
println!("{a}");
|
||||
}
|
||||
eprintln!(
|
||||
"{}",
|
||||
color::dim(&format!(
|
||||
"{} adapters · run: chrome-use site <name>/<cmd> [args]",
|
||||
list.len()
|
||||
))
|
||||
);
|
||||
}
|
||||
Err(e) => {
|
||||
eprintln!("{} {}", color::error_indicator(), e);
|
||||
exit(1);
|
||||
}
|
||||
}
|
||||
return;
|
||||
}
|
||||
Some("info") => {
|
||||
let spec = clean.get(2).cloned().unwrap_or_default();
|
||||
match site::load_adapter(&spec) {
|
||||
Ok(a) => println!(
|
||||
"{}",
|
||||
serde_json::to_string_pretty(&a.meta).unwrap_or_default()
|
||||
),
|
||||
Err(e) => {
|
||||
eprintln!("{} {}", color::error_indicator(), e);
|
||||
exit(1);
|
||||
}
|
||||
}
|
||||
return;
|
||||
}
|
||||
// `site <name>/<cmd> [args]` → fall through to the daemon dispatch.
|
||||
Some(spec) if spec.contains('/') => {}
|
||||
_ => {
|
||||
eprintln!(
|
||||
"{} usage: chrome-use site <name>/<cmd> [args] | site update | site list | \
|
||||
site info <name>/<cmd>",
|
||||
color::error_indicator()
|
||||
);
|
||||
exit(2);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Handle skills command (doesn't need daemon)
|
||||
if clean.first().map(|s| s.as_str()) == Some("skills") {
|
||||
skills::run_skills(&clean, flags.json);
|
||||
|
||||
@@ -1315,6 +1315,7 @@ pub async fn execute_command(cmd: &Value, state: &mut DaemonState) -> Value {
|
||||
"title" => handle_title(state).await,
|
||||
"content" => handle_content(state).await,
|
||||
"evaluate" => handle_evaluate(cmd, state).await,
|
||||
"site" => handle_site(cmd, state).await,
|
||||
"close" => handle_close(state).await,
|
||||
"stealth_status" => handle_stealth_status(state).await,
|
||||
"snapshot" => handle_snapshot(cmd, state).await,
|
||||
@@ -2698,6 +2699,47 @@ async fn handle_evaluate(cmd: &Value, state: &DaemonState) -> Result<Value, Stri
|
||||
Ok(json!({ "result": result, "origin": url }))
|
||||
}
|
||||
|
||||
/// Run a site adapter: navigate to its `@meta.domain` (only if we're not already
|
||||
/// there — the point is to run as you, in the page that's already open) and eval
|
||||
/// the adapter function in the site's own logged-in page. The CLI/commands.rs has
|
||||
/// already loaded the adapter and built the `script`; here we just place the page
|
||||
/// and evaluate. Never disrupts the user's foreground tab — navigation happens on
|
||||
/// the daemon's own tab (same as every other command on the relay).
|
||||
async fn handle_site(cmd: &Value, state: &mut DaemonState) -> Result<Value, String> {
|
||||
let domain = cmd
|
||||
.get("domain")
|
||||
.and_then(|v| v.as_str())
|
||||
.ok_or("site: missing 'domain'")?
|
||||
.to_string();
|
||||
let script = cmd
|
||||
.get("script")
|
||||
.and_then(|v| v.as_str())
|
||||
.ok_or("site: missing 'script'")?
|
||||
.to_string();
|
||||
|
||||
let current = match state.browser.as_ref() {
|
||||
Some(mgr) => mgr.get_url().await.unwrap_or_default(),
|
||||
None => String::new(),
|
||||
};
|
||||
let on_domain = url::Url::parse(¤t)
|
||||
.ok()
|
||||
.and_then(|u| u.host_str().map(|h| h.to_string()))
|
||||
.map(|h| h == domain || h.ends_with(&format!(".{domain}")))
|
||||
.unwrap_or(false);
|
||||
if !on_domain {
|
||||
let nav = json!({ "url": format!("https://{domain}/") });
|
||||
handle_navigate(&nav, state).await?;
|
||||
}
|
||||
|
||||
let eval_cmd = json!({ "script": script });
|
||||
let out = handle_evaluate(&eval_cmd, state).await?;
|
||||
Ok(json!({
|
||||
"result": out.get("result").cloned().unwrap_or(Value::Null),
|
||||
"origin": out.get("origin").cloned().unwrap_or(Value::Null),
|
||||
"domain": domain,
|
||||
}))
|
||||
}
|
||||
|
||||
/// Local stealth self-check: reports the active mode, live fingerprint probes,
|
||||
/// and the list of applied overrides — so an agent (or human) can confirm
|
||||
/// stealth is working without driving an external detector, and audit exactly
|
||||
|
||||
+153
-22
@@ -235,6 +235,43 @@ fn prunable_target_ids(
|
||||
.collect()
|
||||
}
|
||||
|
||||
/// Consecutive missing `getTargets` snapshots before an owned relay tab is
|
||||
/// pruned. >1 so a single churning/partial snapshot (other agents opening/closing
|
||||
/// tabs) or a brief cross-process-nav gap can't drop the tab the agent is driving.
|
||||
const RELAY_PRUNE_MISSES: u32 = 3;
|
||||
|
||||
/// Debounced prune for the relay: target ids to drop, mutating per-target miss
|
||||
/// counters. A tab in `live_ids` resets to 0; an absent (non-pinned) tab
|
||||
/// increments and is pruned only at `RELAY_PRUNE_MISSES`. Counters for
|
||||
/// no-longer-tracked targets are forgotten. Pure, so the multi-agent churn
|
||||
/// tolerance is unit-testable without a live browser.
|
||||
fn debounced_prune_ids(
|
||||
pages: &[PageInfo],
|
||||
live_ids: &HashSet<String>,
|
||||
pinned: Option<&str>,
|
||||
misses: &mut HashMap<String, u32>,
|
||||
) -> Vec<String> {
|
||||
let tracked: HashSet<&str> = pages.iter().map(|p| p.target_id.as_str()).collect();
|
||||
misses.retain(|tid, _| tracked.contains(tid.as_str()));
|
||||
let mut prune = Vec::new();
|
||||
for p in pages {
|
||||
let tid = p.target_id.as_str();
|
||||
if live_ids.contains(tid) {
|
||||
misses.remove(tid);
|
||||
continue;
|
||||
}
|
||||
if pinned == Some(tid) {
|
||||
continue;
|
||||
}
|
||||
let c = misses.entry(p.target_id.clone()).or_insert(0);
|
||||
*c += 1;
|
||||
if *c >= RELAY_PRUNE_MISSES {
|
||||
prune.push(p.target_id.clone());
|
||||
}
|
||||
}
|
||||
prune
|
||||
}
|
||||
|
||||
/// Whether the resolved active page is a tab the session created (its target_id
|
||||
/// is in `created_targets`). Pure core of [`BrowserManager::active_is_session_owned`]
|
||||
/// so the relay no-hijack rule is unit-testable without a live browser.
|
||||
@@ -464,6 +501,14 @@ pub struct BrowserManager {
|
||||
/// the session's commands onto the wrong page — the wrong-origin-fetch hazard
|
||||
/// in the dogfood reports. Falls back to the index if the pinned tab is gone.
|
||||
active_target_id: Option<String>,
|
||||
/// Per-target count of CONSECUTIVE `resync_targets` snapshots in which an
|
||||
/// owned tab was missing from `Target.getTargets`. Over the relay a single
|
||||
/// snapshot routinely omits live tabs (multi-agent churn, a cross-process nav
|
||||
/// briefly dropping the target), so we must not prune on one miss — that lost
|
||||
/// the tab the agent was driving. A tab is removed only after it's been absent
|
||||
/// for `RELAY_PRUNE_MISSES` consecutive snapshots; any snapshot that includes
|
||||
/// it resets the counter. Keyed by stable target_id.
|
||||
relay_target_misses: HashMap<String, u32>,
|
||||
next_tab_id: u32,
|
||||
/// Whether to enable the CDP `Runtime` domain (console / error / exception capture).
|
||||
/// OFF by default for stealth: a live `Runtime.enable` is a detectable CDP signal
|
||||
@@ -600,6 +645,7 @@ impl BrowserManager {
|
||||
visited_origins: HashSet::new(),
|
||||
created_targets: HashSet::new(),
|
||||
active_target_id: None,
|
||||
relay_target_misses: HashMap::new(),
|
||||
next_tab_id: 1,
|
||||
capture_console: console_capture_enabled(),
|
||||
};
|
||||
@@ -702,6 +748,7 @@ impl BrowserManager {
|
||||
visited_origins: HashSet::new(),
|
||||
created_targets: HashSet::new(),
|
||||
active_target_id: None,
|
||||
relay_target_misses: HashMap::new(),
|
||||
next_tab_id: 1,
|
||||
capture_console: console_capture_enabled(),
|
||||
};
|
||||
@@ -823,7 +870,19 @@ impl BrowserManager {
|
||||
self.active_page_index = 0;
|
||||
self.pin_active_target();
|
||||
self.enable_domains(&attach_result.session_id).await?;
|
||||
} else if self.agent_group().is_some() {
|
||||
// STRICT MULTI-AGENT ISOLATION (relay / the user's real Chrome).
|
||||
// `page_targets` here are the USER's and OTHER agents' tabs. A tab
|
||||
// group belongs to exactly ONE agent, so this session must NOT adopt
|
||||
// any of them — it tracks ONLY tabs it creates (its own colored group)
|
||||
// plus popups it opens. Adopting foreign tabs is precisely what let
|
||||
// another concurrent agent's tab churn drop the tab we were driving and
|
||||
// drift eval/click onto the wrong page (multi-agent failure). Open our
|
||||
// own dedicated background tab in the session's group and pin it; the
|
||||
// user's / other agents' tabs stay invisible to us.
|
||||
self.tab_new(None, None).await?;
|
||||
} else {
|
||||
// A browser WE launched: every tab is ours, so adopt them all.
|
||||
for target in &page_targets {
|
||||
let attach_result: AttachToTargetResult = self
|
||||
.client
|
||||
@@ -849,25 +908,11 @@ impl BrowserManager {
|
||||
target_type: target.target_type.clone(),
|
||||
});
|
||||
}
|
||||
|
||||
if self.agent_group().is_some() {
|
||||
// Relay: the adopted tabs above are the USER's, in their real
|
||||
// Chrome. NEVER make one of them the agent's working tab — that is
|
||||
// how commands drifted onto whatever page the user was viewing
|
||||
// between steps (eval/click/get landed on the user's foreground
|
||||
// tab; #35). Open our own dedicated background tab in the session's
|
||||
// group and pin THAT as active. The user's tabs stay adopted (so
|
||||
// `tab list` / explicit `tab switch` can reach them) but are never
|
||||
// auto-selected — the agent only ever drives a tab it owns.
|
||||
self.tab_new(None, None).await?;
|
||||
} else {
|
||||
// A browser we launched: every tab is ours, so the first is fine.
|
||||
self.active_page_index = 0;
|
||||
self.pin_active_target();
|
||||
let session_id = self.pages[0].session_id.clone();
|
||||
self.enable_domains(&session_id).await?;
|
||||
}
|
||||
}
|
||||
|
||||
Ok(())
|
||||
}
|
||||
@@ -1071,6 +1116,17 @@ impl BrowserManager {
|
||||
};
|
||||
|
||||
if let Some(ref error_text) = nav_result.error_text {
|
||||
// `data:` URLs abort over the extension relay: chrome.debugger /
|
||||
// chrome.tabs can't drive a top-frame data: navigation, so it comes
|
||||
// back net::ERR_ABORTED on an about:blank tab. Explain it instead of
|
||||
// leaking the cryptic code (data: works fine under `--launch`).
|
||||
if url.starts_with("data:") && error_text.contains("ERR_ABORTED") {
|
||||
return Err(format!(
|
||||
"Navigation failed: {error_text}. Chrome blocks top-frame `data:` URLs over \
|
||||
the extension relay — use a real http(s):// or file:// URL, or run with \
|
||||
`--launch` (where data: URLs work)."
|
||||
));
|
||||
}
|
||||
return Err(format!("Navigation failed: {}", error_text));
|
||||
}
|
||||
|
||||
@@ -1515,6 +1571,18 @@ impl BrowserManager {
|
||||
/// the active tab, per #7/#8.1); the caller surfaces it so the agent knows a
|
||||
/// tab opened instead of seeing the old page (issue #24-A).
|
||||
pub async fn adopt_newly_opened(&mut self, before: &HashSet<String>) -> Option<PageInfo> {
|
||||
// STRICT MULTI-AGENT ISOLATION: on the relay this session's `before` set is
|
||||
// only its OWN tabs, so EVERY foreign tab (the user's, other agents') looks
|
||||
// "new" relative to it and would be adopted here — exactly the leak where a
|
||||
// concurrent agent's tabs (github/Lark/iphone-use) showed up in this
|
||||
// session mid-flow. A tab the agent itself opened (a pop-up) can't be
|
||||
// distinguished from a foreign tab over the relay (no opener/window/group
|
||||
// in the synthesized targetInfo), so don't adopt anything: the agent drives
|
||||
// only tabs it explicitly created, and pop-ups (e.g. an OAuth/login window)
|
||||
// are the user's. A launched browser (every tab ours) still follows pop-ups.
|
||||
if self.agent_group().is_some() {
|
||||
return None;
|
||||
}
|
||||
let result: GetTargetsResult = self
|
||||
.client
|
||||
.send_command_typed("Target.getTargets", &json!({}), None)
|
||||
@@ -1557,6 +1625,11 @@ impl BrowserManager {
|
||||
title: sanitize_title(&target.title),
|
||||
target_type: target.target_type.clone(),
|
||||
};
|
||||
// A tab that appeared right after THIS session's action (a click that
|
||||
// opened a popup/new tab) is ours — record it as owned so it's tracked,
|
||||
// protected from churn-pruning, and cleaned up on close, consistent with
|
||||
// strict multi-agent isolation (we only ever own tabs we created/opened).
|
||||
self.created_targets.insert(target.target_id.clone());
|
||||
self.add_background_page(page.clone());
|
||||
let _ = self.enable_domains(&attach.session_id).await;
|
||||
if opened.is_none() {
|
||||
@@ -1584,13 +1657,19 @@ impl BrowserManager {
|
||||
.filter(should_track_target)
|
||||
.collect();
|
||||
let live_ids: HashSet<String> = live.iter().map(|t| t.target_id.clone()).collect();
|
||||
let on_relay = self.agent_group().is_some();
|
||||
|
||||
for target in &live {
|
||||
if self.update_page_target_info(target) {
|
||||
continue;
|
||||
}
|
||||
// A target this session hasn't tracked yet — attach and add it in the
|
||||
// background so it's listable/adoptable without stealing the active tab.
|
||||
// STRICT MULTI-AGENT ISOLATION: on the relay (the user's real Chrome,
|
||||
// shared with other agents), NEVER adopt a tab this session didn't
|
||||
// create — it belongs to the user or another agent's group. Only a
|
||||
// browser we launched (every tab ours) adopts unknown targets.
|
||||
if on_relay {
|
||||
continue;
|
||||
}
|
||||
let attach_result: AttachToTargetResult = match self
|
||||
.client
|
||||
.send_command_typed(
|
||||
@@ -1621,12 +1700,26 @@ impl BrowserManager {
|
||||
let _ = self.enable_domains(&attach_result.session_id).await;
|
||||
}
|
||||
|
||||
// Drop tabs that no longer exist so `tab list` doesn't show phantom rows —
|
||||
// but never prune the explicitly-pinned active target on a transient
|
||||
// getTargets snapshot (issue #31; see `prunable_target_ids`).
|
||||
let gone = prunable_target_ids(&self.pages, &live_ids, self.active_target_id.as_deref());
|
||||
for tid in gone {
|
||||
self.remove_page_by_target_id(&tid);
|
||||
// Prune tabs that are gone. On a LAUNCHED browser a missing target really
|
||||
// is closed, so prune immediately. On the RELAY a single `getTargets`
|
||||
// snapshot routinely omits live tabs (multi-agent churn, a brief
|
||||
// cross-process-nav gap) — dropping the tab we're driving on one bad
|
||||
// snapshot is the failure we're fixing — so prune only after the tab has
|
||||
// been absent for several CONSECUTIVE snapshots (debounced). The pinned
|
||||
// active target is protected either way (issue #31).
|
||||
let gone = if on_relay {
|
||||
debounced_prune_ids(
|
||||
&self.pages,
|
||||
&live_ids,
|
||||
self.active_target_id.as_deref(),
|
||||
&mut self.relay_target_misses,
|
||||
)
|
||||
} else {
|
||||
prunable_target_ids(&self.pages, &live_ids, self.active_target_id.as_deref())
|
||||
};
|
||||
for tid in &gone {
|
||||
self.relay_target_misses.remove(tid);
|
||||
self.remove_page_by_target_id(tid);
|
||||
}
|
||||
|
||||
// Refresh url/title from each live tab. The relay only stamps target_info
|
||||
@@ -2536,6 +2629,7 @@ async fn initialize_lightpanda_manager(
|
||||
visited_origins: HashSet::new(),
|
||||
created_targets: HashSet::new(),
|
||||
active_target_id: None,
|
||||
relay_target_misses: HashMap::new(),
|
||||
next_tab_id: 1,
|
||||
capture_console: console_capture_enabled(),
|
||||
};
|
||||
@@ -2992,6 +3086,43 @@ mod tests {
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn debounced_prune_tolerates_transient_churn() {
|
||||
// Multi-agent churn: a single getTargets snapshot omits our owned tab "B"
|
||||
// (another agent opened/closed tabs). It must NOT be pruned on one miss.
|
||||
let pages = vec![page("A"), page("B")];
|
||||
let mut misses = HashMap::new();
|
||||
let empty: HashSet<String> = HashSet::new();
|
||||
// Misses 1 and 2: B absent but under threshold → not pruned.
|
||||
assert!(debounced_prune_ids(&pages, &empty, Some("A"), &mut misses).is_empty());
|
||||
assert!(debounced_prune_ids(&pages, &empty, Some("A"), &mut misses).is_empty());
|
||||
// Miss 3 (== RELAY_PRUNE_MISSES): genuinely gone → pruned.
|
||||
assert_eq!(
|
||||
debounced_prune_ids(&pages, &empty, Some("A"), &mut misses),
|
||||
vec!["B".to_string()]
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn debounced_prune_resets_on_reappearance_and_protects_pin() {
|
||||
let pages = vec![page("A"), page("B")];
|
||||
let mut misses = HashMap::new();
|
||||
let empty: HashSet<String> = HashSet::new();
|
||||
let mut live_b: HashSet<String> = HashSet::new();
|
||||
live_b.insert("B".to_string());
|
||||
// Two misses for B, then it reappears → counter resets, so it survives
|
||||
// indefinitely under intermittent churn.
|
||||
debounced_prune_ids(&pages, &empty, Some("A"), &mut misses);
|
||||
debounced_prune_ids(&pages, &empty, Some("A"), &mut misses);
|
||||
assert!(debounced_prune_ids(&pages, &live_b, Some("A"), &mut misses).is_empty());
|
||||
assert!(debounced_prune_ids(&pages, &empty, Some("A"), &mut misses).is_empty()); // back to miss 1
|
||||
// The pinned active "A" is never pruned no matter how many misses.
|
||||
for _ in 0..5 {
|
||||
let gone = debounced_prune_ids(&pages, &empty, Some("A"), &mut misses);
|
||||
assert!(!gone.contains(&"A".to_string()));
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn resolve_active_index_pin_survives_passive_background_tab() {
|
||||
// A foreign tab ("Z") gets appended by passive discovery after we pinned
|
||||
|
||||
@@ -35,6 +35,7 @@ fn native_test_fixture_html(name: &str) -> &'static str {
|
||||
"html5_drag_probe" => include_str!("test_fixtures/html5_drag_probe.html"),
|
||||
"pointer_capture_probe" => include_str!("test_fixtures/pointer_capture_probe.html"),
|
||||
"upload_probe" => include_str!("test_fixtures/upload_probe.html"),
|
||||
"iframe_button_probe" => include_str!("test_fixtures/iframe_button_probe.html"),
|
||||
_ => panic!("Unknown native test fixture: {}", name),
|
||||
}
|
||||
}
|
||||
@@ -573,6 +574,76 @@ async fn e2e_snapshot_and_click_ref() {
|
||||
assert_success(&resp);
|
||||
}
|
||||
|
||||
/// Clicking a button INSIDE an iframe by `@ref` must deliver a TRUSTED activation
|
||||
/// (`event.isTrusted === true`), not a synthetic DOM `.click()`. Security-sensitive
|
||||
/// embedded forms (Google Payments' `保存`) reject `isTrusted:false` clicks, so an
|
||||
/// enabled submit button silently no-op'd (issue #39). The fix routes iframe-ref
|
||||
/// clicks to a real `Input.dispatchMouseEvent` on the element's own frame session.
|
||||
/// The fixture's iframe button writes `clicked:<isTrusted>` into its own text on
|
||||
/// click, which the cross-frame snapshot reads back.
|
||||
#[tokio::test]
|
||||
#[ignore]
|
||||
async fn e2e_iframe_button_click_is_trusted() {
|
||||
let mut state = DaemonState::new();
|
||||
|
||||
let resp = execute_command(
|
||||
&json!({ "id": "1", "action": "launch", "headless": true }),
|
||||
&mut state,
|
||||
)
|
||||
.await;
|
||||
assert_success(&resp);
|
||||
|
||||
let resp = execute_command(
|
||||
&json!({ "id": "2", "action": "navigate", "url": native_test_fixture_url("iframe_button_probe") }),
|
||||
&mut state,
|
||||
)
|
||||
.await;
|
||||
assert_success(&resp);
|
||||
|
||||
// Snapshot (interactive) — the button lives in the iframe and must appear with
|
||||
// a ref; that ref carries the frame_id so the click resolves into the frame.
|
||||
let resp = execute_command(
|
||||
&json!({ "id": "3", "action": "snapshot", "interactive": true }),
|
||||
&mut state,
|
||||
)
|
||||
.await;
|
||||
assert_success(&resp);
|
||||
let snapshot = get_data(&resp)["snapshot"].as_str().unwrap_or("");
|
||||
let ref_id = snapshot
|
||||
.lines()
|
||||
.find(|l| l.contains("button \"save\""))
|
||||
.and_then(|l| l.split("ref=").nth(1))
|
||||
.map(|r| r.trim_end_matches(']').trim())
|
||||
.unwrap_or_else(|| panic!("iframe button not found in snapshot:\n{snapshot}"));
|
||||
|
||||
// Click it by ref.
|
||||
let resp = execute_command(
|
||||
&json!({ "id": "4", "action": "click", "selector": ref_id }),
|
||||
&mut state,
|
||||
)
|
||||
.await;
|
||||
assert_success(&resp);
|
||||
|
||||
tokio::time::sleep(tokio::time::Duration::from_millis(300)).await;
|
||||
|
||||
// The button rewrote its own text with the click's isTrusted flag; read it
|
||||
// back across frames.
|
||||
let resp = execute_command(
|
||||
&json!({ "id": "5", "action": "snapshot", "interactive": true }),
|
||||
&mut state,
|
||||
)
|
||||
.await;
|
||||
assert_success(&resp);
|
||||
let after = get_data(&resp)["snapshot"].as_str().unwrap_or("");
|
||||
assert!(
|
||||
after.contains("clicked:true"),
|
||||
"iframe button click must be trusted (isTrusted:true); snapshot:\n{after}"
|
||||
);
|
||||
|
||||
let resp = execute_command(&json!({ "id": "99", "action": "close" }), &mut state).await;
|
||||
assert_success(&resp);
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Screenshot
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
@@ -56,19 +56,34 @@ pub async fn click(
|
||||
.await;
|
||||
}
|
||||
|
||||
// Over the extension relay we drive the user's real, in-use Chrome, where a
|
||||
// coordinate `Input.dispatchMouseEvent` is NOT reliably confined to our target
|
||||
// tab — it can be delivered to whatever tab is in the foreground, and an OOPIF
|
||||
// element's box can't be mapped to a top-viewport point at all. This twice
|
||||
// opened an unrelated tab on the user's busy Chrome (issues #31/#36). So on the
|
||||
// relay, never use coordinates for a normal left click: DOM-dispatch invokes
|
||||
// the element's click in its own (frame) session, always hitting the right
|
||||
// element in the right tab. Double/right clicks still need true pointer
|
||||
// semantics, and `coord` mode is an explicit opt-out.
|
||||
// An element INSIDE an iframe needs a TRUSTED activation: a DOM `.click()` is
|
||||
// `isTrusted:false`, which security-sensitive embedded forms reject — Google
|
||||
// Payments' enabled `保存` button silently no-ops on a synthetic click (issue
|
||||
// #39). A coordinate `Input.dispatchMouseEvent` can't help either: `getBoxModel`
|
||||
// for a sub-frame node returns frame-local coordinates that don't compose the
|
||||
// iframe's offset, so the click lands in the wrong place. The frame-agnostic
|
||||
// trusted path is keyboard activation — focus the element in its own frame, then
|
||||
// dispatch a real Enter on the page session; Chrome routes the key to the
|
||||
// focused element regardless of frame (same as `type --focused`), and Enter on a
|
||||
// focused button/link fires a trusted `click`. `coord` mode opts out.
|
||||
let in_iframe = ref_map.ref_is_in_iframe(selector_or_ref);
|
||||
if mode != "coord" && button == "left" && click_count == 1 && in_iframe {
|
||||
return dom_activate(
|
||||
client,
|
||||
session_id,
|
||||
ref_map,
|
||||
selector_or_ref,
|
||||
iframe_sessions,
|
||||
)
|
||||
.await;
|
||||
}
|
||||
// On the relay (the user's real Chrome) a TOP-document coordinate click used to
|
||||
// drift onto the foreground tab; that root cause is fixed (#5: the agent drives
|
||||
// its own pinned tab), but DOM-dispatch stays the conservative default here.
|
||||
if mode != "coord"
|
||||
&& button == "left"
|
||||
&& click_count == 1
|
||||
&& prefer_dom_dispatch(ref_map, selector_or_ref)
|
||||
&& crate::connect::relay_url().is_some()
|
||||
{
|
||||
return dom_click(
|
||||
client,
|
||||
@@ -270,6 +285,71 @@ async fn dom_click(
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Trusted activation of an element inside an iframe (issue #39). Focuses the
|
||||
/// element in its own frame session, then dispatches a real Enter/Space on the
|
||||
/// page session — Chrome routes the key to the focused element across frames, and
|
||||
/// Enter/Space on a focused button/link/checkbox fires a `click` with
|
||||
/// `isTrusted: true`, which security-sensitive embedded forms (Google Payments
|
||||
/// `保存`) require. Non-activatable roles (a `div[onclick]`) can't be keyboard-
|
||||
/// activated, so they fall back to a DOM `.click()`.
|
||||
async fn dom_activate(
|
||||
client: &CdpClient,
|
||||
session_id: &str,
|
||||
ref_map: &RefMap,
|
||||
selector_or_ref: &str,
|
||||
iframe_sessions: &HashMap<String, String>,
|
||||
) -> Result<(), String> {
|
||||
let role = parse_ref(selector_or_ref)
|
||||
.and_then(|r| ref_map.get(&r).map(|e| e.role.clone()))
|
||||
.unwrap_or_default();
|
||||
// Space toggles checkbox-like controls; Enter activates buttons/links/menus.
|
||||
let key = match role.as_str() {
|
||||
"checkbox" | "radio" | "switch" | "option" | "menuitemcheckbox" | "menuitemradio" => {
|
||||
Some("space")
|
||||
}
|
||||
"button" | "link" | "menuitem" | "tab" | "treeitem" => Some("enter"),
|
||||
_ => None,
|
||||
};
|
||||
let Some(key) = key else {
|
||||
// Not keyboard-activatable — best effort via DOM .click() (untrusted).
|
||||
return dom_click(
|
||||
client,
|
||||
session_id,
|
||||
ref_map,
|
||||
selector_or_ref,
|
||||
iframe_sessions,
|
||||
)
|
||||
.await;
|
||||
};
|
||||
|
||||
let (object_id, effective_session_id) = resolve_element_object_id(
|
||||
client,
|
||||
session_id,
|
||||
ref_map,
|
||||
selector_or_ref,
|
||||
iframe_sessions,
|
||||
)
|
||||
.await?;
|
||||
// Focus the element in its OWN frame session so the keystroke lands on it.
|
||||
client
|
||||
.send_command_typed::<_, Value>(
|
||||
"Runtime.callFunctionOn",
|
||||
&CallFunctionOnParams {
|
||||
function_declaration: "function() { this.focus(); }".to_string(),
|
||||
object_id: Some(object_id),
|
||||
arguments: None,
|
||||
return_by_value: Some(true),
|
||||
await_promise: Some(false),
|
||||
},
|
||||
Some(&effective_session_id),
|
||||
)
|
||||
.await?;
|
||||
// Trusted key on the page session — routed to the focused (in-frame) element.
|
||||
press_key(client, session_id, key).await?;
|
||||
wait_for_paint_settled(client, &effective_session_id).await;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// DOM-dispatch a double-click on the element in its own session (no coordinates)
|
||||
/// — the relay/iframe-safe counterpart to a coordinate dblclick. Fires the full
|
||||
/// click,click,dblclick sequence so handlers bound to any of them respond.
|
||||
|
||||
@@ -0,0 +1,28 @@
|
||||
<!doctype html>
|
||||
<html>
|
||||
<head>
|
||||
<meta charset="utf-8" />
|
||||
<title>iframe button probe</title>
|
||||
</head>
|
||||
<body>
|
||||
<h1>iframe button probe</h1>
|
||||
<iframe
|
||||
id="frame"
|
||||
width="320"
|
||||
height="140"
|
||||
srcdoc="
|
||||
<!doctype html>
|
||||
<html>
|
||||
<body style='margin:24px'>
|
||||
<button id='b' style='padding:24px;font-size:22px'>save</button>
|
||||
<script>
|
||||
document.getElementById('b').addEventListener('click', function (e) {
|
||||
this.textContent = 'clicked:' + e.isTrusted;
|
||||
});
|
||||
</script>
|
||||
</body>
|
||||
</html>
|
||||
"
|
||||
></iframe>
|
||||
</body>
|
||||
</html>
|
||||
@@ -3366,6 +3366,14 @@ Batch:
|
||||
batch [--bail] ["cmd" ...] Execute multiple commands sequentially (args or stdin)
|
||||
--bail stops on first error (default: continue all)
|
||||
|
||||
Site adapters: turn a website into a structured-data CLI (runs as you, in your tab)
|
||||
site update Fetch the community adapter pack into ~/.chrome-use/sites
|
||||
site list List installed adapters (name/cmd)
|
||||
site info <name>/<cmd> Show an adapter's @meta (args, domain, capabilities)
|
||||
site <name>/<cmd> [args] Run an adapter: navigate to its site + return JSON
|
||||
e.g. site github/issues epiral/repo, site reddit/search rust
|
||||
Positional args fill declared args in order; --key value overrides
|
||||
|
||||
Auth Vault:
|
||||
auth save <name> [opts] Save auth profile (--url, --username, --password/--password-stdin)
|
||||
auth login <name> Login using saved credentials (waits for form fields)
|
||||
@@ -3577,6 +3585,9 @@ iOS Simulator (requires Xcode and Appium):
|
||||
chrome-use -p ios device list # List simulators
|
||||
chrome-use -p ios swipe up # Swipe gesture
|
||||
chrome-use -p ios tap @e1 # Touch element
|
||||
|
||||
Hit a bug or rough edge? A 30-second issue genuinely sharpens this tool:
|
||||
https://github.com/leeguooooo/chrome-use/issues
|
||||
"#
|
||||
);
|
||||
}
|
||||
@@ -3659,6 +3670,7 @@ fn print_screenshot_diff(data: &serde_json::Map<String, serde_json::Value>) {
|
||||
|
||||
pub fn print_version() {
|
||||
println!("chrome-use {}", env!("CARGO_PKG_VERSION"));
|
||||
println!("report bugs / rough edges: https://github.com/leeguooooo/chrome-use/issues");
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
|
||||
+267
@@ -0,0 +1,267 @@
|
||||
//! Site adapters: turn any website into a structured-data CLI by running a small
|
||||
//! per-command JS adapter inside your real, logged-in browser tab (it reuses the
|
||||
//! site's cookies / same-origin fetch / its own webpack modules — the site thinks
|
||||
//! it's you, because it is).
|
||||
//!
|
||||
//! The adapter format is the community **bb-sites** convention
|
||||
//! (<https://github.com/epiral/bb-sites>): one `.js` file per command, a
|
||||
//! `/* @meta {...} */` JSON header (name, description, domain, args), then an
|
||||
//! `async function(args){ ... return {...} }`. chrome-use ships none of those
|
||||
//! adapters — `chrome-use site update` fetches the upstream repo at runtime into
|
||||
//! `~/.chrome-use/sites` (like a package manager pulling a dependency), so the
|
||||
//! adapters stay the property of their authors. Running an adapter navigates to
|
||||
//! its `@meta.domain` and `eval`s the function in the site's own logged-in page.
|
||||
|
||||
use std::path::PathBuf;
|
||||
|
||||
use serde_json::Value;
|
||||
|
||||
const SITES_ZIP_URL: &str = "https://github.com/epiral/bb-sites/archive/refs/heads/main.zip";
|
||||
|
||||
/// `~/.chrome-use/sites` — where synced adapters live.
|
||||
pub fn sites_dir() -> Option<PathBuf> {
|
||||
dirs_home().map(|h| h.join(".chrome-use").join("sites"))
|
||||
}
|
||||
|
||||
fn dirs_home() -> Option<PathBuf> {
|
||||
std::env::var_os("HOME").map(PathBuf::from)
|
||||
}
|
||||
|
||||
/// Parsed adapter: its `@meta` JSON and the raw `async function(args){...}` source.
|
||||
pub struct Adapter {
|
||||
pub meta: Value,
|
||||
pub func_src: String,
|
||||
}
|
||||
|
||||
impl Adapter {
|
||||
pub fn domain(&self) -> Option<&str> {
|
||||
self.meta.get("domain").and_then(|v| v.as_str())
|
||||
}
|
||||
}
|
||||
|
||||
/// Load `<sites>/<name>/<cmd>.js`, splitting the `/* @meta {...} */` header from
|
||||
/// the function body. `spec` is `name/cmd`.
|
||||
pub fn load_adapter(spec: &str) -> Result<Adapter, String> {
|
||||
let (name, cmd) = spec
|
||||
.split_once('/')
|
||||
.ok_or_else(|| format!("site: expected <name>/<command>, got `{spec}`"))?;
|
||||
if name.is_empty()
|
||||
|| cmd.is_empty()
|
||||
|| name.contains("..")
|
||||
|| cmd.contains("..")
|
||||
|| name.contains('/')
|
||||
|| cmd.contains('/')
|
||||
{
|
||||
return Err(format!("site: invalid adapter spec `{spec}`"));
|
||||
}
|
||||
let dir = sites_dir().ok_or("site: cannot resolve home dir")?;
|
||||
let path = dir.join(name).join(format!("{cmd}.js"));
|
||||
if !path.exists() {
|
||||
return Err(format!(
|
||||
"site: adapter `{spec}` not found. Run `chrome-use site update` to sync adapters, \
|
||||
or `chrome-use site list` to see what's installed."
|
||||
));
|
||||
}
|
||||
let raw = std::fs::read_to_string(&path).map_err(|e| format!("site: read {spec}: {e}"))?;
|
||||
parse_adapter(&raw, spec)
|
||||
}
|
||||
|
||||
/// Split the `@meta` JSON block and the function source from an adapter file.
|
||||
pub fn parse_adapter(raw: &str, spec: &str) -> Result<Adapter, String> {
|
||||
let start = raw
|
||||
.find("@meta")
|
||||
.and_then(|i| raw[i..].find('{').map(|j| i + j))
|
||||
.ok_or_else(|| format!("site: {spec} missing /* @meta {{...}} */ header"))?;
|
||||
// Find the matching close brace for the @meta object (brace-count, string-aware).
|
||||
let bytes = raw.as_bytes();
|
||||
let mut depth = 0i32;
|
||||
let mut in_str = false;
|
||||
let mut esc = false;
|
||||
let mut end = None;
|
||||
for (k, &b) in bytes.iter().enumerate().skip(start) {
|
||||
if in_str {
|
||||
if esc {
|
||||
esc = false;
|
||||
} else if b == b'\\' {
|
||||
esc = true;
|
||||
} else if b == b'"' {
|
||||
in_str = false;
|
||||
}
|
||||
continue;
|
||||
}
|
||||
match b {
|
||||
b'"' => in_str = true,
|
||||
b'{' => depth += 1,
|
||||
b'}' => {
|
||||
depth -= 1;
|
||||
if depth == 0 {
|
||||
end = Some(k + 1);
|
||||
break;
|
||||
}
|
||||
}
|
||||
_ => {}
|
||||
}
|
||||
}
|
||||
let end = end.ok_or_else(|| format!("site: {spec} @meta header has no closing brace"))?;
|
||||
let meta: Value = serde_json::from_str(&raw[start..end])
|
||||
.map_err(|e| format!("site: {spec} @meta is not valid JSON: {e}"))?;
|
||||
// The function is everything after the meta comment's closing `*/`.
|
||||
let after = raw[end..].find("*/").map(|i| end + i + 2).unwrap_or(end);
|
||||
let func_src = raw[after..].trim().to_string();
|
||||
if func_src.is_empty() {
|
||||
return Err(format!("site: {spec} has no function body after @meta"));
|
||||
}
|
||||
Ok(Adapter { meta, func_src })
|
||||
}
|
||||
|
||||
/// Build the JS to eval: `(<adapter function>)(<args JSON>)`. The adapter's
|
||||
/// `async function(args)` returns a promise; chrome-use's eval awaits it.
|
||||
pub fn build_eval(adapter: &Adapter, args: &Value) -> String {
|
||||
let args_json = serde_json::to_string(args).unwrap_or_else(|_| "{}".to_string());
|
||||
format!("({})({})", adapter.func_src, args_json)
|
||||
}
|
||||
|
||||
/// List installed adapters as `name/cmd` strings (sorted).
|
||||
pub fn list_adapters() -> Result<Vec<String>, String> {
|
||||
let dir = sites_dir().ok_or("site: cannot resolve home dir")?;
|
||||
if !dir.exists() {
|
||||
return Ok(Vec::new());
|
||||
}
|
||||
let mut out = Vec::new();
|
||||
for site in std::fs::read_dir(&dir)
|
||||
.map_err(|e| e.to_string())?
|
||||
.flatten()
|
||||
{
|
||||
if !site.path().is_dir() {
|
||||
continue;
|
||||
}
|
||||
let name = site.file_name().to_string_lossy().to_string();
|
||||
for cmd in std::fs::read_dir(site.path())
|
||||
.map_err(|e| e.to_string())?
|
||||
.flatten()
|
||||
{
|
||||
let p = cmd.path();
|
||||
if p.extension().and_then(|e| e.to_str()) == Some("js") {
|
||||
if let Some(stem) = p.file_stem().and_then(|s| s.to_str()) {
|
||||
out.push(format!("{name}/{stem}"));
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
out.sort();
|
||||
Ok(out)
|
||||
}
|
||||
|
||||
/// Download the bb-sites repo zip and extract its adapters into `~/.chrome-use/sites`.
|
||||
pub async fn update() -> Result<usize, String> {
|
||||
let dir = sites_dir().ok_or("site: cannot resolve home dir")?;
|
||||
let client = reqwest::Client::builder()
|
||||
.user_agent("chrome-use")
|
||||
.build()
|
||||
.map_err(|e| e.to_string())?;
|
||||
let bytes = client
|
||||
.get(SITES_ZIP_URL)
|
||||
.send()
|
||||
.await
|
||||
.map_err(|e| format!("site update: download failed: {e}"))?
|
||||
.error_for_status()
|
||||
.map_err(|e| format!("site update: {e}"))?
|
||||
.bytes()
|
||||
.await
|
||||
.map_err(|e| format!("site update: read body: {e}"))?;
|
||||
|
||||
let cursor = std::io::Cursor::new(bytes);
|
||||
let mut zip = zip::ZipArchive::new(cursor).map_err(|e| format!("site update: bad zip: {e}"))?;
|
||||
std::fs::create_dir_all(&dir).map_err(|e| e.to_string())?;
|
||||
let mut count = 0usize;
|
||||
for i in 0..zip.len() {
|
||||
let mut f = zip.by_index(i).map_err(|e| e.to_string())?;
|
||||
let Some(enclosed) = f.enclosed_name() else {
|
||||
continue;
|
||||
};
|
||||
// Strip the top-level `bb-sites-main/` component from the archive path.
|
||||
let rel: PathBuf = enclosed.components().skip(1).collect();
|
||||
if rel.as_os_str().is_empty() {
|
||||
continue;
|
||||
}
|
||||
let out = dir.join(&rel);
|
||||
if f.is_dir() {
|
||||
let _ = std::fs::create_dir_all(&out);
|
||||
continue;
|
||||
}
|
||||
if let Some(parent) = out.parent() {
|
||||
let _ = std::fs::create_dir_all(parent);
|
||||
}
|
||||
let mut buf = Vec::new();
|
||||
std::io::copy(&mut f, &mut buf).map_err(|e| e.to_string())?;
|
||||
std::fs::write(&out, &buf).map_err(|e| e.to_string())?;
|
||||
if out.extension().and_then(|e| e.to_str()) == Some("js") {
|
||||
count += 1;
|
||||
}
|
||||
}
|
||||
Ok(count)
|
||||
}
|
||||
|
||||
/// Map CLI args to the adapter's `args` object. Positional args fill the adapter's
|
||||
/// declared `args` keys in order; `--key value` overrides by name. The adapter
|
||||
/// validates required args itself.
|
||||
pub fn map_args(adapter: &Adapter, positional: &[String], named: &[(String, String)]) -> Value {
|
||||
let mut obj = serde_json::Map::new();
|
||||
let keys: Vec<String> = adapter
|
||||
.meta
|
||||
.get("args")
|
||||
.and_then(|a| a.as_object())
|
||||
.map(|m| m.keys().cloned().collect())
|
||||
.unwrap_or_default();
|
||||
for (i, val) in positional.iter().enumerate() {
|
||||
if let Some(k) = keys.get(i) {
|
||||
obj.insert(k.clone(), Value::String(val.clone()));
|
||||
}
|
||||
}
|
||||
for (k, v) in named {
|
||||
obj.insert(k.clone(), Value::String(v.clone()));
|
||||
}
|
||||
Value::Object(obj)
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
const SAMPLE: &str = r#"/* @meta
|
||||
{
|
||||
"name": "github/issues",
|
||||
"domain": "github.com",
|
||||
"args": { "repo": {"required": true}, "state": {"required": false} }
|
||||
}
|
||||
*/
|
||||
|
||||
async function(args) { return { repo: args.repo }; }"#;
|
||||
|
||||
#[test]
|
||||
fn parses_meta_and_function() {
|
||||
let a = parse_adapter(SAMPLE, "github/issues").unwrap();
|
||||
assert_eq!(a.domain(), Some("github.com"));
|
||||
assert!(a.func_src.starts_with("async function(args)"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn build_eval_wraps_and_passes_args() {
|
||||
let a = parse_adapter(SAMPLE, "github/issues").unwrap();
|
||||
let args = map_args(
|
||||
&a,
|
||||
&["owner/repo".into()],
|
||||
&[("state".into(), "closed".into())],
|
||||
);
|
||||
let js = build_eval(&a, &args);
|
||||
assert!(js.contains("async function(args)"));
|
||||
assert!(js.contains("\"repo\":\"owner/repo\""));
|
||||
assert!(js.contains("\"state\":\"closed\""));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn rejects_bad_spec() {
|
||||
assert!(load_adapter("noslash").is_err());
|
||||
assert!(load_adapter("../etc/passwd").is_err());
|
||||
}
|
||||
}
|
||||
+1
-1
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "chrome-use",
|
||||
"version": "1.5.14",
|
||||
"version": "1.5.19",
|
||||
"description": "chrome-use — drive your real, logged-in Chrome from any AI agent, stealth by default",
|
||||
"type": "module",
|
||||
"packageManager": "pnpm@11.1.3",
|
||||
|
||||
@@ -68,6 +68,7 @@ need a **real, logged-in browser** — not for reading text off a public page.
|
||||
|---|---|
|
||||
| Discover what exists / find sources | `WebSearch` |
|
||||
| Specific facts from a static or public page | `WebFetch` or `curl` (no browser) |
|
||||
| **Structured data from a known site** (GitHub issues, Reddit/HN search, Bilibili/Twitter feed, …) — esp. behind login | `chrome-use site <name>/<cmd>` (see below) — skip snapshot+click entirely |
|
||||
| Login state, interaction, JS-rendered or anti-bot pages | **chrome-use** (this skill) |
|
||||
| A page the user saved before / an internal system | `chrome-use find-url <keywords>` (their bookmarks), then open it |
|
||||
| The user's **own already-open, logged-in** Chrome window | the **extension connect** flow (below) |
|
||||
@@ -131,7 +132,17 @@ Each `--session` that connects gets its **own colored Chrome tab group** (named
|
||||
after the session) and drives only its own tabs — multiple agents share the one
|
||||
real browser without cross-talk, and the user's own tabs are never grouped. CDP
|
||||
drives the page without moving the user's mouse/keyboard, so it doesn't fight
|
||||
them for control. **Anti-detection ranking: this real logged-in Chrome (extension
|
||||
them for control.
|
||||
|
||||
**Strict multi-agent isolation.** A session over the relay tracks and drives
|
||||
**only the tabs it created** (its own group). It does **not** adopt the user's
|
||||
existing tabs, other agents' tabs, or pop-ups (e.g. an OAuth/login window — that's
|
||||
the user's), so several agents (and other tools opening tabs) can work in the same
|
||||
real Chrome concurrently without ever dropping or stealing each other's tabs —
|
||||
another agent's tab churn can't make your bound tab vanish or drift your commands
|
||||
onto the wrong page. Consequence: `tab list` shows only *your* session's tabs; to
|
||||
drive a specific page, navigate to it in your own tab instead of expecting a
|
||||
pre-existing or popped-up tab to appear in the list. **Anti-detection ranking: this real logged-in Chrome (extension
|
||||
connect) > a headed launched browser > headless (forbidden).** A genuine human
|
||||
browser has no headless/automation tells at all, so prefer it for anything
|
||||
anti-bot-sensitive.
|
||||
@@ -186,6 +197,27 @@ chrome-use eval "[...document.forms[0].elements].filter(e=>!e.validity.valid).ma
|
||||
chrome-use eval "document.querySelector('#stubborn').click()" # direct DOM click, bypasses overlays
|
||||
```
|
||||
|
||||
## Site adapters — the cheapest path for "read structured data from site X"
|
||||
|
||||
Before you `open` + `snapshot` + click your way through GitHub/Reddit/Bilibili/etc.,
|
||||
check whether a **site adapter** already exists. An adapter is a community-written JS
|
||||
function that hits the site's own JSON API *from inside your logged-in tab* and returns
|
||||
clean structured data — no clicking, no scraping, no screenshots. It's the same idea as
|
||||
`eval`, packaged per-site.
|
||||
|
||||
```bash
|
||||
chrome-use site update # one-time: fetch the adapter pack (~145 cmds)
|
||||
chrome-use site list # what's installed (github/issues, reddit/search, …)
|
||||
chrome-use site info github/issues # an adapter's args + which domain it runs on
|
||||
chrome-use site github/issues owner/repo --json # run it → JSON (navigates there for you)
|
||||
```
|
||||
|
||||
- Positional args fill the adapter's declared args **in order**; `--key value` overrides by name.
|
||||
- It navigates to the adapter's domain (reusing the current tab if you're already on it), so
|
||||
login-gated feeds (`bilibili/feed`, `twitter/...`) work because they run as *you*.
|
||||
- If no adapter fits, fall back to the normal `snapshot`/`eval` loop. Adapters come from the
|
||||
[bb-sites](https://github.com/epiral/bb-sites) community pack; chrome-use fetches & runs them.
|
||||
|
||||
## Quickstart
|
||||
|
||||
```bash
|
||||
|
||||
Reference in New Issue
Block a user