Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
d8f484eded | ||
|
|
1eb40eabd5 | ||
|
|
601404ba72 | ||
|
|
fd10766762 | ||
|
|
ba9b167ede | ||
|
|
c077593e99 | ||
|
|
02e23ebe11 | ||
|
|
d5cd9cd621 | ||
|
|
284a60a54c | ||
|
|
10d196b6eb | ||
|
|
5be01e292d | ||
|
|
a83d1b1df9 | ||
|
|
50b27ac0e0 | ||
|
|
d81bc01645 | ||
|
|
c667e0e704 | ||
|
|
08cb8dbeb9 | ||
|
|
fd2cdcde77 | ||
|
|
8e001e3d88 | ||
|
|
eb2bc343a0 | ||
|
|
d86c9c4be2 | ||
|
|
32c25a6627 | ||
|
|
a8ce3dd3f8 | ||
|
|
997373fd57 | ||
|
|
5a858af93f | ||
|
|
58dc02bfdc | ||
|
|
c47601bd7b | ||
|
|
0296bc7a88 | ||
|
|
32e203b908 | ||
|
|
fc51cd63ba | ||
|
|
f714c7920b | ||
|
|
1ac8ef7732 | ||
|
|
9f24e66033 | ||
|
|
70ab38d35f | ||
|
|
6830df50ea |
@@ -205,6 +205,50 @@ chrome-use --launch --profile auto open https://x.com/home
|
||||
|
||||
In CI environments, standalone mode is used automatically.
|
||||
|
||||
## Site adapters — turn a website into a structured-data CLI
|
||||
|
||||
Most "read GitHub issues" / "search Reddit" / "get my Bilibili feed" tasks don't
|
||||
need clicking and screenshotting at all — the site already has a JSON API behind
|
||||
its own login. A **site adapter** is a tiny JS function that calls that API *from
|
||||
inside your logged-in tab* (your cookies, same-origin `fetch`, the site's own
|
||||
modules) and returns clean JSON. The site can't tell it apart from you, because it
|
||||
*is* you.
|
||||
|
||||
chrome-use ships none of these adapters — `site update` fetches the community
|
||||
[**bb-sites**](https://github.com/epiral/bb-sites) pack at runtime (like a package
|
||||
manager pulling a dependency), then runs them over chrome-use's stealth transport:
|
||||
|
||||
```bash
|
||||
chrome-use site update # fetch the adapter pack (~145 commands)
|
||||
chrome-use site list # github/issues, reddit/search, bilibili/feed, …
|
||||
chrome-use site info github/issues # see an adapter's args + domain
|
||||
|
||||
# Run one — navigates to the site (reusing the tab if you're already there) and returns JSON
|
||||
chrome-use site github/issues epiral/bb-browser --json
|
||||
chrome-use site reddit/search "rust async" --json
|
||||
chrome-use site bilibili/feed --json # works because it's your logged-in session
|
||||
```
|
||||
|
||||
Positional args fill the adapter's declared args in order; `--key value` overrides
|
||||
by name. Adapters are authored by the bb-sites community and remain their authors'
|
||||
property — chrome-use just runs them.
|
||||
|
||||
**Auto-sync + auto-suggest.** You rarely type `site update` yourself: chrome-use
|
||||
syncs the pack on first use and refreshes it weekly in the background (tune with
|
||||
`AGENT_BROWSER_SITES_TTL_DAYS`, disable with `AGENT_BROWSER_SITES_NO_AUTO_UPDATE=1`).
|
||||
And when you `open`/`snapshot` a page whose domain has adapters, chrome-use surfaces
|
||||
them right in the output — a `💡 site adapters for <domain>` line, plus a
|
||||
`siteAdapters` field under `--json` — so an agent reaches for the structured-data
|
||||
adapter instead of scraping the DOM:
|
||||
|
||||
```text
|
||||
$ chrome-use open https://github.com
|
||||
💡 site adapters for github.com — prefer these for structured data:
|
||||
github/issues, github/me, github/repo, …
|
||||
e.g. chrome-use site github/issues --json
|
||||
✓ GitHub
|
||||
```
|
||||
|
||||
## Automated testing (`chrome-use test`)
|
||||
|
||||
Turn the repetitive "open it, click around, check it's right" work into a
|
||||
@@ -326,3 +370,6 @@ We deliberately **don't ship our own bot detector** — the strongest, most hone
|
||||
## License
|
||||
|
||||
Apache-2.0
|
||||
---
|
||||
|
||||
> Built by **leeguooooo** — field notes on AI agents, reverse engineering & Cloudflare Workers at **[blog.misonote.com](https://blog.misonote.com)** · follow on **[X @leeguooooo](https://x.com/leeguooooo)**
|
||||
|
||||
+138
-1
@@ -63,6 +63,25 @@ chrome-use 让**任意** agent(Claude Code、Cursor、Codex、你自己的脚
|
||||
|
||||
每个 `--session` 拿到**自己的彩色标签组**,多个 agent 共用同一个真实浏览器、互不干扰,也不动你自己的标签页。
|
||||
|
||||
## 为什么用扩展(而非裸调试端口)
|
||||
|
||||
其他本地工具走裸 `--remote-debugging-port`(CDP)驱动 Chrome。从 **Chrome 136** 起,每次这样连接都会弹出一个阻塞式的 **"Allow remote debugging?"** 同意框 —— 而且端口得提前开好。我们的扩展改用原生消息:**装一次,之后零确认。**
|
||||
|
||||
| | **chrome-use**(本扩展) | web-access(裸 CDP 端口) | Claude in Chrome(chrome.debugger) |
|
||||
|---|---|---|---|
|
||||
| 连接方式 | 原生消息 —— 无端口、无 token | `--remote-debugging-port` | `chrome.debugger` |
|
||||
| **"Allow remote debugging?" 弹框** | **从不** ✅ | **每次连都弹** 🔴 | 无 |
|
||||
| 复用你的真实登录 | 是 | 是 | 是 |
|
||||
| `Runtime.enable`(CDP)泄漏¹ | **默认关闭 → 干净** ✅ | 域已启用 | 不适用 |
|
||||
| CreepJS 隐身分² | **0% stealth · 0% headless** ✅ | 真实 Chrome | 真实 Chrome |
|
||||
| 每会话标签组 / 并发 agent | **支持** ✅ | 无 | 无 |
|
||||
| 为 chrome-use CLI 打造 | 是 | 独立代理 | 单 app 助手 |
|
||||
|
||||
> ¹ 对 [rebrowser-bot-detector](https://bot-detector.rebrowser.net/) 实测:我们的中继报 `runtimeEnableLeak: 🟢 No leak`、`navigatorWebdriver: 🟢`。
|
||||
> ² 对 [CreepJS](https://abrahamjuliot.github.io/creepjs/) 在「连接真实 Chrome」路径上实测 —— 见 [反检测](#反检测)。
|
||||
>
|
||||
> 同意框不是假想:裸端口工具**每次** attach 都会弹(Chrome 136+ 安全策略)。扩展路径从不弹。
|
||||
|
||||
## 安装
|
||||
|
||||
```bash
|
||||
@@ -71,6 +90,15 @@ curl -fsSL https://raw.githubusercontent.com/leeguooooo/chrome-use/main/install.
|
||||
|
||||
从最新的 [GitHub Release](https://github.com/leeguooooo/chrome-use/releases) 下载对应平台的预编译二进制,安装 `chrome-use`(以及 `abs` 别名)。无需 npm,无需 token。
|
||||
|
||||
<details>
|
||||
<summary>其他安装方式</summary>
|
||||
|
||||
- **锁定版本:** `AGENT_BROWSER_VERSION=v0.27.0-fork.12 curl -fsSL https://raw.githubusercontent.com/leeguooooo/chrome-use/main/install.sh | sh`
|
||||
- **自定义路径:** `AGENT_BROWSER_BIN_DIR=$HOME/bin curl -fsSL … | sh`
|
||||
- **Windows:** 从 [Releases 页](https://github.com/leeguooooo/chrome-use/releases) 下载 `chrome-use-win32-x64.tar.gz`,把 `chrome-use.exe` 放进 PATH。
|
||||
- **npm(旧渠道):** `npm install -g chrome-use` —— 仍在发布,但 GitHub Releases 现在是主渠道。
|
||||
</details>
|
||||
|
||||
### 安装 AI agent skills
|
||||
|
||||
```bash
|
||||
@@ -79,6 +107,10 @@ npx skills add leeguooooo/chrome-use
|
||||
|
||||
把 `skills/chrome-use` 拉进当前项目,让你的 AI agent 拿到正确的用法和预授权的 bash 权限。
|
||||
|
||||
## 命令名
|
||||
|
||||
`chrome-use`、`chrome-use`、`abs` 是**同一个二进制** —— `abs` 只是短别名。没有单独的「隐身可执行文件」;隐身是**运行时行为**(见下方 [反检测](#反检测)),根据你是连接真实 Chrome 还是 `--launch` 全新实例自动启用。
|
||||
|
||||
## 连接你的 Chrome
|
||||
|
||||
**推荐 —— 浏览器扩展(一键,无弹窗)。** 从 Chrome 应用商店安装 [**chrome-use** 扩展](https://chromewebstore.google.com/detail/chrome-use/knfcmbamhjmaonkfnjhldjedeobeafmk),再注册一次本地桥:
|
||||
@@ -128,8 +160,83 @@ chrome-use --launch open https://example.com
|
||||
|
||||
# 保留登录:用你真实的 Chrome profile 启动
|
||||
chrome-use --launch --profile auto open https://x.com/home
|
||||
# 或显式指定:--profile Default / --profile "Profile 1"
|
||||
```
|
||||
|
||||
## 站点适配器 —— 把一个网站变成「结构化数据 CLI」
|
||||
|
||||
大多数「读 GitHub issue」「搜 Reddit」「拉我的 B 站动态」这类任务,根本不需要点击 +
|
||||
截图 —— 网站登录态背后本来就有 JSON 接口。**站点适配器**就是一小段 JS 函数,它在你
|
||||
**已登录的标签页内**调用那个接口(用你的 cookie、同源 `fetch`、网站自己的模块),返回
|
||||
干净的 JSON。网站分辨不出它和你的区别,因为它**就是你**。
|
||||
|
||||
chrome-use 本身不附带任何适配器 —— `site update` 会在运行时拉取社区的
|
||||
[**bb-sites**](https://github.com/epiral/bb-sites) 适配器包(就像包管理器拉依赖),
|
||||
然后在 chrome-use 的隐身通道上运行它们:
|
||||
|
||||
```bash
|
||||
chrome-use site update # 拉取适配器包(约 145 条命令)
|
||||
chrome-use site list # github/issues、reddit/search、bilibili/feed…
|
||||
chrome-use site info github/issues # 查看某个适配器的参数 + 域名
|
||||
|
||||
# 运行一个 —— 会导航到对应站点(已在该站点则复用当前标签页)并返回 JSON
|
||||
chrome-use site github/issues epiral/bb-browser --json
|
||||
chrome-use site reddit/search "rust async" --json
|
||||
chrome-use site bilibili/feed --json # 能用,因为走的是你的登录态
|
||||
```
|
||||
|
||||
位置参数按适配器声明的参数顺序填入;`--key value` 按名覆盖。适配器由 bb-sites 社区编写、
|
||||
版权归各自作者所有 —— chrome-use 只负责运行它们。
|
||||
|
||||
**自动同步 + 自动提示。** 你基本不用手动 `site update`:chrome-use 首次使用时自动拉取,
|
||||
之后每周后台刷新一次(`AGENT_BROWSER_SITES_TTL_DAYS` 调周期,`AGENT_BROWSER_SITES_NO_AUTO_UPDATE=1`
|
||||
关闭)。而当你 `open`/`snapshot` 一个有适配器的域名时,chrome-use 会在输出里直接把可用命令
|
||||
亮出来 —— 一行 `💡 site adapters for <域名>`,`--json` 下则是 `siteAdapters` 字段 —— 这样
|
||||
agent 会直接改用结构化适配器,而不是去扒 DOM:
|
||||
|
||||
```text
|
||||
$ chrome-use open https://github.com
|
||||
💡 site adapters for github.com — prefer these for structured data:
|
||||
github/issues, github/me, github/repo, …
|
||||
e.g. chrome-use site github/issues --json
|
||||
✓ GitHub
|
||||
```
|
||||
|
||||
## 自动化测试(`chrome-use test`)
|
||||
|
||||
把反复的「打开它、点一圈、看对不对」变成**可重跑的测试套件** —— 前端的单元测试。用 YAML 写用例;步骤复用 chrome-use 自己的命令,断言编译成一次检查:
|
||||
|
||||
```yaml
|
||||
# smoke.yaml
|
||||
suite: chatgpt smoke
|
||||
setup:
|
||||
- account: chatgpt/huayue # 注入一个 cookie-use 登录(可选)
|
||||
cases:
|
||||
- name: home loads logged in
|
||||
steps:
|
||||
- open: https://chatgpt.com/
|
||||
- wait: { load: networkidle }
|
||||
assert:
|
||||
- url: { contains: chatgpt.com }
|
||||
- visible: "#prompt-textarea"
|
||||
```
|
||||
|
||||
```bash
|
||||
chrome-use test smoke.yaml # 启动隔离浏览器,跑用例
|
||||
chrome-use test smoke.yaml --session default # …或对你已连接的 Chrome 跑
|
||||
```
|
||||
|
||||
```
|
||||
suite: chatgpt smoke (session cu-test)
|
||||
✓ home loads logged in 1.2s
|
||||
✗ composer takes text 0.8s
|
||||
assert text "#prompt-textarea" contains "hi" → got ""
|
||||
↳ cu-test-artifacts/composer-takes-text.png
|
||||
2 cases · 1 passed · 1 failed
|
||||
```
|
||||
|
||||
任一用例失败时退出码非零(可直接丢进 CI),失败用例会存截图。断言:`url` · `visible` · `hidden` · `text` · `count` · `eval`。步骤:`open` · `click` · `fill` · `type` · `press` · `wait` · `scroll` · `eval`。完整指南:`chrome-use skills get test`。发现回归?加个用例 —— 用得越多,套件越值钱。
|
||||
|
||||
## 反检测
|
||||
|
||||
连接你真实 Chrome 时,我们**零** JS 注入 —— 浏览器指纹完全是真的。指导原则是 **native CDP/Chrome 覆盖优先于 JS 谎言**:被重定义的 getter 本身可被检测,原生覆盖则不会。
|
||||
@@ -146,7 +253,9 @@ chrome-use --launch --profile auto open https://x.com/home
|
||||
| [rebrowser-bot-detector](https://bot-detector.rebrowser.net/) | `runtimeEnableLeak` 🟢 · `pwInitScripts` 🟢 |
|
||||
| [bot.sannysoft.com](https://bot.sannysoft.com) | 全绿 |
|
||||
|
||||
`--launch` 独立模式下会改用一整套隐身补丁,同样过上述检测。
|
||||
CreepJS 上的 `0% stealth` 是关键数字:因为连接路径**什么都不打补丁**,根本没有可供说谎检测器抓的 override。(读 `navigator.languages` 顺序或 IP 地理位置的面板可能给个软性的「navigator」/「location」标记 —— 那反映的是*你真实 Chrome* 的语言列表和网络,不是自动化破绽。)
|
||||
|
||||
`--launch` 独立模式(全新浏览器)会改用一整套隐身补丁,也能过上述检测 —— 唯一例外:CreepJS 报 **~20% stealth**,因为 srcdoc-iframe 的 `contentWindow` 补丁触发了它的 `hasIframeProxy` 探测(用来藏自动化的 proxy 本身成了破绽)。其余全干净(`0% headless`、sannysoft/browserscan 全绿、Cloudflare 通过)。设 **`AGENT_BROWSER_DISABLE_IFRAME_PROXY=1`** 去掉那个补丁即可拿到干净的 **0% stealth**(代价是放弃小众的 srcdoc-iframe 遮蔽)。**扩展连接路径**(你的真实 Chrome)零 JS 注入、不受影响 —— 它才是货真价实的 0% 路径。
|
||||
|
||||
### 类人输入(行为隐身)
|
||||
|
||||
@@ -167,6 +276,30 @@ chrome-use --launch --profile auto open https://x.com/home
|
||||
|
||||
操作你的真实 Chrome 不该打断你的工作。agent **全程在后台操作**:新标签后台打开(在自己的彩色会话标签组里),**从不强制把标签拽到前台**,并用 `Emulation.setFocusEmulationEnabled` 让每个 agent 标签照常渲染、`document.hasFocus()` / `visibilityState` 仍报 `visible`。于是截图正常、页面不被降频,"标签全程隐藏"也不会变成新的机器人信号。你在自己的标签里照常工作,agent 在旁边默默干活。(想置顶某个标签仍可显式调用命令。)
|
||||
|
||||
### 自己验证
|
||||
|
||||
别光听我们说 —— 把你连接的 Chrome 指向最硬的公开检测器,自己对比:
|
||||
|
||||
- **[CreepJS](https://abrahamjuliot.github.io/creepjs/)** —— 最全面的指纹 / 说谎检测器
|
||||
- **[bot.incolumitas.com](https://bot.incolumitas.com/)** —— 行为 + 指纹打分,方法公开
|
||||
- **[BrowserScan](https://www.browserscan.net/bot-detection)** —— Webdriver / User-Agent / CDP / Navigator
|
||||
- **[bot.sannysoft.com](https://bot.sannysoft.com)** —— 经典自动化特征清单
|
||||
- **[pixelscan.net](https://pixelscan.net/)** · **[iphey.com](https://iphey.com/)** —— 一致性与身份
|
||||
|
||||
我们故意**不自带 bot 检测器** —— 最强、最诚实的基准,就是拿市面上最好的检测器去测你的真实浏览器。
|
||||
|
||||
### 调参(环境变量)
|
||||
|
||||
| 变量 | 默认 | 作用 |
|
||||
|---|---|---|
|
||||
| `AGENT_BROWSER_CAPTURE_CONSOLE` | 关 | 启用 `Runtime` 域,让 `console` / `errors` 捕获页面输出。关闭可保持最隐身的画像。 |
|
||||
| `AGENT_BROWSER_HUMANIZE` | 关 | 类人输入动作:`off`(瞬时)、`fast`(轻量缓动轨迹)、`human`(全套曲线轨迹 + 落点抖动 + 击键节奏 + 缓动滚动/拖拽)。也可用 `--humanize`。默认 `off`;自适应检测器会把 Akamai/PerimeterX/DataDome 守护的页面自动升到 `human`。 |
|
||||
| `AGENT_BROWSER_TIMEZONE` | 未设 | 仅 `--launch`。IANA id(如 `Asia/Tokyo`)原生设置时区(Intl + Date 跟随,无 JS 谎言)以匹配代理;`auto` 按 locale 推导。 |
|
||||
| `AGENT_BROWSER_BLOCK_WEBRTC` | auto | 仅 `--launch`。设了代理时自动强制 WebRTC 走代理(不泄漏真实 IP)。`1` 无代理时也隐藏本地 IP;`0` 退出。 |
|
||||
| `AGENT_BROWSER_HIDE_CANVAS` | 关 | 仅 `--launch`。加入会话稳定的 canvas/audio 指纹噪声。默认关(噪声本身就是一种「谎言」)。 |
|
||||
| `AGENT_BROWSER_ADAPTIVE_REF` | 开 | 当保存的 `@ref` 移动且 role/name 重查失败时,按指纹相似度重定位(需高分 + 明显领先,否则明确报错)。`0` 关闭。 |
|
||||
| `AGENT_BROWSER_CLICK_MODE` | _(auto)_ | 点击策略。默认先滚动入视、派发坐标点击,若被浮层遮挡则回退 DOM `.click()`。`dom` 始终用 `.click()`(适合 blur 即关的自动补全/菜单项);`coord` 严格只用坐标(遮挡时硬失败)。 |
|
||||
|
||||
## chrome-use 的独特之处
|
||||
|
||||
- **默认 auto-connect** —— `chrome-use open` 连你现有的 Chrome 而非启新的
|
||||
@@ -181,3 +314,7 @@ chrome-use --launch --profile auto open https://x.com/home
|
||||
## License
|
||||
|
||||
Apache-2.0
|
||||
|
||||
---
|
||||
|
||||
> 由 **leeguooooo** 打造 —— AI agent、逆向工程与 Cloudflare Workers 的实战笔记见 **[blog.misonote.com](https://blog.misonote.com)** · 关注 **[X @leeguooooo](https://x.com/leeguooooo)**
|
||||
|
||||
Generated
+1
-1
@@ -290,7 +290,7 @@ checksum = "613afe47fcd5fac7ccf1db93babcb082c5994d996f20b8b159f2ad1658eb5724"
|
||||
|
||||
[[package]]
|
||||
name = "chrome-use"
|
||||
version = "1.5.10"
|
||||
version = "1.5.27"
|
||||
dependencies = [
|
||||
"aes",
|
||||
"aes-gcm",
|
||||
|
||||
+1
-1
@@ -1,6 +1,6 @@
|
||||
[package]
|
||||
name = "chrome-use"
|
||||
version = "1.5.10"
|
||||
version = "1.5.27"
|
||||
edition = "2021"
|
||||
description = "Fast browser automation CLI for AI agents"
|
||||
license = "Apache-2.0"
|
||||
|
||||
+520
-52
@@ -34,11 +34,57 @@ pub enum ParseError {
|
||||
/// suggestions on an unknown command (issue #29). Not exhaustive — just the
|
||||
/// common verbs plus a few known wrong-guesses mapped to the real command.
|
||||
const KNOWN_COMMANDS: &[&str] = &[
|
||||
"open", "navigate", "click", "fill", "type", "press", "snapshot", "screenshot", "eval", "get",
|
||||
"text", "html", "frames", "find", "wait", "scroll", "hover", "select", "check", "uncheck",
|
||||
"tab", "tabs", "close", "back", "forward", "reload", "sessions", "status", "daemon", "doctor",
|
||||
"upgrade", "connect", "cookies", "mouse", "keyboard", "stream", "frame", "profiles", "title",
|
||||
"url", "is", "drag", "dialog", "upload",
|
||||
"open",
|
||||
"navigate",
|
||||
"click",
|
||||
"fill",
|
||||
"type",
|
||||
"press",
|
||||
"snapshot",
|
||||
"screenshot",
|
||||
"eval",
|
||||
"get",
|
||||
"text",
|
||||
"html",
|
||||
"frames",
|
||||
"find",
|
||||
"wait",
|
||||
"scroll",
|
||||
"hover",
|
||||
"select",
|
||||
"check",
|
||||
"uncheck",
|
||||
"tab",
|
||||
"tabs",
|
||||
"close",
|
||||
"back",
|
||||
"forward",
|
||||
"reload",
|
||||
"sessions",
|
||||
"status",
|
||||
"daemon",
|
||||
"doctor",
|
||||
"upgrade",
|
||||
"connect",
|
||||
"cookies",
|
||||
"mouse",
|
||||
"keyboard",
|
||||
"stream",
|
||||
"frame",
|
||||
"profiles",
|
||||
"title",
|
||||
"url",
|
||||
"is",
|
||||
"drag",
|
||||
"dialog",
|
||||
"upload",
|
||||
"site",
|
||||
"box",
|
||||
"adopt",
|
||||
"canvas",
|
||||
"viewport",
|
||||
"resize",
|
||||
"keep",
|
||||
];
|
||||
|
||||
/// Levenshtein distance, capped — small inputs only (command names).
|
||||
@@ -520,25 +566,64 @@ fn parse_command_inner(args: &[String], flags: &Flags) -> Result<Value, ParseErr
|
||||
"fill" => {
|
||||
let sel = rest.first().ok_or_else(|| ParseError::MissingArguments {
|
||||
context: "fill".to_string(),
|
||||
usage: "fill <selector> <text>",
|
||||
usage: "fill <selector> <text> | fill <selector> --file <path> | fill <selector> --stdin",
|
||||
})?;
|
||||
Ok(json!({ "id": id, "action": "fill", "selector": sel, "value": rest[1..].join(" ") }))
|
||||
// Large/multiline content without shell-escaping hell (issue #41):
|
||||
// `fill <sel> --file <path>` reads the value from a UTF-8 file, and
|
||||
// `fill <sel> --stdin` reads it from stdin — sent verbatim, so backticks,
|
||||
// quotes, newlines and non-ASCII pass through untouched.
|
||||
let value = match rest.get(1).copied() {
|
||||
Some("--file") => {
|
||||
let path = rest.get(2).ok_or(ParseError::InvalidValue {
|
||||
message: "fill --file requires a path".to_string(),
|
||||
usage: "fill <selector> --file <path>",
|
||||
})?;
|
||||
std::fs::read_to_string(path).map_err(|e| ParseError::InvalidValue {
|
||||
message: format!("fill --file: cannot read {path}: {e}"),
|
||||
usage: "fill <selector> --file <path>",
|
||||
})?
|
||||
}
|
||||
Some("--stdin") => {
|
||||
use std::io::Read;
|
||||
let mut buf = String::new();
|
||||
io::stdin()
|
||||
.read_to_string(&mut buf)
|
||||
.map_err(|e| ParseError::InvalidValue {
|
||||
message: format!("fill --stdin: {e}"),
|
||||
usage: "fill <selector> --stdin",
|
||||
})?;
|
||||
buf
|
||||
}
|
||||
_ => rest[1..].join(" "),
|
||||
};
|
||||
Ok(json!({ "id": id, "action": "fill", "selector": sel, "value": value }))
|
||||
}
|
||||
"type" => {
|
||||
// `--key-events` (alias `--keys`): send real per-character keystrokes
|
||||
// instead of Input.insertText, so autocomplete/combobox widgets that
|
||||
// only react to key events fire (e.g. Google address postal lookup).
|
||||
let key_events = rest.iter().any(|a| *a == "--key-events" || *a == "--keys");
|
||||
let rest: Vec<&str> = rest
|
||||
.iter()
|
||||
.copied()
|
||||
.filter(|a| *a != "--key-events" && *a != "--keys")
|
||||
.collect();
|
||||
// `type --focused <text>` types into whatever element currently has
|
||||
// focus (no selector) — for custom widgets that move focus to a hidden
|
||||
// input after you open them.
|
||||
if rest.first() == Some(&"--focused") {
|
||||
return Ok(json!({
|
||||
"id": id, "action": "type", "focused": true,
|
||||
"text": rest[1..].join(" "),
|
||||
"text": rest[1..].join(" "), "keyEvents": key_events,
|
||||
}));
|
||||
}
|
||||
let sel = rest.first().ok_or_else(|| ParseError::MissingArguments {
|
||||
context: "type".to_string(),
|
||||
usage: "type <selector> <text> (or: type --focused <text>)",
|
||||
usage: "type <selector> <text> (or: type --focused <text>) [--key-events]",
|
||||
})?;
|
||||
Ok(json!({ "id": id, "action": "type", "selector": sel, "text": rest[1..].join(" ") }))
|
||||
Ok(
|
||||
json!({ "id": id, "action": "type", "selector": sel, "text": rest[1..].join(" "), "keyEvents": key_events }),
|
||||
)
|
||||
}
|
||||
"pick" => {
|
||||
// pick <selector|@ref> --option "<text>" — atomic combobox select:
|
||||
@@ -729,10 +814,57 @@ fn parse_command_inner(args: &[String], flags: &Flags) -> Result<Value, ParseErr
|
||||
} else {
|
||||
return Err(ParseError::MissingArguments {
|
||||
context: "scroll --selector".to_string(),
|
||||
usage: "scroll [direction] [amount] [--selector <sel>]",
|
||||
usage: "scroll [direction] [amount] [--selector <sel>] [--at <x,y>] [--frame <n>]",
|
||||
});
|
||||
}
|
||||
}
|
||||
"--at" => {
|
||||
// `--at x,y`: dispatch the wheel at this viewport pixel, so it
|
||||
// scrolls whatever element/iframe is under the pointer — including
|
||||
// cross-origin iframes that `window.scrollBy` can't reach (#36).
|
||||
let val = rest.get(i + 1).ok_or(ParseError::MissingArguments {
|
||||
context: "scroll --at".to_string(),
|
||||
usage: "scroll [direction] [amount] --at <x,y>",
|
||||
})?;
|
||||
let mut parts = val.split(',');
|
||||
match (
|
||||
parts.next().and_then(|s| s.trim().parse::<f64>().ok()),
|
||||
parts.next().and_then(|s| s.trim().parse::<f64>().ok()),
|
||||
) {
|
||||
(Some(x), Some(y)) => {
|
||||
obj.insert("at".to_string(), json!([x, y]));
|
||||
}
|
||||
_ => {
|
||||
return Err(ParseError::InvalidValue {
|
||||
message: format!("scroll --at: invalid coordinate `{}`", val),
|
||||
usage:
|
||||
"scroll [direction] [amount] --at <x,y> (e.g. --at 640,400)",
|
||||
})
|
||||
}
|
||||
}
|
||||
i += 1;
|
||||
}
|
||||
"--frame" => {
|
||||
// `--frame n`: scroll the n-th frame from `chrome-use frames` by
|
||||
// dispatching the wheel at that frame's center — reaches content in
|
||||
// a cross-origin iframe without needing a selector into it (#36).
|
||||
let val = rest.get(i + 1).ok_or(ParseError::MissingArguments {
|
||||
context: "scroll --frame".to_string(),
|
||||
usage: "scroll [direction] [amount] --frame <n>",
|
||||
})?;
|
||||
match val.trim().parse::<usize>() {
|
||||
Ok(n) => {
|
||||
obj.insert("frame".to_string(), json!(n));
|
||||
}
|
||||
Err(_) => {
|
||||
return Err(ParseError::InvalidValue {
|
||||
message: format!("scroll --frame: invalid index `{}`", val),
|
||||
usage: "scroll [direction] [amount] --frame <n> (index from `chrome-use frames`)",
|
||||
})
|
||||
}
|
||||
}
|
||||
i += 1;
|
||||
}
|
||||
arg if arg.starts_with('-') => {}
|
||||
_ => {
|
||||
match positional_index {
|
||||
@@ -908,24 +1040,72 @@ fn parse_command_inner(args: &[String], flags: &Flags) -> Result<Value, ParseErr
|
||||
// path: file path (contains / or . or ends with known extension)
|
||||
let mut full_page = false;
|
||||
let mut clip: Option<Value> = None;
|
||||
let mut max_width: Option<u32> = None;
|
||||
let mut max_height: Option<u32> = None;
|
||||
let mut scale: Option<f64> = None;
|
||||
let mut positional: Vec<&str> = Vec::new();
|
||||
let mut i = 0;
|
||||
// Parse a numeric value for a downscale flag (issue #42).
|
||||
let parse_num = |i: &mut usize, flag: &str| -> Result<String, ParseError> {
|
||||
let v = rest
|
||||
.get(*i + 1)
|
||||
.ok_or_else(|| ParseError::MissingArguments {
|
||||
context: format!("screenshot {flag}"),
|
||||
usage: "screenshot [--max-width <px>] [--max-height <px>] [--scale <0..1>]",
|
||||
})?;
|
||||
*i += 1;
|
||||
Ok(v.to_string())
|
||||
};
|
||||
while i < rest.len() {
|
||||
match rest[i] {
|
||||
"--full" | "-f" => full_page = true,
|
||||
// Downscale the saved image so retina/full-page shots fit an
|
||||
// agent's image reader and screenshot px line up with click px (#42).
|
||||
"--max-width" => {
|
||||
let v = parse_num(&mut i, "--max-width")?;
|
||||
max_width = Some(v.parse().map_err(|_| ParseError::InvalidValue {
|
||||
message: format!("--max-width expects a number, got '{v}'"),
|
||||
usage: "screenshot --max-width <px>",
|
||||
})?);
|
||||
}
|
||||
"--max-height" => {
|
||||
let v = parse_num(&mut i, "--max-height")?;
|
||||
max_height = Some(v.parse().map_err(|_| ParseError::InvalidValue {
|
||||
message: format!("--max-height expects a number, got '{v}'"),
|
||||
usage: "screenshot --max-height <px>",
|
||||
})?);
|
||||
}
|
||||
"--scale" => {
|
||||
let v = parse_num(&mut i, "--scale")?;
|
||||
let s: f64 = v.parse().map_err(|_| ParseError::InvalidValue {
|
||||
message: format!("--scale expects a number like 0.5, got '{v}'"),
|
||||
usage: "screenshot --scale <0..1>",
|
||||
})?;
|
||||
if s <= 0.0 || s > 1.0 {
|
||||
return Err(ParseError::InvalidValue {
|
||||
message: format!("--scale must be in (0, 1], got '{v}'"),
|
||||
usage: "screenshot --scale <0..1>",
|
||||
});
|
||||
}
|
||||
scale = Some(s);
|
||||
}
|
||||
// `--clip x,y,w,h` captures a pixel region (issue #34).
|
||||
"--clip" => {
|
||||
let raw = rest.get(i + 1).ok_or_else(|| ParseError::MissingArguments {
|
||||
context: "screenshot --clip".to_string(),
|
||||
usage: "screenshot --clip <x,y,w,h> [path]",
|
||||
})?;
|
||||
let raw = rest
|
||||
.get(i + 1)
|
||||
.ok_or_else(|| ParseError::MissingArguments {
|
||||
context: "screenshot --clip".to_string(),
|
||||
usage: "screenshot --clip <x,y,w,h> [path]",
|
||||
})?;
|
||||
let nums: Vec<f64> = raw
|
||||
.split(',')
|
||||
.filter_map(|n| n.trim().parse::<f64>().ok())
|
||||
.collect();
|
||||
if nums.len() != 4 {
|
||||
return Err(ParseError::InvalidValue {
|
||||
message: format!("--clip expects 'x,y,w,h' (4 numbers), got '{raw}'"),
|
||||
message: format!(
|
||||
"--clip expects 'x,y,w,h' (4 numbers), got '{raw}'"
|
||||
),
|
||||
usage: "screenshot --clip <x,y,w,h> [path]",
|
||||
});
|
||||
}
|
||||
@@ -971,6 +1151,15 @@ fn parse_command_inner(args: &[String], flags: &Flags) -> Result<Value, ParseErr
|
||||
if let Some(c) = clip {
|
||||
cmd["clip"] = c;
|
||||
}
|
||||
if let Some(w) = max_width {
|
||||
cmd["maxWidth"] = json!(w);
|
||||
}
|
||||
if let Some(h) = max_height {
|
||||
cmd["maxHeight"] = json!(h);
|
||||
}
|
||||
if let Some(s) = scale {
|
||||
cmd["scale"] = json!(s);
|
||||
}
|
||||
if let Some(ref fmt) = flags.screenshot_format {
|
||||
cmd["format"] = json!(fmt);
|
||||
}
|
||||
@@ -988,6 +1177,54 @@ fn parse_command_inner(args: &[String], flags: &Flags) -> Result<Value, ParseErr
|
||||
}
|
||||
Ok(cmd)
|
||||
}
|
||||
// Canvas/WebGL apps (Figma, games, maps, charts, drawing tools) render to a
|
||||
// <canvas> with no DOM/refs to read. `canvas` extracts what's actually
|
||||
// rendered: `list` enumerates canvases; `capture` saves a canvas to PNG via
|
||||
// toDataURL (full backing-store resolution) with a CDP-screenshot fallback
|
||||
// for WebGL contexts (no preserveDrawingBuffer) or cross-origin-tainted ones.
|
||||
"canvas" => {
|
||||
match rest.first().copied() {
|
||||
Some("list") => Ok(json!({ "id": id, "action": "canvas_list" })),
|
||||
Some("capture") => {
|
||||
let force_screenshot = rest.contains(&"--screenshot");
|
||||
let pos: Vec<&str> = rest[1..]
|
||||
.iter()
|
||||
.copied()
|
||||
.filter(|a| !a.starts_with("--"))
|
||||
.collect();
|
||||
// `capture [selector] [path]`: a selector starts with . # @ or is
|
||||
// a tag; a path contains / or ends in an image extension.
|
||||
let is_path = |s: &str| {
|
||||
s.contains('/')
|
||||
|| s.ends_with(".png")
|
||||
|| s.ends_with(".jpg")
|
||||
|| s.ends_with(".jpeg")
|
||||
|| s.ends_with(".webp")
|
||||
};
|
||||
let (selector, path) = match (pos.first(), pos.get(1)) {
|
||||
(Some(a), Some(b)) => (Some(*a), Some(*b)),
|
||||
(Some(a), None) if is_path(a) => (None, Some(*a)),
|
||||
(Some(a), None) => (Some(*a), None),
|
||||
_ => (None, None),
|
||||
};
|
||||
let mut cmd = json!({ "id": id, "action": "canvas_capture" });
|
||||
if let Some(s) = selector {
|
||||
cmd["selector"] = json!(s);
|
||||
}
|
||||
if let Some(p) = path {
|
||||
cmd["path"] = json!(p);
|
||||
}
|
||||
if force_screenshot {
|
||||
cmd["forceScreenshot"] = json!(true);
|
||||
}
|
||||
Ok(cmd)
|
||||
}
|
||||
_ => Err(ParseError::InvalidValue {
|
||||
message: "canvas needs a subcommand".to_string(),
|
||||
usage: "canvas list | canvas capture [selector] [path] [--screenshot]",
|
||||
}),
|
||||
}
|
||||
}
|
||||
"pdf" => {
|
||||
let path = rest.first().ok_or_else(|| ParseError::MissingArguments {
|
||||
context: "pdf".to_string(),
|
||||
@@ -1092,6 +1329,57 @@ fn parse_command_inner(args: &[String], flags: &Flags) -> Result<Value, ParseErr
|
||||
Ok(json!({ "id": id, "action": "evaluate", "script": script }))
|
||||
}
|
||||
|
||||
"site" => {
|
||||
// `site <name>/<command> [positional...] [--key value]`. The
|
||||
// `update`/`list`/`info` subcommands are handled CLI-side (main.rs)
|
||||
// and never reach here — by this point `rest[0]` is a `name/cmd`
|
||||
// adapter spec. Load it, map the args onto the adapter's declared
|
||||
// `args`, and emit a `site` action: the daemon navigates to the
|
||||
// adapter's @meta.domain (reusing the tab if already there) and evals
|
||||
// the adapter function in the site's own logged-in page.
|
||||
let spec = rest.first().ok_or(ParseError::InvalidValue {
|
||||
message: "site requires <name>/<command> (run `chrome-use site list`)".to_string(),
|
||||
usage: "site <name>/<command> [args]",
|
||||
})?;
|
||||
let adapter =
|
||||
crate::site::load_adapter(spec).map_err(|e| ParseError::InvalidValue {
|
||||
message: e,
|
||||
usage: "site <name>/<command> [args]",
|
||||
})?;
|
||||
let domain = adapter
|
||||
.domain()
|
||||
.ok_or(ParseError::InvalidValue {
|
||||
message: format!("site: adapter `{spec}` @meta is missing a \"domain\""),
|
||||
usage: "site <name>/<command>",
|
||||
})?
|
||||
.to_string();
|
||||
// Split remaining args: `--key value` → named, everything else → positional.
|
||||
let mut positional: Vec<String> = Vec::new();
|
||||
let mut named: Vec<(String, String)> = Vec::new();
|
||||
let mut it = rest[1..].iter();
|
||||
while let Some(a) = it.next() {
|
||||
if let Some(key) = a.strip_prefix("--") {
|
||||
let val = it.next().map(|s| s.to_string()).unwrap_or_default();
|
||||
named.push((key.to_string(), val));
|
||||
} else {
|
||||
positional.push(a.to_string());
|
||||
}
|
||||
}
|
||||
let mapped = crate::site::map_args(&adapter, &positional, &named);
|
||||
let script = crate::site::build_eval(&adapter, &mapped);
|
||||
Ok(json!({ "id": id, "action": "site", "domain": domain, "script": script }))
|
||||
}
|
||||
|
||||
// `adopt <url|targetId>`: the adoption happens at daemon connect (driven by
|
||||
// the AGENT_BROWSER_ADOPT env main.rs set + a forced-fresh daemon), so by
|
||||
// the time this command runs the tab is already attached. Resolve to a
|
||||
// `url` read so the response confirms which tab got adopted.
|
||||
"adopt" => Ok(json!({ "id": id, "action": "url" })),
|
||||
|
||||
// `keep`: leave the active tab for the user — exempt it from the daemon's
|
||||
// auto-close/idle cleanup and remove it from the session's tab group.
|
||||
"keep" => Ok(json!({ "id": id, "action": "keep" })),
|
||||
|
||||
// === Stealth self-check ===
|
||||
"stealth" => {
|
||||
// `stealth [status]` — local stealth self-check: mode, live probes
|
||||
@@ -1415,6 +1703,13 @@ fn parse_command_inner(args: &[String], flags: &Flags) -> Result<Value, ParseErr
|
||||
// === Mouse ===
|
||||
"mouse" => parse_mouse(&rest, &id),
|
||||
|
||||
// === Viewport / window size ===
|
||||
// Top-level shortcut for `set viewport` — agents (and the author, in
|
||||
// issue #47) reach for `viewport`/`resize` first. Uses a CDP virtual
|
||||
// viewport, so it also works on the extension relay without yanking the
|
||||
// user's real window around.
|
||||
"viewport" | "resize" => parse_viewport(&rest, &id),
|
||||
|
||||
// === Set (browser settings) ===
|
||||
"set" => parse_set(&rest, &id),
|
||||
|
||||
@@ -2863,6 +3158,97 @@ fn parse_mouse(rest: &[&str], id: &str) -> Result<Value, ParseError> {
|
||||
}
|
||||
}
|
||||
|
||||
/// Parse a viewport / resize spec, shared by the top-level `viewport` / `resize`
|
||||
/// commands and `set viewport`. Sets a CDP device-metrics override
|
||||
/// (`Emulation.setDeviceMetricsOverride`) — a *virtual* viewport for the tab, so
|
||||
/// it works headless AND on the extension relay without physically resizing the
|
||||
/// user's real Chrome window (issue #47). Forms:
|
||||
/// viewport <width> <height> [scale] [--dpr N] [--mobile]
|
||||
/// viewport <width>x<height> (e.g. 1280x800)
|
||||
/// viewport reset | clear (drop the override, restore real size)
|
||||
fn parse_viewport(rest: &[&str], id: &str) -> Result<Value, ParseError> {
|
||||
const USAGE: &str = "viewport <width> <height> [scale] [--dpr N] [--mobile] | viewport reset";
|
||||
|
||||
if matches!(
|
||||
rest.first().copied(),
|
||||
Some("reset") | Some("clear") | Some("off")
|
||||
) {
|
||||
return Ok(json!({ "id": id, "action": "viewport", "reset": true }));
|
||||
}
|
||||
|
||||
// Positional (non-flag) tokens. A `WxH` token counts as one positional.
|
||||
let positionals: Vec<&str> = rest
|
||||
.iter()
|
||||
.copied()
|
||||
.filter(|a| !a.starts_with("--"))
|
||||
.collect();
|
||||
|
||||
let (w, h, scale_tok): (i32, i32, Option<&str>) = match positionals.first() {
|
||||
Some(first) if first.contains('x') || first.contains('X') => {
|
||||
let mut parts = first.split(['x', 'X']);
|
||||
let w = parts.next().and_then(|s| s.parse::<i32>().ok());
|
||||
let h = parts.next().and_then(|s| s.parse::<i32>().ok());
|
||||
match (w, h) {
|
||||
(Some(w), Some(h)) => (w, h, positionals.get(1).copied()),
|
||||
_ => {
|
||||
return Err(ParseError::InvalidValue {
|
||||
message: format!("Invalid viewport size: {}", first),
|
||||
usage: USAGE,
|
||||
})
|
||||
}
|
||||
}
|
||||
}
|
||||
Some(w_str) => {
|
||||
let h_str = positionals.get(1).ok_or(ParseError::MissingArguments {
|
||||
context: "viewport".to_string(),
|
||||
usage: USAGE,
|
||||
})?;
|
||||
let w = w_str.parse::<i32>().map_err(|_| ParseError::InvalidValue {
|
||||
message: format!("Invalid width: {}", w_str),
|
||||
usage: USAGE,
|
||||
})?;
|
||||
let h = h_str.parse::<i32>().map_err(|_| ParseError::InvalidValue {
|
||||
message: format!("Invalid height: {}", h_str),
|
||||
usage: USAGE,
|
||||
})?;
|
||||
(w, h, positionals.get(2).copied())
|
||||
}
|
||||
None => {
|
||||
return Err(ParseError::MissingArguments {
|
||||
context: "viewport".to_string(),
|
||||
usage: USAGE,
|
||||
})
|
||||
}
|
||||
};
|
||||
|
||||
let mut cmd = json!({ "id": id, "action": "viewport", "width": w, "height": h });
|
||||
|
||||
// Device-scale-factor: positional, overridden by --dpr / --scale.
|
||||
let mut scale: Option<f64> = match scale_tok {
|
||||
Some(s) => Some(s.parse::<f64>().map_err(|_| ParseError::InvalidValue {
|
||||
message: format!("Invalid scale: {}", s),
|
||||
usage: USAGE,
|
||||
})?),
|
||||
None => None,
|
||||
};
|
||||
if let Some(i) = rest.iter().position(|a| *a == "--dpr" || *a == "--scale") {
|
||||
let v = rest.get(i + 1).and_then(|s| s.parse::<f64>().ok()).ok_or(
|
||||
ParseError::InvalidValue {
|
||||
message: "--dpr/--scale needs a number".to_string(),
|
||||
usage: USAGE,
|
||||
},
|
||||
)?;
|
||||
scale = Some(v);
|
||||
}
|
||||
if let Some(s) = scale {
|
||||
cmd["deviceScaleFactor"] = json!(s);
|
||||
}
|
||||
if rest.contains(&"--mobile") {
|
||||
cmd["mobile"] = json!(true);
|
||||
}
|
||||
Ok(cmd)
|
||||
}
|
||||
|
||||
fn parse_set(rest: &[&str], id: &str) -> Result<Value, ParseError> {
|
||||
const VALID: &[&str] = &[
|
||||
"viewport",
|
||||
@@ -2877,39 +3263,8 @@ fn parse_set(rest: &[&str], id: &str) -> Result<Value, ParseError> {
|
||||
];
|
||||
|
||||
match rest.first().copied() {
|
||||
Some("viewport") => {
|
||||
let w_str = rest.get(1).ok_or_else(|| ParseError::MissingArguments {
|
||||
context: "set viewport".to_string(),
|
||||
usage: "set viewport <width> <height> [scale]",
|
||||
})?;
|
||||
let h_str = rest.get(2).ok_or_else(|| ParseError::MissingArguments {
|
||||
context: "set viewport".to_string(),
|
||||
usage: "set viewport <width> <height> [scale]",
|
||||
})?;
|
||||
let w = w_str
|
||||
.parse::<i32>()
|
||||
.map_err(|_| ParseError::MissingArguments {
|
||||
context: "set viewport".to_string(),
|
||||
usage: "set viewport <width> <height> [scale]",
|
||||
})?;
|
||||
let h = h_str
|
||||
.parse::<i32>()
|
||||
.map_err(|_| ParseError::MissingArguments {
|
||||
context: "set viewport".to_string(),
|
||||
usage: "set viewport <width> <height> [scale]",
|
||||
})?;
|
||||
let mut cmd = json!({ "id": id, "action": "viewport", "width": w, "height": h });
|
||||
if let Some(scale_str) = rest.get(3) {
|
||||
let scale = scale_str
|
||||
.parse::<f64>()
|
||||
.map_err(|_| ParseError::MissingArguments {
|
||||
context: "set viewport".to_string(),
|
||||
usage: "set viewport <width> <height> [scale]",
|
||||
})?;
|
||||
cmd["deviceScaleFactor"] = json!(scale);
|
||||
}
|
||||
Ok(cmd)
|
||||
}
|
||||
// `set viewport ...` is an alias for the top-level `viewport` command.
|
||||
Some("viewport") => parse_viewport(&rest[1..], id),
|
||||
Some("device") => {
|
||||
let dev = rest.get(1).ok_or_else(|| ParseError::MissingArguments {
|
||||
context: "set device".to_string(),
|
||||
@@ -4066,6 +4421,28 @@ mod tests {
|
||||
assert_eq!(cmd["action"], "type");
|
||||
assert_eq!(cmd["selector"], "#input");
|
||||
assert_eq!(cmd["text"], "some text");
|
||||
assert_eq!(cmd["keyEvents"], false);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_type_key_events() {
|
||||
// --key-events sends real keystrokes (for autocomplete/combobox) and must
|
||||
// not be swallowed into the typed text.
|
||||
let cmd = parse_command(
|
||||
&args("type #postal 201-0001 --key-events"),
|
||||
&default_flags(),
|
||||
)
|
||||
.unwrap();
|
||||
assert_eq!(cmd["action"], "type");
|
||||
assert_eq!(cmd["selector"], "#postal");
|
||||
assert_eq!(cmd["text"], "201-0001");
|
||||
assert_eq!(cmd["keyEvents"], true);
|
||||
|
||||
let focused =
|
||||
parse_command(&args("type --focused 201-0001 --keys"), &default_flags()).unwrap();
|
||||
assert_eq!(focused["focused"], true);
|
||||
assert_eq!(focused["text"], "201-0001");
|
||||
assert_eq!(focused["keyEvents"], true);
|
||||
}
|
||||
|
||||
#[test]
|
||||
@@ -4353,8 +4730,11 @@ mod tests {
|
||||
#[test]
|
||||
fn test_screenshot_clip() {
|
||||
// `--clip x,y,w,h` captures a pixel region (issue #34); the path still parses.
|
||||
let cmd = parse_command(&args("screenshot --clip 10,20,200,40 out.png"), &default_flags())
|
||||
.unwrap();
|
||||
let cmd = parse_command(
|
||||
&args("screenshot --clip 10,20,200,40 out.png"),
|
||||
&default_flags(),
|
||||
)
|
||||
.unwrap();
|
||||
assert_eq!(cmd["action"], "screenshot");
|
||||
assert_eq!(cmd["clip"]["x"], 10.0);
|
||||
assert_eq!(cmd["clip"]["y"], 20.0);
|
||||
@@ -4932,7 +5312,10 @@ mod tests {
|
||||
assert_eq!(nearest_command("sesions").as_deref(), Some("sessions"));
|
||||
assert_eq!(nearest_command("session").as_deref(), Some("sessions"));
|
||||
assert_eq!(nearest_command("clik").as_deref(), Some("click"));
|
||||
assert_eq!(nearest_command("screenshits").as_deref(), Some("screenshot"));
|
||||
assert_eq!(
|
||||
nearest_command("screenshits").as_deref(),
|
||||
Some("screenshot")
|
||||
);
|
||||
// Nonsense with no close match stays silent.
|
||||
assert_eq!(nearest_command("xyzzy"), None);
|
||||
// The unknown-command error embeds the suggestion.
|
||||
@@ -5042,6 +5425,62 @@ mod tests {
|
||||
assert!(result.is_err());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_viewport_toplevel() {
|
||||
let cmd = parse_command(&args("viewport 1280 800"), &default_flags()).unwrap();
|
||||
assert_eq!(cmd["action"], "viewport");
|
||||
assert_eq!(cmd["width"], 1280);
|
||||
assert_eq!(cmd["height"], 800);
|
||||
assert!(cmd.get("deviceScaleFactor").is_none());
|
||||
assert!(cmd.get("mobile").is_none());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_resize_alias() {
|
||||
let cmd = parse_command(&args("resize 700 800"), &default_flags()).unwrap();
|
||||
assert_eq!(cmd["action"], "viewport");
|
||||
assert_eq!(cmd["width"], 700);
|
||||
assert_eq!(cmd["height"], 800);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_viewport_wxh_form() {
|
||||
let cmd = parse_command(&args("viewport 375x812"), &default_flags()).unwrap();
|
||||
assert_eq!(cmd["action"], "viewport");
|
||||
assert_eq!(cmd["width"], 375);
|
||||
assert_eq!(cmd["height"], 812);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_viewport_dpr_and_mobile_flags() {
|
||||
let cmd =
|
||||
parse_command(&args("viewport 375 812 --dpr 3 --mobile"), &default_flags()).unwrap();
|
||||
assert_eq!(cmd["action"], "viewport");
|
||||
assert_eq!(cmd["width"], 375);
|
||||
assert_eq!(cmd["height"], 812);
|
||||
assert_eq!(cmd["deviceScaleFactor"], 3.0);
|
||||
assert_eq!(cmd["mobile"], true);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_viewport_reset() {
|
||||
for spec in ["viewport reset", "viewport clear", "resize reset"] {
|
||||
let cmd = parse_command(&args(spec), &default_flags()).unwrap();
|
||||
assert_eq!(cmd["action"], "viewport", "{spec}");
|
||||
assert_eq!(cmd["reset"], true, "{spec}");
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_viewport_missing_height() {
|
||||
assert!(parse_command(&args("viewport 1280"), &default_flags()).is_err());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_viewport_invalid_width() {
|
||||
assert!(parse_command(&args("viewport abc 800"), &default_flags()).is_err());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_find_first_no_value() {
|
||||
let cmd = parse_command(&args("find first a click"), &default_flags()).unwrap();
|
||||
@@ -5868,6 +6307,35 @@ mod tests {
|
||||
assert_eq!(cmd["selector"], ".sidebar");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_scroll_at_coordinate() {
|
||||
// `--at x,y` carries a [x, y] array for a wheel dispatched at that pixel
|
||||
// (issue #36: cross-origin iframe scroll).
|
||||
let cmd = parse_command(&args("scroll down 700 --at 640,400"), &default_flags()).unwrap();
|
||||
assert_eq!(cmd["action"], "scroll");
|
||||
assert_eq!(cmd["direction"], "down");
|
||||
assert_eq!(cmd["amount"], 700);
|
||||
assert_eq!(cmd["at"], json!([640.0, 400.0]));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_scroll_at_rejects_garbage() {
|
||||
assert!(parse_command(&args("scroll --at nope"), &default_flags()).is_err());
|
||||
assert!(parse_command(&args("scroll --at 1"), &default_flags()).is_err());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_scroll_frame_index() {
|
||||
let cmd = parse_command(&args("scroll down 700 --frame 2"), &default_flags()).unwrap();
|
||||
assert_eq!(cmd["action"], "scroll");
|
||||
assert_eq!(cmd["frame"], 2);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_scroll_frame_rejects_non_integer() {
|
||||
assert!(parse_command(&args("scroll --frame two"), &default_flags()).is_err());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_scroll_selector_before_positional() {
|
||||
let cmd =
|
||||
|
||||
@@ -595,7 +595,7 @@ fn query_current_url(session: &str) -> Option<String> {
|
||||
}
|
||||
|
||||
/// Kill a running daemon by reading its PID file and sending a kill signal.
|
||||
fn kill_stale_daemon(session: &str) {
|
||||
pub fn kill_stale_daemon(session: &str) {
|
||||
// Remove the socket first so no new connections reach the old daemon
|
||||
#[cfg(unix)]
|
||||
{
|
||||
|
||||
+180
@@ -10,6 +10,7 @@ mod flags;
|
||||
mod install;
|
||||
mod native;
|
||||
mod output;
|
||||
mod site;
|
||||
mod skills;
|
||||
mod test_runner;
|
||||
#[cfg(test)]
|
||||
@@ -304,6 +305,49 @@ fn run_session(args: &[String], session: &str, json_mode: bool) {
|
||||
}
|
||||
}
|
||||
}
|
||||
// Stop a specific session daemon (issue #48). Graceful: kill_stale_daemon
|
||||
// sends SIGTERM first, so the daemon's shutdown handler runs `close()` and
|
||||
// tidies the tabs IT created (its tab group) before exiting.
|
||||
Some("stop") => {
|
||||
let target = args.get(2).map(|s| s.as_str()).unwrap_or(session);
|
||||
connection::kill_stale_daemon(target);
|
||||
if json_mode {
|
||||
print_json_value(json!({ "success": true, "data": { "stopped": target } }));
|
||||
} else {
|
||||
println!(
|
||||
"{} stopped session daemon: {}",
|
||||
color::success_indicator(),
|
||||
target
|
||||
);
|
||||
}
|
||||
}
|
||||
// Reclaim ALL session daemons now (issue #48) — for clearing the pile of
|
||||
// idle daemons left after a round of automation/debugging without waiting
|
||||
// for the idle timeout. Each is stopped gracefully (closes its own tabs);
|
||||
// they respawn clean on next use. The `__nm-host` relay is not a tracked
|
||||
// session daemon, so the extension/live-Chrome connection survives.
|
||||
Some("prune") => {
|
||||
let sessions: Vec<String> = walk_daemons()
|
||||
.sessions
|
||||
.into_iter()
|
||||
.map(|s| s.name)
|
||||
.collect();
|
||||
for s in &sessions {
|
||||
connection::kill_stale_daemon(s);
|
||||
}
|
||||
if json_mode {
|
||||
print_json_value(json!({ "success": true, "data": { "pruned": sessions } }));
|
||||
} else if sessions.is_empty() {
|
||||
println!("No session daemons to prune");
|
||||
} else {
|
||||
println!(
|
||||
"{} pruned {} session daemon(s): {}",
|
||||
color::success_indicator(),
|
||||
sessions.len(),
|
||||
sessions.join(", ")
|
||||
);
|
||||
}
|
||||
}
|
||||
None | Some(_) => {
|
||||
// Just show current session
|
||||
if json_mode {
|
||||
@@ -834,6 +878,105 @@ fn main() {
|
||||
exit(test_runner::run_test(suite, &flags));
|
||||
}
|
||||
|
||||
// Handle `site`: site adapters — turn a website into a structured-data CLI by
|
||||
// running a per-command JS adapter inside your logged-in tab. `update`/`list`/
|
||||
// `info` are CLI-side (download/filesystem); `site <name>/<cmd> [args]` falls
|
||||
// through to the daemon dispatch below (navigate to the adapter's domain + eval).
|
||||
if clean.first().map(|s| s.as_str()) == Some("site") {
|
||||
// Auto-sync the adapter pack on first use and periodically (TTL, default
|
||||
// 7d) so adapters stay fresh without a manual `site update`. Skipped for an
|
||||
// explicit `update` (full sync below). Best-effort: offline → cached pack.
|
||||
// Disable with AGENT_BROWSER_SITES_NO_AUTO_UPDATE=1.
|
||||
if clean.get(1).map(|s| s.as_str()) != Some("update") && site::needs_refresh() {
|
||||
let rt = tokio::runtime::Runtime::new().expect("Failed to create tokio runtime");
|
||||
match rt.block_on(site::update()) {
|
||||
Ok(n) => {
|
||||
eprintln!(
|
||||
"{}",
|
||||
color::dim(&format!("site: synced {n} adapters (auto)"))
|
||||
)
|
||||
}
|
||||
Err(e) => eprintln!(
|
||||
"{}",
|
||||
color::dim(&format!(
|
||||
"site: auto-sync skipped ({e}); using cached adapters"
|
||||
))
|
||||
),
|
||||
}
|
||||
}
|
||||
match clean.get(1).map(|s| s.as_str()) {
|
||||
Some("update") => {
|
||||
let rt = tokio::runtime::Runtime::new().expect("Failed to create tokio runtime");
|
||||
match rt.block_on(site::update()) {
|
||||
Ok(n) if flags.json => {
|
||||
println!("{}", json!({ "success": true, "adapters": n }))
|
||||
}
|
||||
Ok(n) => println!(
|
||||
"{} synced {} site adapters → ~/.chrome-use/sites (run `chrome-use site list`)",
|
||||
color::success_indicator(),
|
||||
n
|
||||
),
|
||||
Err(e) => {
|
||||
eprintln!("{} {}", color::error_indicator(), e);
|
||||
exit(1);
|
||||
}
|
||||
}
|
||||
return;
|
||||
}
|
||||
Some("list") => {
|
||||
match site::list_adapters() {
|
||||
Ok(list) if flags.json => {
|
||||
println!("{}", json!({ "success": true, "adapters": list }))
|
||||
}
|
||||
Ok(list) if list.is_empty() => {
|
||||
println!("no site adapters installed — run `chrome-use site update`")
|
||||
}
|
||||
Ok(list) => {
|
||||
for a in &list {
|
||||
println!("{a}");
|
||||
}
|
||||
eprintln!(
|
||||
"{}",
|
||||
color::dim(&format!(
|
||||
"{} adapters · run: chrome-use site <name>/<cmd> [args]",
|
||||
list.len()
|
||||
))
|
||||
);
|
||||
}
|
||||
Err(e) => {
|
||||
eprintln!("{} {}", color::error_indicator(), e);
|
||||
exit(1);
|
||||
}
|
||||
}
|
||||
return;
|
||||
}
|
||||
Some("info") => {
|
||||
let spec = clean.get(2).cloned().unwrap_or_default();
|
||||
match site::load_adapter(&spec) {
|
||||
Ok(a) => println!(
|
||||
"{}",
|
||||
serde_json::to_string_pretty(&a.meta).unwrap_or_default()
|
||||
),
|
||||
Err(e) => {
|
||||
eprintln!("{} {}", color::error_indicator(), e);
|
||||
exit(1);
|
||||
}
|
||||
}
|
||||
return;
|
||||
}
|
||||
// `site <name>/<cmd> [args]` → fall through to the daemon dispatch.
|
||||
Some(spec) if spec.contains('/') => {}
|
||||
_ => {
|
||||
eprintln!(
|
||||
"{} usage: chrome-use site <name>/<cmd> [args] | site update | site list | \
|
||||
site info <name>/<cmd>",
|
||||
color::error_indicator()
|
||||
);
|
||||
exit(2);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Handle skills command (doesn't need daemon)
|
||||
if clean.first().map(|s| s.as_str()) == Some("skills") {
|
||||
skills::run_skills(&clean, flags.json);
|
||||
@@ -881,6 +1024,43 @@ fn main() {
|
||||
}
|
||||
}
|
||||
|
||||
// `adopt <url|targetId>`: read a PRE-EXISTING tab (the user's own, or another
|
||||
// session's) WITHOUT opening a new one. Forces a fresh daemon and points it at
|
||||
// the relay (like `extension connect`); the AGENT_BROWSER_ADOPT env makes the
|
||||
// daemon's first connect ADOPT the matching tab instead of creating an
|
||||
// about:blank. Rewrites into `connect <relay-url>` BEFORE parse_command so the
|
||||
// daemon attaches to the user's real Chrome. Must run before parse_command.
|
||||
if clean.first().map(|s| s.as_str()) == Some("adopt") {
|
||||
match clean.get(1) {
|
||||
Some(spec) if !spec.trim().is_empty() => {
|
||||
std::env::set_var("AGENT_BROWSER_ADOPT", spec.trim());
|
||||
connection::kill_stale_daemon(&flags.session);
|
||||
match connect::relay_url() {
|
||||
Some(url) => {
|
||||
flags.cdp = Some(url.clone());
|
||||
flags.auto_connect = false;
|
||||
clean = vec!["connect".to_string(), url];
|
||||
}
|
||||
None => {
|
||||
eprintln!(
|
||||
"{} extension relay not connected — open Chrome with the ab-connect \
|
||||
extension first (this command reads an EXISTING tab, it won't launch one).",
|
||||
color::error_indicator()
|
||||
);
|
||||
exit(1);
|
||||
}
|
||||
}
|
||||
}
|
||||
_ => {
|
||||
eprintln!(
|
||||
"{} usage: chrome-use adopt <url-substring|targetId> (reads an existing tab, no new tab)",
|
||||
color::error_indicator()
|
||||
);
|
||||
exit(2);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Handle session separately (doesn't need daemon)
|
||||
if clean.first().map(|s| s.as_str()) == Some("session") {
|
||||
run_session(&clean, &flags.session, flags.json);
|
||||
|
||||
+586
-17
@@ -1315,10 +1315,14 @@ pub async fn execute_command(cmd: &Value, state: &mut DaemonState) -> Value {
|
||||
"title" => handle_title(state).await,
|
||||
"content" => handle_content(state).await,
|
||||
"evaluate" => handle_evaluate(cmd, state).await,
|
||||
"site" => handle_site(cmd, state).await,
|
||||
"close" => handle_close(state).await,
|
||||
"keep" => handle_keep(state).await,
|
||||
"stealth_status" => handle_stealth_status(state).await,
|
||||
"snapshot" => handle_snapshot(cmd, state).await,
|
||||
"screenshot" => handle_screenshot(cmd, state).await,
|
||||
"canvas_list" => handle_canvas_list(state).await,
|
||||
"canvas_capture" => handle_canvas_capture(cmd, state).await,
|
||||
"click" => handle_click(cmd, state).await,
|
||||
"dblclick" => handle_dblclick(cmd, state).await,
|
||||
"fill" => handle_fill(cmd, state).await,
|
||||
@@ -2477,7 +2481,10 @@ async fn handle_navigate(cmd: &Value, state: &mut DaemonState) -> Result<Value,
|
||||
wb.navigate(url).await?;
|
||||
let new_url = wb.get_url().await.unwrap_or_else(|_| url.to_string());
|
||||
let title = wb.get_title().await.unwrap_or_default();
|
||||
return Ok(json!({ "url": new_url, "title": title }));
|
||||
return Ok(with_site_hint(
|
||||
json!({ "url": new_url, "title": title }),
|
||||
url,
|
||||
));
|
||||
}
|
||||
}
|
||||
|
||||
@@ -2544,7 +2551,7 @@ async fn handle_navigate(cmd: &Value, state: &mut DaemonState) -> Result<Value,
|
||||
.unwrap_or(false)
|
||||
{
|
||||
if let Ok(Some(switched)) = mgr.reuse_tab_for_url(url).await {
|
||||
return Ok(switched);
|
||||
return Ok(with_site_hint(switched, url));
|
||||
}
|
||||
}
|
||||
|
||||
@@ -2552,7 +2559,34 @@ async fn handle_navigate(cmd: &Value, state: &mut DaemonState) -> Result<Value,
|
||||
// Adaptive humanize: sample the freshly loaded page for known behavioural
|
||||
// anti-bot vendors and escalate this session to Human if any are present.
|
||||
detect_and_set_humanize(mgr).await;
|
||||
Ok(result)
|
||||
Ok(with_site_hint(result, url))
|
||||
}
|
||||
|
||||
/// Annotate a navigation/snapshot result with the `site` adapters available for
|
||||
/// the page's domain (auto-trigger): when you land on e.g. github.com, the
|
||||
/// response carries `siteAdapters: { domain, commands: ["github/issues", …] }` so
|
||||
/// the agent reaches for a structured-data adapter instead of scraping. No-op
|
||||
/// when nothing matches or the pack isn't synced yet.
|
||||
fn with_site_hint(mut result: Value, fallback_url: &str) -> Value {
|
||||
let url = result
|
||||
.get("url")
|
||||
.and_then(|v| v.as_str())
|
||||
.unwrap_or(fallback_url);
|
||||
let host = url::Url::parse(url)
|
||||
.ok()
|
||||
.and_then(|u| u.host_str().map(String::from));
|
||||
if let Some(host) = host {
|
||||
let adapters = crate::site::adapters_for_domain(&host);
|
||||
if !adapters.is_empty() {
|
||||
if let Some(obj) = result.as_object_mut() {
|
||||
obj.insert(
|
||||
"siteAdapters".to_string(),
|
||||
json!({ "domain": host, "commands": adapters }),
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
result
|
||||
}
|
||||
|
||||
/// After navigation, probe the page for known anti-bot vendor fingerprints
|
||||
@@ -2698,6 +2732,47 @@ async fn handle_evaluate(cmd: &Value, state: &DaemonState) -> Result<Value, Stri
|
||||
Ok(json!({ "result": result, "origin": url }))
|
||||
}
|
||||
|
||||
/// Run a site adapter: navigate to its `@meta.domain` (only if we're not already
|
||||
/// there — the point is to run as you, in the page that's already open) and eval
|
||||
/// the adapter function in the site's own logged-in page. The CLI/commands.rs has
|
||||
/// already loaded the adapter and built the `script`; here we just place the page
|
||||
/// and evaluate. Never disrupts the user's foreground tab — navigation happens on
|
||||
/// the daemon's own tab (same as every other command on the relay).
|
||||
async fn handle_site(cmd: &Value, state: &mut DaemonState) -> Result<Value, String> {
|
||||
let domain = cmd
|
||||
.get("domain")
|
||||
.and_then(|v| v.as_str())
|
||||
.ok_or("site: missing 'domain'")?
|
||||
.to_string();
|
||||
let script = cmd
|
||||
.get("script")
|
||||
.and_then(|v| v.as_str())
|
||||
.ok_or("site: missing 'script'")?
|
||||
.to_string();
|
||||
|
||||
let current = match state.browser.as_ref() {
|
||||
Some(mgr) => mgr.get_url().await.unwrap_or_default(),
|
||||
None => String::new(),
|
||||
};
|
||||
let on_domain = url::Url::parse(¤t)
|
||||
.ok()
|
||||
.and_then(|u| u.host_str().map(|h| h.to_string()))
|
||||
.map(|h| h == domain || h.ends_with(&format!(".{domain}")))
|
||||
.unwrap_or(false);
|
||||
if !on_domain {
|
||||
let nav = json!({ "url": format!("https://{domain}/") });
|
||||
handle_navigate(&nav, state).await?;
|
||||
}
|
||||
|
||||
let eval_cmd = json!({ "script": script });
|
||||
let out = handle_evaluate(&eval_cmd, state).await?;
|
||||
Ok(json!({
|
||||
"result": out.get("result").cloned().unwrap_or(Value::Null),
|
||||
"origin": out.get("origin").cloned().unwrap_or(Value::Null),
|
||||
"domain": domain,
|
||||
}))
|
||||
}
|
||||
|
||||
/// Local stealth self-check: reports the active mode, live fingerprint probes,
|
||||
/// and the list of applied overrides — so an agent (or human) can confirm
|
||||
/// stealth is working without driving an external detector, and audit exactly
|
||||
@@ -2781,6 +2856,34 @@ async fn handle_stealth_status(state: &DaemonState) -> Result<Value, String> {
|
||||
}))
|
||||
}
|
||||
|
||||
/// `keep` — leave the ACTIVE tab for the user: stop owning it (so the daemon's
|
||||
/// `close()`/idle-shutdown won't close it) and best-effort remove it from this
|
||||
/// session's tab group so it looks like a normal user tab. The "leave for the
|
||||
/// user" half of the auto-close-on-idle cleanup: scratch tabs get closed, tabs
|
||||
/// the agent explicitly `keep`s stay. (Adopted user tabs are never owned, so
|
||||
/// they're already safe.)
|
||||
async fn handle_keep(state: &mut DaemonState) -> Result<Value, String> {
|
||||
let mgr = state.browser.as_mut().ok_or("Browser not launched")?;
|
||||
let target_id = mgr.active_target_id()?.to_string();
|
||||
let session_id = mgr.active_session_id()?.to_string();
|
||||
let was_owned = mgr.unown_target(&target_id);
|
||||
// Best-effort: ask the extension to ungroup the tab (relay only; no-ops on a
|
||||
// launched browser or an older extension that doesn't know ABExt.ungroupTab).
|
||||
let _ = mgr
|
||||
.client
|
||||
.send_command_typed::<_, Value>(
|
||||
"ABExt.ungroupTab",
|
||||
&json!({ "sessionId": session_id, "targetId": target_id }),
|
||||
None,
|
||||
)
|
||||
.await;
|
||||
Ok(json!({
|
||||
"kept": target_id,
|
||||
"wasOwned": was_owned,
|
||||
"note": "tab left for the user — exempt from auto-close, removed from the session tab group",
|
||||
}))
|
||||
}
|
||||
|
||||
async fn handle_close(state: &mut DaemonState) -> Result<Value, String> {
|
||||
if let Some(ref mgr) = state.browser {
|
||||
if let Some(ref session_name) = state.session_name {
|
||||
@@ -2894,6 +2997,13 @@ async fn handle_snapshot(cmd: &Value, state: &mut DaemonState) -> Result<Value,
|
||||
let ref_count = refs.len();
|
||||
let mut out = json!({ "snapshot": tree, "origin": url, "refs": refs });
|
||||
|
||||
// Auto-trigger: if this domain has site adapters, surface them so the agent
|
||||
// pulls structured data instead of walking the tree. `with_site_hint` reads
|
||||
// the `url` field, so pass it under that key.
|
||||
if let Some(hint) = with_site_hint(json!({ "url": url }), &url).get("siteAdapters") {
|
||||
out["siteAdapters"] = hint.clone();
|
||||
}
|
||||
|
||||
// Canvas/WebGL apps (games, map/3D viewers, drawing tools) paint to a
|
||||
// <canvas> and expose almost no accessibility tree, so `snapshot` comes back
|
||||
// near-empty and agents get stuck looking for refs that will never exist
|
||||
@@ -3035,7 +3145,40 @@ async fn handle_screenshot(cmd: &Value, state: &mut DaemonState) -> Result<Value
|
||||
)
|
||||
.await?;
|
||||
|
||||
// Downscale the saved image so retina/full-page shots fit an agent's image
|
||||
// reader and screenshot pixels line up with `click x y` CSS px (issue #42).
|
||||
// Explicit --scale / --max-width / --max-height win; otherwise a default cap
|
||||
// (2000px longest edge, AGENT_BROWSER_SCREENSHOT_MAX_EDGE overrides, 0 = off)
|
||||
// applies. Annotated shots are left untouched so ref overlays stay aligned.
|
||||
let mut resized: Option<(u32, u32)> = None;
|
||||
if !annotate {
|
||||
let scale = cmd.get("scale").and_then(|v| v.as_f64());
|
||||
let max_w = cmd
|
||||
.get("maxWidth")
|
||||
.and_then(|v| v.as_u64())
|
||||
.map(|v| v as u32);
|
||||
let max_h = cmd
|
||||
.get("maxHeight")
|
||||
.and_then(|v| v.as_u64())
|
||||
.map(|v| v as u32);
|
||||
let default_edge = if scale.is_none() && max_w.is_none() && max_h.is_none() {
|
||||
std::env::var("AGENT_BROWSER_SCREENSHOT_MAX_EDGE")
|
||||
.ok()
|
||||
.and_then(|s| s.parse::<u32>().ok())
|
||||
.or(Some(2000))
|
||||
.filter(|&e| e > 0)
|
||||
} else {
|
||||
None
|
||||
};
|
||||
resized = downscale_screenshot(&result.path, scale, max_w, max_h, default_edge);
|
||||
}
|
||||
|
||||
let mut response = json!({ "path": absolutize_saved_path(&result.path) });
|
||||
if let Some((w, h)) = resized {
|
||||
response["width"] = json!(w);
|
||||
response["height"] = json!(h);
|
||||
response["resized"] = json!(true);
|
||||
}
|
||||
if !result.annotations.is_empty() {
|
||||
response["annotations"] = serde_json::to_value(&result.annotations)
|
||||
.map_err(|e| format!("Failed to serialize annotations: {}", e))?;
|
||||
@@ -3051,6 +3194,204 @@ async fn handle_screenshot(cmd: &Value, state: &mut DaemonState) -> Result<Value
|
||||
Ok(response)
|
||||
}
|
||||
|
||||
/// Downscale a saved screenshot in place (issue #42). Resolves the target longest
|
||||
/// edge from `scale` (fraction of current), explicit `max_w`/`max_h` caps, or a
|
||||
/// `default_edge` cap — whichever yields the smaller image. Only ever shrinks;
|
||||
/// no-op (returns None) if the image is already within bounds or can't be read.
|
||||
/// Returns the new (width, height) when it actually resized.
|
||||
fn downscale_screenshot(
|
||||
path: &str,
|
||||
scale: Option<f64>,
|
||||
max_w: Option<u32>,
|
||||
max_h: Option<u32>,
|
||||
default_edge: Option<u32>,
|
||||
) -> Option<(u32, u32)> {
|
||||
let img = image::open(path).ok()?;
|
||||
let (w, h) = (img.width(), img.height());
|
||||
if w == 0 || h == 0 {
|
||||
return None;
|
||||
}
|
||||
|
||||
// Collect candidate scale factors (≤ 1.0); the smallest wins.
|
||||
let mut factor = 1.0f64;
|
||||
if let Some(s) = scale {
|
||||
factor = factor.min(s);
|
||||
}
|
||||
if let Some(mw) = max_w {
|
||||
if w > mw {
|
||||
factor = factor.min(mw as f64 / w as f64);
|
||||
}
|
||||
}
|
||||
if let Some(mh) = max_h {
|
||||
if h > mh {
|
||||
factor = factor.min(mh as f64 / h as f64);
|
||||
}
|
||||
}
|
||||
if let Some(edge) = default_edge {
|
||||
let longest = w.max(h);
|
||||
if longest > edge {
|
||||
factor = factor.min(edge as f64 / longest as f64);
|
||||
}
|
||||
}
|
||||
|
||||
if factor >= 1.0 {
|
||||
return None; // already within bounds — never upscale
|
||||
}
|
||||
let nw = ((w as f64 * factor).round() as u32).max(1);
|
||||
let nh = ((h as f64 * factor).round() as u32).max(1);
|
||||
let resized = img.resize(nw, nh, image::imageops::FilterType::Lanczos3);
|
||||
resized.save(path).ok()?;
|
||||
Some((resized.width(), resized.height()))
|
||||
}
|
||||
|
||||
/// `canvas list` — enumerate <canvas> elements (size, visibility, whether
|
||||
/// toDataURL is usable) so an agent can pick one to capture on a canvas/WebGL app
|
||||
/// where snapshot/DOM reads see nothing.
|
||||
async fn handle_canvas_list(state: &DaemonState) -> Result<Value, String> {
|
||||
let mgr = state.browser.as_ref().ok_or("Browser not launched")?;
|
||||
let js = r#"(() => [...document.querySelectorAll('canvas')].map((c, i) => {
|
||||
const r = c.getBoundingClientRect();
|
||||
let toDataUrl = true, tainted = false;
|
||||
try { c.toDataURL('image/png'); } catch (e) { toDataUrl = false; tainted = true; }
|
||||
return {
|
||||
index: i,
|
||||
backingWidth: c.width, backingHeight: c.height,
|
||||
cssWidth: Math.round(r.width), cssHeight: Math.round(r.height),
|
||||
visible: r.width > 0 && r.height > 0 && r.bottom > 0 && r.top < innerHeight,
|
||||
id: c.id || null, className: c.className || null,
|
||||
toDataUrl, tainted,
|
||||
};
|
||||
}))()"#;
|
||||
let canvases = mgr.evaluate(js, None).await?;
|
||||
Ok(json!({ "canvases": canvases }))
|
||||
}
|
||||
|
||||
/// `canvas capture [selector] [path]` — save a canvas's rendered pixels to PNG.
|
||||
/// Prefers `toDataURL` (full backing-store resolution); falls back to a CDP
|
||||
/// screenshot of the canvas element when toDataURL is blank (WebGL without
|
||||
/// preserveDrawingBuffer, e.g. Figma) or throws (cross-origin tainted). This is
|
||||
/// how chrome-use "sees" canvas/WebGL apps that expose no DOM or refs.
|
||||
async fn handle_canvas_capture(cmd: &Value, state: &mut DaemonState) -> Result<Value, String> {
|
||||
let selector = cmd
|
||||
.get("selector")
|
||||
.and_then(|v| v.as_str())
|
||||
.map(String::from);
|
||||
let force_screenshot = cmd
|
||||
.get("forceScreenshot")
|
||||
.and_then(|v| v.as_bool())
|
||||
.unwrap_or(false);
|
||||
let out_path = cmd.get("path").and_then(|v| v.as_str()).map(String::from);
|
||||
let id = cmd.get("id").and_then(|v| v.as_str()).unwrap_or("0");
|
||||
|
||||
// Probe: locate the canvas, get its bbox, and try toDataURL (unless forced).
|
||||
let probe = {
|
||||
let mgr = state.browser.as_ref().ok_or("Browser not launched")?;
|
||||
let sel_lit = match &selector {
|
||||
Some(s) => serde_json::to_string(s).unwrap_or_else(|_| "null".to_string()),
|
||||
None => "null".to_string(),
|
||||
};
|
||||
let js = format!(
|
||||
r#"(() => {{
|
||||
const sel = {sel_lit};
|
||||
const c = sel ? document.querySelector(sel)
|
||||
: [...document.querySelectorAll('canvas')].sort((a,b)=>(b.width*b.height)-(a.width*a.height))[0];
|
||||
if (!c) return {{ found: false, count: document.querySelectorAll('canvas').length }};
|
||||
const r = c.getBoundingClientRect();
|
||||
let dataUrl = null, err = null;
|
||||
if ({try_data}) {{ try {{ dataUrl = c.toDataURL('image/png'); }} catch (e) {{ err = String(e && e.message || e); }} }}
|
||||
return {{ found: true, w: c.width, h: c.height, x: r.x, y: r.y, cw: r.width, ch: r.height, dataUrl, err }};
|
||||
}})()"#,
|
||||
try_data = !force_screenshot
|
||||
);
|
||||
mgr.evaluate(&js, None).await?
|
||||
};
|
||||
|
||||
if !probe
|
||||
.get("found")
|
||||
.and_then(|v| v.as_bool())
|
||||
.unwrap_or(false)
|
||||
{
|
||||
let count = probe.get("count").and_then(|v| v.as_u64()).unwrap_or(0);
|
||||
return Err(format!(
|
||||
"canvas: no canvas matched{} ({count} canvas element(s) on the page — try `canvas list`)",
|
||||
selector
|
||||
.as_deref()
|
||||
.map(|s| format!(" `{s}`"))
|
||||
.unwrap_or_default()
|
||||
));
|
||||
}
|
||||
let backing_w = probe.get("w").and_then(|v| v.as_u64()).unwrap_or(0);
|
||||
let backing_h = probe.get("h").and_then(|v| v.as_u64()).unwrap_or(0);
|
||||
|
||||
// Decode toDataURL if present; a WebGL canvas without preserveDrawingBuffer
|
||||
// returns a blank PNG (tiny when compressed), so reject suspiciously small
|
||||
// results and fall back to the screenshot path.
|
||||
let decoded: Option<Vec<u8>> = probe
|
||||
.get("dataUrl")
|
||||
.and_then(|v| v.as_str())
|
||||
.and_then(|u| u.split_once(",").map(|(_, b)| b.to_string()))
|
||||
.and_then(|b64| {
|
||||
base64::Engine::decode(&base64::engine::general_purpose::STANDARD, b64.as_bytes()).ok()
|
||||
});
|
||||
let use_data = !force_screenshot && decoded.as_ref().map(|d| d.len() > 1024).unwrap_or(false);
|
||||
|
||||
if use_data {
|
||||
let bytes = decoded.unwrap();
|
||||
let path = out_path.unwrap_or_else(|| {
|
||||
std::env::temp_dir()
|
||||
.join(format!("canvas-{id}.png"))
|
||||
.to_string_lossy()
|
||||
.into_owned()
|
||||
});
|
||||
std::fs::write(&path, &bytes).map_err(|e| format!("canvas: write {path}: {e}"))?;
|
||||
Ok(json!({
|
||||
"path": absolutize_saved_path(&path),
|
||||
"method": "toDataURL",
|
||||
"width": backing_w,
|
||||
"height": backing_h,
|
||||
}))
|
||||
} else {
|
||||
// Fallback: screenshot the canvas element (or its bbox clip).
|
||||
let clip = if selector.is_none() {
|
||||
let x = probe.get("x").and_then(|v| v.as_f64()).unwrap_or(0.0);
|
||||
let y = probe.get("y").and_then(|v| v.as_f64()).unwrap_or(0.0);
|
||||
let cw = probe.get("cw").and_then(|v| v.as_f64()).unwrap_or(0.0);
|
||||
let ch = probe.get("ch").and_then(|v| v.as_f64()).unwrap_or(0.0);
|
||||
Some((x, y, cw, ch))
|
||||
} else {
|
||||
None
|
||||
};
|
||||
let options = ScreenshotOptions {
|
||||
selector: selector.clone(),
|
||||
path: out_path,
|
||||
clip,
|
||||
..ScreenshotOptions::default()
|
||||
};
|
||||
let mgr = state.browser.as_ref().ok_or("Browser not launched")?;
|
||||
let session_id = mgr.active_session_id()?.to_string();
|
||||
let result = screenshot::take_screenshot(
|
||||
&mgr.client,
|
||||
&session_id,
|
||||
&state.ref_map,
|
||||
&options,
|
||||
&state.iframe_sessions,
|
||||
)
|
||||
.await?;
|
||||
let why = probe
|
||||
.get("err")
|
||||
.and_then(|v| v.as_str())
|
||||
.map(|e| format!("toDataURL failed ({e})"))
|
||||
.unwrap_or_else(|| {
|
||||
"toDataURL blank/unavailable (WebGL no preserveDrawingBuffer)".into()
|
||||
});
|
||||
Ok(json!({
|
||||
"path": absolutize_saved_path(&result.path),
|
||||
"method": "screenshot",
|
||||
"note": format!("captured rendered pixels via screenshot — {why}"),
|
||||
}))
|
||||
}
|
||||
}
|
||||
|
||||
async fn handle_click(cmd: &Value, state: &mut DaemonState) -> Result<Value, String> {
|
||||
// First-class coordinate click (issue #8.4): click a raw viewport point with
|
||||
// no element resolution. Parsed from `click <x> <y>` / `click --coords x,y`.
|
||||
@@ -3209,7 +3550,7 @@ async fn handle_fill(cmd: &Value, state: &mut DaemonState) -> Result<Value, Stri
|
||||
let mgr = state.browser.as_ref().ok_or("Browser not launched")?;
|
||||
let session_id = mgr.active_session_id()?.to_string();
|
||||
|
||||
interaction::fill(
|
||||
let engine = interaction::fill(
|
||||
&mgr.client,
|
||||
&session_id,
|
||||
&state.ref_map,
|
||||
@@ -3218,13 +3559,23 @@ async fn handle_fill(cmd: &Value, state: &mut DaemonState) -> Result<Value, Stri
|
||||
&state.iframe_sessions,
|
||||
)
|
||||
.await?;
|
||||
Ok(json!({ "filled": selector }))
|
||||
// Echo the input path used (input/contenteditable/codemirror5/monaco/select)
|
||||
// so the agent can confirm a rich editor was handled, not silently no-op'd (#41).
|
||||
Ok(json!({ "filled": selector, "engine": engine }))
|
||||
}
|
||||
|
||||
async fn handle_type(cmd: &Value, state: &mut DaemonState) -> Result<Value, String> {
|
||||
let mgr = state.browser.as_ref().ok_or("Browser not launched")?;
|
||||
let session_id = mgr.active_session_id()?.to_string();
|
||||
|
||||
// `--key-events`: dispatch real per-character keyDown/keyUp instead of
|
||||
// Input.insertText, so autocomplete/combobox widgets that only react to key
|
||||
// events fire (e.g. Google's address postal-code lookup) (issue #4/#36).
|
||||
let key_events = cmd
|
||||
.get("keyEvents")
|
||||
.and_then(|v| v.as_bool())
|
||||
.unwrap_or(false);
|
||||
|
||||
// `type --focused <text>`: type into the currently-focused element without a
|
||||
// selector (custom widgets that move focus to a hidden input on open).
|
||||
if cmd
|
||||
@@ -3236,7 +3587,14 @@ async fn handle_type(cmd: &Value, state: &mut DaemonState) -> Result<Value, Stri
|
||||
.get("text")
|
||||
.and_then(|v| v.as_str())
|
||||
.ok_or("Missing 'text' parameter")?;
|
||||
interaction::type_text_into_active_context(&mgr.client, &session_id, text, None).await?;
|
||||
interaction::type_text_into_active_context(
|
||||
&mgr.client,
|
||||
&session_id,
|
||||
text,
|
||||
None,
|
||||
key_events,
|
||||
)
|
||||
.await?;
|
||||
return Ok(json!({ "typed": text, "focused": true }));
|
||||
}
|
||||
|
||||
@@ -3260,6 +3618,7 @@ async fn handle_type(cmd: &Value, state: &mut DaemonState) -> Result<Value, Stri
|
||||
clear,
|
||||
delay,
|
||||
&state.iframe_sessions,
|
||||
key_events,
|
||||
)
|
||||
.await?;
|
||||
Ok(json!({ "typed": text }))
|
||||
@@ -3467,17 +3826,182 @@ async fn handle_scroll(cmd: &Value, state: &mut DaemonState) -> Result<Value, St
|
||||
}
|
||||
}
|
||||
|
||||
// An explicit `--selector` keeps the precise element-scroll path (scrollBy on
|
||||
// the resolved node, same-origin only).
|
||||
if let Some(sel) = selector {
|
||||
interaction::scroll(
|
||||
&mgr.client,
|
||||
&session_id,
|
||||
&state.ref_map,
|
||||
Some(sel),
|
||||
dx,
|
||||
dy,
|
||||
&state.iframe_sessions,
|
||||
)
|
||||
.await?;
|
||||
return Ok(json!({ "scrolled": true, "via": "selector" }));
|
||||
}
|
||||
|
||||
// `--at x,y` / `--frame n`: dispatch a real (isTrusted) wheel at a viewport
|
||||
// coordinate. This hits the compositor and scrolls whatever scroll container
|
||||
// is under the pointer — including cross-origin iframes that `window.scrollBy`
|
||||
// on the top document silently no-ops on (issue #36).
|
||||
if cmd.get("at").is_some() || cmd.get("frame").is_some() {
|
||||
let (x, y, via) = if let Some(at) = cmd.get("at").and_then(|v| v.as_array()) {
|
||||
let x = at.first().and_then(|v| v.as_f64()).unwrap_or(0.0);
|
||||
let y = at.get(1).and_then(|v| v.as_f64()).unwrap_or(0.0);
|
||||
(x, y, "at")
|
||||
} else {
|
||||
let n = cmd.get("frame").and_then(|v| v.as_u64()).unwrap_or(0);
|
||||
let (x, y) = frame_center(mgr, &session_id, &state.iframe_sessions, n as usize).await?;
|
||||
(x, y, "frame")
|
||||
};
|
||||
dispatch_wheel(&mgr.client, &session_id, x, y, dx, dy).await?;
|
||||
return Ok(json!({ "scrolled": true, "via": via, "at": [x, y] }));
|
||||
}
|
||||
|
||||
// Default (no selector/at/frame): scroll the page with `window.scrollBy`. This
|
||||
// is the reliable path for ordinary page scrolling; a coordinate wheel at the
|
||||
// viewport centre is NOT a dependable substitute (it no-ops on some pages,
|
||||
// e.g. headless), so the wheel stays opt-in via `--at`/`--frame` for the
|
||||
// cross-origin-iframe case (issue #36).
|
||||
interaction::scroll(
|
||||
&mgr.client,
|
||||
&session_id,
|
||||
&state.ref_map,
|
||||
selector,
|
||||
None,
|
||||
dx,
|
||||
dy,
|
||||
&state.iframe_sessions,
|
||||
)
|
||||
.await?;
|
||||
Ok(json!({ "scrolled": true }))
|
||||
Ok(json!({ "scrolled": true, "via": "page" }))
|
||||
}
|
||||
|
||||
/// Viewport center in CSS pixels, used as the default wheel landing point for
|
||||
/// `scroll` (issue #36). Falls back to a sane 640×400 center if the page can't
|
||||
/// be evaluated (e.g. a restricted document).
|
||||
async fn viewport_center(mgr: &BrowserManager, session_id: &str) -> Result<(f64, f64), String> {
|
||||
let dims = mgr
|
||||
.client
|
||||
.send_command_typed::<_, Value>(
|
||||
"Runtime.evaluate",
|
||||
&super::cdp::types::EvaluateParams {
|
||||
expression: "[window.innerWidth, window.innerHeight]".to_string(),
|
||||
return_by_value: Some(true),
|
||||
await_promise: Some(false),
|
||||
},
|
||||
Some(session_id),
|
||||
)
|
||||
.await
|
||||
.ok();
|
||||
let arr = dims
|
||||
.as_ref()
|
||||
.and_then(|v| v.get("result"))
|
||||
.and_then(|v| v.get("value"))
|
||||
.and_then(|v| v.as_array());
|
||||
let w = arr
|
||||
.and_then(|a| a.first())
|
||||
.and_then(|v| v.as_f64())
|
||||
.filter(|w| *w > 0.0)
|
||||
.unwrap_or(1280.0);
|
||||
let h = arr
|
||||
.and_then(|a| a.get(1))
|
||||
.and_then(|v| v.as_f64())
|
||||
.filter(|h| *h > 0.0)
|
||||
.unwrap_or(800.0);
|
||||
Ok((w / 2.0, h / 2.0))
|
||||
}
|
||||
|
||||
/// Center of the `n`-th frame (as listed by `chrome-use frames`) in top-viewport
|
||||
/// CSS pixels, so `scroll --frame n` lands its wheel inside a cross-origin iframe
|
||||
/// without needing a selector into it (issue #36). Resolves the frame's owning
|
||||
/// `<iframe>` element box via `DOM.getFrameOwner` + `DOM.getBoxModel` — exact for
|
||||
/// a frame nested directly under the top document; for a deeper nesting the box is
|
||||
/// relative to the intermediate frame, so prefer `--at x,y` from a screenshot.
|
||||
async fn frame_center(
|
||||
mgr: &BrowserManager,
|
||||
session_id: &str,
|
||||
iframe_sessions: &HashMap<String, String>,
|
||||
n: usize,
|
||||
) -> Result<(f64, f64), String> {
|
||||
let frames =
|
||||
super::element::collect_all_frames_text(&mgr.client, session_id, iframe_sessions).await?;
|
||||
let frame = frames.get(n).ok_or_else(|| {
|
||||
format!(
|
||||
"frame index {} out of range (run `chrome-use frames`: {} frame(s))",
|
||||
n,
|
||||
frames.len()
|
||||
)
|
||||
})?;
|
||||
if n == 0 {
|
||||
// Frame 0 is the top document — there's no owner element; scroll its center.
|
||||
return viewport_center(mgr, session_id).await;
|
||||
}
|
||||
let owner = mgr
|
||||
.client
|
||||
.send_command_typed::<_, Value>(
|
||||
"DOM.getFrameOwner",
|
||||
&json!({ "frameId": frame.frame_id }),
|
||||
Some(session_id),
|
||||
)
|
||||
.await
|
||||
.map_err(|e| format!("can't locate frame {}'s owner element: {}", n, e))?;
|
||||
let backend_node_id = owner
|
||||
.get("backendNodeId")
|
||||
.and_then(|v| v.as_i64())
|
||||
.ok_or_else(|| format!("frame {} has no owner <iframe> element", n))?;
|
||||
let box_model = mgr
|
||||
.client
|
||||
.send_command_typed::<_, Value>(
|
||||
"DOM.getBoxModel",
|
||||
&json!({ "backendNodeId": backend_node_id }),
|
||||
Some(session_id),
|
||||
)
|
||||
.await
|
||||
.map_err(|e| format!("can't measure frame {}'s box: {}", n, e))?;
|
||||
let content = box_model
|
||||
.get("model")
|
||||
.and_then(|m| m.get("content"))
|
||||
.and_then(|c| c.as_array())
|
||||
.ok_or_else(|| format!("frame {} box model has no content quad", n))?;
|
||||
let coord = |i: usize| content.get(i).and_then(|v| v.as_f64()).unwrap_or(0.0);
|
||||
// content quad is [x1,y1, x2,y2, x3,y3, x4,y4]; opposite corners are 0 and 2.
|
||||
let cx = (coord(0) + coord(4)) / 2.0;
|
||||
let cy = (coord(1) + coord(5)) / 2.0;
|
||||
Ok((cx, cy))
|
||||
}
|
||||
|
||||
/// Dispatch a trusted mouse wheel at `(x, y)`, humanized like `handle_wheel`.
|
||||
async fn dispatch_wheel(
|
||||
client: &super::cdp::client::CdpClient,
|
||||
session_id: &str,
|
||||
x: f64,
|
||||
y: f64,
|
||||
delta_x: f64,
|
||||
delta_y: f64,
|
||||
) -> Result<(), String> {
|
||||
let level = humanize::active_level();
|
||||
let seed = humanize::next_seed();
|
||||
for (dx, dy, delay) in humanize::scroll_segments(delta_x, delta_y, level, seed) {
|
||||
client
|
||||
.send_command(
|
||||
"Input.dispatchMouseEvent",
|
||||
Some(json!({
|
||||
"type": "mouseWheel",
|
||||
"x": x,
|
||||
"y": y,
|
||||
"deltaX": dx,
|
||||
"deltaY": dy,
|
||||
})),
|
||||
Some(session_id),
|
||||
)
|
||||
.await?;
|
||||
if !delay.is_zero() {
|
||||
tokio::time::sleep(delay).await;
|
||||
}
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
async fn handle_select(cmd: &Value, state: &mut DaemonState) -> Result<Value, String> {
|
||||
@@ -3674,12 +4198,9 @@ async fn handle_gettext(cmd: &Value, state: &mut DaemonState) -> Result<Value, S
|
||||
async fn handle_frames(_cmd: &Value, state: &mut DaemonState) -> Result<Value, String> {
|
||||
let mgr = state.browser.as_ref().ok_or("Browser not launched")?;
|
||||
let session_id = mgr.active_session_id()?.to_string();
|
||||
let frames = super::element::collect_all_frames_text(
|
||||
&mgr.client,
|
||||
&session_id,
|
||||
&state.iframe_sessions,
|
||||
)
|
||||
.await?;
|
||||
let frames =
|
||||
super::element::collect_all_frames_text(&mgr.client, &session_id, &state.iframe_sessions)
|
||||
.await?;
|
||||
let list: Vec<Value> = frames
|
||||
.iter()
|
||||
.enumerate()
|
||||
@@ -4175,7 +4696,10 @@ async fn handle_cf_status(_cmd: &Value, state: &mut DaemonState) -> Result<Value
|
||||
let url = mgr.get_url().await.unwrap_or_default();
|
||||
|
||||
// 1. Is the page a Cloudflare challenge right now?
|
||||
let probe_raw = mgr.evaluate(CF_CHALLENGE_JS, None).await.unwrap_or(Value::Null);
|
||||
let probe_raw = mgr
|
||||
.evaluate(CF_CHALLENGE_JS, None)
|
||||
.await
|
||||
.unwrap_or(Value::Null);
|
||||
let probe = parse_json_string(probe_raw, "cf challenge probe").unwrap_or(Value::Null);
|
||||
let challenged = probe
|
||||
.get("challenged")
|
||||
@@ -4570,8 +5094,18 @@ async fn handle_keyboard(cmd: &Value, state: &DaemonState) -> Result<Value, Stri
|
||||
.get("text")
|
||||
.and_then(|v| v.as_str())
|
||||
.ok_or("Missing 'text' parameter")?;
|
||||
interaction::type_text_into_active_context(&mgr.client, &session_id, text, None)
|
||||
.await?;
|
||||
let key_events = cmd
|
||||
.get("keyEvents")
|
||||
.and_then(|v| v.as_bool())
|
||||
.unwrap_or(false);
|
||||
interaction::type_text_into_active_context(
|
||||
&mgr.client,
|
||||
&session_id,
|
||||
text,
|
||||
None,
|
||||
key_events,
|
||||
)
|
||||
.await?;
|
||||
return Ok(json!({ "typed": text }));
|
||||
}
|
||||
Some("insertText") => {
|
||||
@@ -4747,6 +5281,19 @@ async fn handle_tab_close(cmd: &Value, state: &mut DaemonState) -> Result<Value,
|
||||
|
||||
async fn handle_viewport(cmd: &Value, state: &mut DaemonState) -> Result<Value, String> {
|
||||
let mgr = state.browser.as_ref().ok_or("Browser not launched")?;
|
||||
|
||||
// `viewport reset` clears the device-metrics override and restores the real
|
||||
// layout viewport (the launched window's size, or — over the relay — the
|
||||
// user's actual Chrome window).
|
||||
if cmd.get("reset").and_then(|v| v.as_bool()).unwrap_or(false) {
|
||||
mgr.clear_viewport().await?;
|
||||
state.viewport = None;
|
||||
if let Some(ref server) = state.stream_server {
|
||||
server.set_viewport(1280, 720).await;
|
||||
}
|
||||
return Ok(json!({ "reset": true }));
|
||||
}
|
||||
|
||||
let width = cmd.get("width").and_then(|v| v.as_i64()).unwrap_or(1280) as i32;
|
||||
let height = cmd.get("height").and_then(|v| v.as_i64()).unwrap_or(720) as i32;
|
||||
let scale = cmd
|
||||
@@ -7100,6 +7647,28 @@ async fn handle_drag(cmd: &Value, state: &mut DaemonState) -> Result<Value, Stri
|
||||
.and_then(|v| v.as_str())
|
||||
.ok_or("Missing 'target' parameter")?;
|
||||
|
||||
// Over the relay (or into an iframe) a coordinate drag drifts to the
|
||||
// foreground tab and can't reach an OOPIF — DOM-dispatch an HTML5 drag in the
|
||||
// element's own session instead (issues #31/#36). `coord` mode forces the
|
||||
// coordinate path for pointer-driven drags (canvas/sliders) on a launched
|
||||
// browser.
|
||||
if std::env::var("AGENT_BROWSER_CLICK_MODE").as_deref() != Ok("coord")
|
||||
&& (crate::connect::relay_url().is_some()
|
||||
|| state.ref_map.ref_is_in_iframe(source)
|
||||
|| state.ref_map.ref_is_in_iframe(target))
|
||||
{
|
||||
super::interaction::dom_drag(
|
||||
&mgr.client,
|
||||
&session_id,
|
||||
&state.ref_map,
|
||||
source,
|
||||
target,
|
||||
&state.iframe_sessions,
|
||||
)
|
||||
.await?;
|
||||
return Ok(json!({ "dragged": { "source": source, "target": target }, "via": "dom" }));
|
||||
}
|
||||
|
||||
let (sx, sy, _, _, source_session_id) = super::element::resolve_element_center(
|
||||
&mgr.client,
|
||||
&session_id,
|
||||
|
||||
+576
-58
@@ -235,6 +235,43 @@ fn prunable_target_ids(
|
||||
.collect()
|
||||
}
|
||||
|
||||
/// Consecutive missing `getTargets` snapshots before an owned relay tab is
|
||||
/// pruned. >1 so a single churning/partial snapshot (other agents opening/closing
|
||||
/// tabs) or a brief cross-process-nav gap can't drop the tab the agent is driving.
|
||||
const RELAY_PRUNE_MISSES: u32 = 3;
|
||||
|
||||
/// Debounced prune for the relay: target ids to drop, mutating per-target miss
|
||||
/// counters. A tab in `live_ids` resets to 0; an absent (non-pinned) tab
|
||||
/// increments and is pruned only at `RELAY_PRUNE_MISSES`. Counters for
|
||||
/// no-longer-tracked targets are forgotten. Pure, so the multi-agent churn
|
||||
/// tolerance is unit-testable without a live browser.
|
||||
fn debounced_prune_ids(
|
||||
pages: &[PageInfo],
|
||||
live_ids: &HashSet<String>,
|
||||
pinned: Option<&str>,
|
||||
misses: &mut HashMap<String, u32>,
|
||||
) -> Vec<String> {
|
||||
let tracked: HashSet<&str> = pages.iter().map(|p| p.target_id.as_str()).collect();
|
||||
misses.retain(|tid, _| tracked.contains(tid.as_str()));
|
||||
let mut prune = Vec::new();
|
||||
for p in pages {
|
||||
let tid = p.target_id.as_str();
|
||||
if live_ids.contains(tid) {
|
||||
misses.remove(tid);
|
||||
continue;
|
||||
}
|
||||
if pinned == Some(tid) {
|
||||
continue;
|
||||
}
|
||||
let c = misses.entry(p.target_id.clone()).or_insert(0);
|
||||
*c += 1;
|
||||
if *c >= RELAY_PRUNE_MISSES {
|
||||
prune.push(p.target_id.clone());
|
||||
}
|
||||
}
|
||||
prune
|
||||
}
|
||||
|
||||
/// Whether the resolved active page is a tab the session created (its target_id
|
||||
/// is in `created_targets`). Pure core of [`BrowserManager::active_is_session_owned`]
|
||||
/// so the relay no-hijack rule is unit-testable without a live browser.
|
||||
@@ -254,6 +291,21 @@ fn active_index_is_owned(
|
||||
.unwrap_or(false)
|
||||
}
|
||||
|
||||
/// Whether a CDP error means the bound relay target is gone — the tab was
|
||||
/// closed, navigated across processes (renderer swap), or lost after an
|
||||
/// extension/service-worker restart, and the relay could not re-attach. The
|
||||
/// ab-connect relay surfaces these as `stale sessionId … its tab is gone`,
|
||||
/// `unknown sessionId …`, or `no attached tab …`. `navigate` keys its
|
||||
/// auto-reattach recovery off this (issue #35) so a dead session rebinds to a
|
||||
/// fresh tab instead of erroring on every command until the user runs `tab new`.
|
||||
fn is_stale_target_error(error: &str) -> bool {
|
||||
let lower = error.to_lowercase();
|
||||
lower.contains("its tab is gone")
|
||||
|| lower.contains("stale sessionid")
|
||||
|| lower.contains("unknown sessionid")
|
||||
|| lower.contains("no attached tab")
|
||||
}
|
||||
|
||||
/// Converts common error messages into AI-friendly, actionable descriptions.
|
||||
pub fn to_ai_friendly_error(error: &str) -> String {
|
||||
let lower = error.to_lowercase();
|
||||
@@ -449,6 +501,21 @@ pub struct BrowserManager {
|
||||
/// the session's commands onto the wrong page — the wrong-origin-fetch hazard
|
||||
/// in the dogfood reports. Falls back to the index if the pinned tab is gone.
|
||||
active_target_id: Option<String>,
|
||||
/// Per-target count of CONSECUTIVE `resync_targets` snapshots in which an
|
||||
/// owned tab was missing from `Target.getTargets`. Over the relay a single
|
||||
/// snapshot routinely omits live tabs (multi-agent churn, a cross-process nav
|
||||
/// briefly dropping the target), so we must not prune on one miss — that lost
|
||||
/// the tab the agent was driving. A tab is removed only after it's been absent
|
||||
/// for `RELAY_PRUNE_MISSES` consecutive snapshots; any snapshot that includes
|
||||
/// it resets the counter. Keyed by stable target_id.
|
||||
relay_target_misses: HashMap<String, u32>,
|
||||
/// Whether the relay accepted this session's group announcement and is
|
||||
/// therefore scoping `Target.getTargets` to our own tab group (issue #40).
|
||||
/// When true the daemon can safely adopt new targets again (follow-popup,
|
||||
/// cross-session adopt) — the relay has already filtered out foreign tabs.
|
||||
/// When false (launch-on-real-CDP, or an older relay that didn't answer the
|
||||
/// announce) the daemon keeps strict daemon-side isolation.
|
||||
relay_scoped: bool,
|
||||
next_tab_id: u32,
|
||||
/// Whether to enable the CDP `Runtime` domain (console / error / exception capture).
|
||||
/// OFF by default for stealth: a live `Runtime.enable` is a detectable CDP signal
|
||||
@@ -564,7 +631,10 @@ impl BrowserManager {
|
||||
crate::connect::log_connect_mode(
|
||||
&ws_url,
|
||||
true,
|
||||
DAEMON_SESSION.get().map(String::as_str).unwrap_or("default"),
|
||||
DAEMON_SESSION
|
||||
.get()
|
||||
.map(String::as_str)
|
||||
.unwrap_or("default"),
|
||||
);
|
||||
let manager = if engine == "lightpanda" {
|
||||
initialize_lightpanda_manager(ws_url, process).await?
|
||||
@@ -582,6 +652,8 @@ impl BrowserManager {
|
||||
visited_origins: HashSet::new(),
|
||||
created_targets: HashSet::new(),
|
||||
active_target_id: None,
|
||||
relay_target_misses: HashMap::new(),
|
||||
relay_scoped: false,
|
||||
next_tab_id: 1,
|
||||
capture_console: console_capture_enabled(),
|
||||
};
|
||||
@@ -666,7 +738,10 @@ impl BrowserManager {
|
||||
crate::connect::log_connect_mode(
|
||||
&ws_url,
|
||||
false,
|
||||
DAEMON_SESSION.get().map(String::as_str).unwrap_or("default"),
|
||||
DAEMON_SESSION
|
||||
.get()
|
||||
.map(String::as_str)
|
||||
.unwrap_or("default"),
|
||||
);
|
||||
let client = Arc::new(CdpClient::connect_with_headers(&ws_url, headers).await?);
|
||||
let mut manager = Self {
|
||||
@@ -681,6 +756,8 @@ impl BrowserManager {
|
||||
visited_origins: HashSet::new(),
|
||||
created_targets: HashSet::new(),
|
||||
active_target_id: None,
|
||||
relay_target_misses: HashMap::new(),
|
||||
relay_scoped: false,
|
||||
next_tab_id: 1,
|
||||
capture_console: console_capture_enabled(),
|
||||
};
|
||||
@@ -710,6 +787,143 @@ impl BrowserManager {
|
||||
Self::connect_cdp(&ws_url).await
|
||||
}
|
||||
|
||||
/// Page targets to adopt, merging several `Target.getTargets` snapshots over
|
||||
/// the extension relay. A single relay snapshot is flaky on a busy real Chrome
|
||||
/// — it can omit live tabs (a different window's set, or a partial list; issue
|
||||
/// #31) — so a tab the daemon should adopt would silently vanish (e.g. after a
|
||||
/// daemon restart the page being driven disappeared from the tab list). Taking
|
||||
/// the union of a few snapshots makes adoption resilient to a transient miss.
|
||||
/// Off the relay (a browser we launched) one snapshot is authoritative.
|
||||
async fn collect_page_targets(&self) -> Result<Vec<TargetInfo>, String> {
|
||||
let rounds = if crate::connect::relay_url().is_some() {
|
||||
3
|
||||
} else {
|
||||
1
|
||||
};
|
||||
let mut by_id: HashMap<String, TargetInfo> = HashMap::new();
|
||||
let mut any_ok = false;
|
||||
for i in 0..rounds {
|
||||
if i > 0 {
|
||||
tokio::time::sleep(Duration::from_millis(150)).await;
|
||||
}
|
||||
match self
|
||||
.client
|
||||
.send_command_typed::<_, GetTargetsResult>("Target.getTargets", &json!({}), None)
|
||||
.await
|
||||
{
|
||||
Ok(result) => {
|
||||
any_ok = true;
|
||||
for t in result.target_infos.into_iter().filter(should_track_target) {
|
||||
by_id.entry(t.target_id.clone()).or_insert(t);
|
||||
}
|
||||
}
|
||||
Err(e) if i == rounds - 1 && !any_ok => return Err(e),
|
||||
Err(_) => {}
|
||||
}
|
||||
}
|
||||
Ok(by_id.into_values().collect())
|
||||
}
|
||||
|
||||
/// Every tab the relay knows, UNSCOPED (ignores group scoping) — for explicit
|
||||
/// cross-group adoption (`chrome-use adopt`). Falls back to the scoped
|
||||
/// `collect_page_targets` on a relay/browser that doesn't support the
|
||||
/// unscoped query. Retries a few times over the relay (discovery is eventual).
|
||||
async fn collect_all_targets(&self) -> Result<Vec<TargetInfo>, String> {
|
||||
let rounds = if crate::connect::relay_url().is_some() {
|
||||
3
|
||||
} else {
|
||||
1
|
||||
};
|
||||
let mut by_id: HashMap<String, TargetInfo> = HashMap::new();
|
||||
let mut any_ok = false;
|
||||
for i in 0..rounds {
|
||||
if i > 0 {
|
||||
tokio::time::sleep(Duration::from_millis(150)).await;
|
||||
}
|
||||
if let Ok(result) = self
|
||||
.client
|
||||
.send_command_typed::<_, GetTargetsResult>(
|
||||
"ABRelay.getAllTargets",
|
||||
&json!({}),
|
||||
None,
|
||||
)
|
||||
.await
|
||||
{
|
||||
any_ok = true;
|
||||
for t in result.target_infos.into_iter().filter(should_track_target) {
|
||||
by_id.entry(t.target_id.clone()).or_insert(t);
|
||||
}
|
||||
}
|
||||
}
|
||||
if any_ok {
|
||||
Ok(by_id.into_values().collect())
|
||||
} else {
|
||||
// Older relay without ABRelay.getAllTargets → best-effort scoped list.
|
||||
self.collect_page_targets().await
|
||||
}
|
||||
}
|
||||
|
||||
/// Adopt a specific pre-existing tab matched by `spec` (an exact CDP
|
||||
/// `targetId`, or a case-insensitive substring of the tab URL) WITHOUT opening
|
||||
/// a new tab — for `chrome-use adopt`. Attaches it (the relay tags it into our
|
||||
/// group), tracks + pins it. Errors if nothing matches (never creates a tab).
|
||||
async fn adopt_existing_target(&mut self, spec: &str) -> Result<(), String> {
|
||||
let all = self.collect_all_targets().await?;
|
||||
let spec_l = spec.to_lowercase();
|
||||
let target = all
|
||||
.iter()
|
||||
.find(|t| t.target_id == spec)
|
||||
.or_else(|| all.iter().find(|t| t.url.to_lowercase().contains(&spec_l)))
|
||||
.ok_or_else(|| {
|
||||
let mut open: Vec<String> = all
|
||||
.iter()
|
||||
.map(|t| {
|
||||
let u = if t.url.len() > 80 {
|
||||
&t.url[..80]
|
||||
} else {
|
||||
&t.url
|
||||
};
|
||||
u.to_string()
|
||||
})
|
||||
.collect();
|
||||
open.sort();
|
||||
open.dedup();
|
||||
format!(
|
||||
"adopt: no open tab matching `{spec}` (by targetId or URL substring).\n\
|
||||
{} tab(s) the extension can see:\n {}",
|
||||
open.len(),
|
||||
open.join("\n ")
|
||||
)
|
||||
})?
|
||||
.clone();
|
||||
|
||||
let attach: AttachToTargetResult = self
|
||||
.client
|
||||
.send_command_typed(
|
||||
"Target.attachToTarget",
|
||||
&AttachToTargetParams {
|
||||
target_id: target.target_id.clone(),
|
||||
flatten: true,
|
||||
},
|
||||
None,
|
||||
)
|
||||
.await?;
|
||||
let tab_id = self.assign_tab_id();
|
||||
self.pages.push(PageInfo {
|
||||
tab_id,
|
||||
label: None,
|
||||
target_id: target.target_id.clone(),
|
||||
session_id: attach.session_id.clone(),
|
||||
url: target.url.clone(),
|
||||
title: sanitize_title(&target.title),
|
||||
target_type: target.target_type.clone(),
|
||||
});
|
||||
self.active_page_index = self.pages.len() - 1;
|
||||
self.pin_active_target();
|
||||
self.enable_domains(&attach.session_id).await?;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
async fn discover_and_attach_targets(&mut self) -> Result<(), String> {
|
||||
self.client
|
||||
.send_command_typed::<_, Value>(
|
||||
@@ -719,16 +933,22 @@ impl BrowserManager {
|
||||
)
|
||||
.await?;
|
||||
|
||||
let result: GetTargetsResult = self
|
||||
.client
|
||||
.send_command_typed("Target.getTargets", &json!({}), None)
|
||||
.await?;
|
||||
// Announce our group FIRST so the relay scopes the getTargets below to our
|
||||
// own tab group (issue #40). On a launched browser this is a no-op.
|
||||
let scoped = self.announce_group().await;
|
||||
|
||||
let page_targets: Vec<TargetInfo> = result
|
||||
.target_infos
|
||||
.into_iter()
|
||||
.filter(should_track_target)
|
||||
.collect();
|
||||
// `chrome-use adopt <spec>`: adopt a specific PRE-EXISTING tab instead of
|
||||
// creating one — true zero-new-tab reading of the user's own tab. The
|
||||
// directive rides in via env so it takes effect at first connect (before
|
||||
// any about:blank would be made). If nothing matches, error out rather
|
||||
// than fall back to creating a tab.
|
||||
if let Ok(spec) = std::env::var("AGENT_BROWSER_ADOPT") {
|
||||
if !spec.trim().is_empty() {
|
||||
return self.adopt_existing_target(spec.trim()).await;
|
||||
}
|
||||
}
|
||||
|
||||
let page_targets: Vec<TargetInfo> = self.collect_page_targets().await?;
|
||||
|
||||
if page_targets.is_empty() {
|
||||
// Create a new tab
|
||||
@@ -774,7 +994,19 @@ impl BrowserManager {
|
||||
self.active_page_index = 0;
|
||||
self.pin_active_target();
|
||||
self.enable_domains(&attach_result.session_id).await?;
|
||||
} else if self.agent_group().is_some() && !scoped {
|
||||
// STRICT MULTI-AGENT ISOLATION fallback (relay, but the group announce
|
||||
// didn't take — e.g. an older relay). Without relay-side scoping,
|
||||
// `page_targets` could be the USER's and OTHER agents' tabs, so this
|
||||
// session must NOT adopt any of them — adopting foreign tabs is what let
|
||||
// another agent's tab churn drop the tab we were driving (multi-agent
|
||||
// failure). Open our own dedicated background tab and pin it instead.
|
||||
self.tab_new(None, None).await?;
|
||||
} else {
|
||||
// Either a browser WE launched (every tab is ours) or the relay has
|
||||
// scoped getTargets to our own tab group (#40) — so `page_targets` are
|
||||
// all ours: adopt them (this restores follow-popup + cross-session
|
||||
// adopt under isolation, since foreign tabs were already filtered out).
|
||||
for target in &page_targets {
|
||||
let attach_result: AttachToTargetResult = self
|
||||
.client
|
||||
@@ -800,7 +1032,6 @@ impl BrowserManager {
|
||||
target_type: target.target_type.clone(),
|
||||
});
|
||||
}
|
||||
|
||||
self.active_page_index = 0;
|
||||
self.pin_active_target();
|
||||
let session_id = self.pages[0].session_id.clone();
|
||||
@@ -915,6 +1146,24 @@ impl BrowserManager {
|
||||
)
|
||||
}
|
||||
|
||||
/// Drop the page bound to `session_id` from the tracked list — used when the
|
||||
/// relay reports its tab is gone (issue #35) so the stale entry can't keep
|
||||
/// resolving as active. Forgets ownership, unpins it if it was pinned, and
|
||||
/// keeps `active_page_index` in range.
|
||||
fn drop_page_by_session(&mut self, session_id: &str) {
|
||||
let Some(pos) = self.pages.iter().position(|p| p.session_id == session_id) else {
|
||||
return;
|
||||
};
|
||||
let target_id = self.pages[pos].target_id.clone();
|
||||
self.pages.remove(pos);
|
||||
self.created_targets.remove(&target_id);
|
||||
if self.active_target_id.as_deref() == Some(target_id.as_str()) {
|
||||
self.active_target_id = None;
|
||||
}
|
||||
self.active_page_index =
|
||||
active_page_index_after_removal(self.active_page_index, pos, self.pages.len());
|
||||
}
|
||||
|
||||
/// Pin the current active page by target_id so later commands stick to it.
|
||||
/// Call after any explicit open / tab new / tab switch.
|
||||
fn pin_active_target(&mut self) {
|
||||
@@ -944,10 +1193,10 @@ impl BrowserManager {
|
||||
if self.agent_group().is_some() && !self.active_is_session_owned() {
|
||||
self.tab_new(None, None).await?;
|
||||
}
|
||||
let session_id = self.active_session_id()?.to_string();
|
||||
let mut session_id = self.active_session_id()?.to_string();
|
||||
let mut lifecycle_rx = self.client.subscribe();
|
||||
|
||||
let nav_result: PageNavigateResult = self
|
||||
let nav_result: PageNavigateResult = match self
|
||||
.client
|
||||
.send_command_typed(
|
||||
"Page.navigate",
|
||||
@@ -957,9 +1206,51 @@ impl BrowserManager {
|
||||
},
|
||||
Some(&session_id),
|
||||
)
|
||||
.await?;
|
||||
.await
|
||||
{
|
||||
Ok(r) => r,
|
||||
// Auto-reattach when the bound tab is gone (issue #35). On the shared
|
||||
// real browser the human can close/swap the agent's tab, and a
|
||||
// cross-process nav can destroy the target without a re-attachable
|
||||
// tabId — both leave the cached `cb-tab-<id>` session stale, so every
|
||||
// command (including `open`) failed on it and only `tab new`
|
||||
// recovered. The relay error literally says "re-open your target URL
|
||||
// to re-attach"; fulfil that here: drop the dead page, open a fresh
|
||||
// owned tab in this session's group, and navigate THAT. Gated on the
|
||||
// relay (`agent_group`) and on the explicit navigation intent — read
|
||||
// commands deliberately still fail loudly rather than silently
|
||||
// recover onto a blank tab and return wrong data (issue #8.1).
|
||||
Err(e) if self.agent_group().is_some() && is_stale_target_error(&e) => {
|
||||
self.drop_page_by_session(&session_id);
|
||||
self.tab_new(None, None).await?;
|
||||
session_id = self.active_session_id()?.to_string();
|
||||
lifecycle_rx = self.client.subscribe();
|
||||
self.client
|
||||
.send_command_typed(
|
||||
"Page.navigate",
|
||||
&PageNavigateParams {
|
||||
url: url.to_string(),
|
||||
referrer: None,
|
||||
},
|
||||
Some(&session_id),
|
||||
)
|
||||
.await?
|
||||
}
|
||||
Err(e) => return Err(e),
|
||||
};
|
||||
|
||||
if let Some(ref error_text) = nav_result.error_text {
|
||||
// `data:` URLs abort over the extension relay: chrome.debugger /
|
||||
// chrome.tabs can't drive a top-frame data: navigation, so it comes
|
||||
// back net::ERR_ABORTED on an about:blank tab. Explain it instead of
|
||||
// leaking the cryptic code (data: works fine under `--launch`).
|
||||
if url.starts_with("data:") && error_text.contains("ERR_ABORTED") {
|
||||
return Err(format!(
|
||||
"Navigation failed: {error_text}. Chrome blocks top-frame `data:` URLs over \
|
||||
the extension relay — use a real http(s):// or file:// URL, or run with \
|
||||
`--launch` (where data: URLs work)."
|
||||
));
|
||||
}
|
||||
return Err(format!("Navigation failed: {}", error_text));
|
||||
}
|
||||
|
||||
@@ -1097,15 +1388,31 @@ impl BrowserManager {
|
||||
pub async fn evaluate(&self, script: &str, _args: Option<Value>) -> Result<Value, String> {
|
||||
let session_id = self.active_session_id()?.to_string();
|
||||
|
||||
// `replMode: true` lets successive `eval`s re-declare top-level
|
||||
// `let`/`const` instead of throwing "Identifier 'x' has already been
|
||||
// declared" (issue #38 — independent `eval` steps in a test suite collided
|
||||
// in the page's shared lexical scope). BUT replMode and `awaitPromise` are
|
||||
// mutually exclusive in Chrome: under replMode a returned promise is NOT
|
||||
// awaited (it serialises to `{}`), which breaks `fetch(...).then(...)` and
|
||||
// every other async eval. So enable replMode ONLY for synchronous scripts
|
||||
// that declare a top-level `let`/`const`; promise-returning scripts keep
|
||||
// `awaitPromise` (no replMode) — exactly the pre-#38 behaviour.
|
||||
let mentions_async = script.contains("await")
|
||||
|| script.contains(".then(")
|
||||
|| script.contains("fetch(")
|
||||
|| script.contains("Promise");
|
||||
let declares = script.contains("let ") || script.contains("const ");
|
||||
let repl_mode = declares && !mentions_async;
|
||||
let result: EvaluateResult = self
|
||||
.client
|
||||
.send_command_typed(
|
||||
"Runtime.evaluate",
|
||||
&EvaluateParams {
|
||||
expression: script.to_string(),
|
||||
return_by_value: Some(true),
|
||||
await_promise: Some(true),
|
||||
},
|
||||
&json!({
|
||||
"expression": script,
|
||||
"returnByValue": true,
|
||||
"awaitPromise": !repl_mode,
|
||||
"replMode": repl_mode,
|
||||
}),
|
||||
Some(&session_id),
|
||||
)
|
||||
.await?;
|
||||
@@ -1239,6 +1546,13 @@ impl BrowserManager {
|
||||
.ok_or_else(|| "No active page".to_string())
|
||||
}
|
||||
|
||||
/// Stop owning a tab — drop it from `created_targets` so it survives `close()`
|
||||
/// and idle-shutdown (the agent is leaving it for the user). Returns true if it
|
||||
/// was owned. Used by `keep`.
|
||||
pub fn unown_target(&mut self, target_id: &str) -> bool {
|
||||
self.created_targets.remove(target_id)
|
||||
}
|
||||
|
||||
/// Returns true if this manager was connected via CDP (as opposed to local launch).
|
||||
pub fn is_cdp_connection(&self) -> bool {
|
||||
self.browser_process.is_none()
|
||||
@@ -1388,6 +1702,22 @@ impl BrowserManager {
|
||||
/// the active tab, per #7/#8.1); the caller surfaces it so the agent knows a
|
||||
/// tab opened instead of seeing the old page (issue #24-A).
|
||||
pub async fn adopt_newly_opened(&mut self, before: &HashSet<String>) -> Option<PageInfo> {
|
||||
// STRICT MULTI-AGENT ISOLATION: on the relay this session's `before` set is
|
||||
// only its OWN tabs, so EVERY foreign tab (the user's, other agents') looks
|
||||
// "new" relative to it and would be adopted here — exactly the leak where a
|
||||
// concurrent agent's tabs (github/Lark/iphone-use) showed up in this
|
||||
// session mid-flow. A tab the agent itself opened (a pop-up) can't be
|
||||
// distinguished from a foreign tab over the relay (no opener/window/group
|
||||
// in the synthesized targetInfo), so don't adopt anything: the agent drives
|
||||
// only tabs it explicitly created, and pop-ups (e.g. an OAuth/login window)
|
||||
// are the user's. A launched browser (every tab ours) still follows pop-ups.
|
||||
// Strict isolation only when on the relay WITHOUT group scoping: there a
|
||||
// pop-up can't be told apart from a foreign tab, so adopt nothing. When the
|
||||
// relay IS scoping (#40), getTargets returns only our group, so a tab that
|
||||
// appeared after our own action is genuinely ours (a pop-up) — adopt it.
|
||||
if self.agent_group().is_some() && !self.relay_scoped {
|
||||
return None;
|
||||
}
|
||||
let result: GetTargetsResult = self
|
||||
.client
|
||||
.send_command_typed("Target.getTargets", &json!({}), None)
|
||||
@@ -1430,6 +1760,11 @@ impl BrowserManager {
|
||||
title: sanitize_title(&target.title),
|
||||
target_type: target.target_type.clone(),
|
||||
};
|
||||
// A tab that appeared right after THIS session's action (a click that
|
||||
// opened a popup/new tab) is ours — record it as owned so it's tracked,
|
||||
// protected from churn-pruning, and cleaned up on close, consistent with
|
||||
// strict multi-agent isolation (we only ever own tabs we created/opened).
|
||||
self.created_targets.insert(target.target_id.clone());
|
||||
self.add_background_page(page.clone());
|
||||
let _ = self.enable_domains(&attach.session_id).await;
|
||||
if opened.is_none() {
|
||||
@@ -1457,13 +1792,20 @@ impl BrowserManager {
|
||||
.filter(should_track_target)
|
||||
.collect();
|
||||
let live_ids: HashSet<String> = live.iter().map(|t| t.target_id.clone()).collect();
|
||||
let on_relay = self.agent_group().is_some();
|
||||
// When the relay scopes getTargets to our group (#40), `live` is already
|
||||
// only our own tabs, so adopting unknown ones is safe (a freshly-opened
|
||||
// pop-up). Without scoping, keep strict isolation: never adopt a tab we
|
||||
// didn't create — it belongs to the user or another agent.
|
||||
let strict_isolation = on_relay && !self.relay_scoped;
|
||||
|
||||
for target in &live {
|
||||
if self.update_page_target_info(target) {
|
||||
continue;
|
||||
}
|
||||
// A target this session hasn't tracked yet — attach and add it in the
|
||||
// background so it's listable/adoptable without stealing the active tab.
|
||||
if strict_isolation {
|
||||
continue;
|
||||
}
|
||||
let attach_result: AttachToTargetResult = match self
|
||||
.client
|
||||
.send_command_typed(
|
||||
@@ -1494,12 +1836,26 @@ impl BrowserManager {
|
||||
let _ = self.enable_domains(&attach_result.session_id).await;
|
||||
}
|
||||
|
||||
// Drop tabs that no longer exist so `tab list` doesn't show phantom rows —
|
||||
// but never prune the explicitly-pinned active target on a transient
|
||||
// getTargets snapshot (issue #31; see `prunable_target_ids`).
|
||||
let gone = prunable_target_ids(&self.pages, &live_ids, self.active_target_id.as_deref());
|
||||
for tid in gone {
|
||||
self.remove_page_by_target_id(&tid);
|
||||
// Prune tabs that are gone. On a LAUNCHED browser a missing target really
|
||||
// is closed, so prune immediately. On the RELAY a single `getTargets`
|
||||
// snapshot routinely omits live tabs (multi-agent churn, a brief
|
||||
// cross-process-nav gap) — dropping the tab we're driving on one bad
|
||||
// snapshot is the failure we're fixing — so prune only after the tab has
|
||||
// been absent for several CONSECUTIVE snapshots (debounced). The pinned
|
||||
// active target is protected either way (issue #31).
|
||||
let gone = if on_relay {
|
||||
debounced_prune_ids(
|
||||
&self.pages,
|
||||
&live_ids,
|
||||
self.active_target_id.as_deref(),
|
||||
&mut self.relay_target_misses,
|
||||
)
|
||||
} else {
|
||||
prunable_target_ids(&self.pages, &live_ids, self.active_target_id.as_deref())
|
||||
};
|
||||
for tid in &gone {
|
||||
self.relay_target_misses.remove(tid);
|
||||
self.remove_page_by_target_id(tid);
|
||||
}
|
||||
|
||||
// Refresh url/title from each live tab. The relay only stamps target_info
|
||||
@@ -1601,9 +1957,17 @@ impl BrowserManager {
|
||||
/// CDP browser the endpoint is strict, so we must NOT send the custom param —
|
||||
/// hence `None` there. We detect the relay by matching our `ws_url` against
|
||||
/// the live relay URL the native-messaging host published.
|
||||
/// Whether this manager is driving the user's real Chrome through the
|
||||
/// `ab-connect` extension relay (vs. a browser we launched or a direct CDP
|
||||
/// endpoint). Detected by matching our `ws_url` against the live relay URL
|
||||
/// the native-messaging host published. Used to avoid relay-unsafe CDP that
|
||||
/// would disturb the user's window (e.g. Browser.setContentsSize, issue #47).
|
||||
fn via_relay(&self) -> bool {
|
||||
crate::connect::relay_url().as_deref() == Some(self.ws_url.as_str())
|
||||
}
|
||||
|
||||
fn agent_group(&self) -> Option<String> {
|
||||
let via_relay = crate::connect::relay_url().as_deref() == Some(self.ws_url.as_str());
|
||||
if !via_relay {
|
||||
if !self.via_relay() {
|
||||
return None;
|
||||
}
|
||||
let name = DAEMON_SESSION
|
||||
@@ -1617,6 +1981,25 @@ impl BrowserManager {
|
||||
}
|
||||
}
|
||||
|
||||
/// Tell the relay which tab group this session owns so it can scope
|
||||
/// `Target.getTargets` to us (issue #40). Only meaningful on the relay; a
|
||||
/// no-op (returns false) on a launched/real-CDP connection. Sets and returns
|
||||
/// `relay_scoped`: when true, the daemon can trust getTargets to contain only
|
||||
/// our group and re-enable adopting new tabs (pop-ups, cross-session adopt).
|
||||
async fn announce_group(&mut self) -> bool {
|
||||
let Some(group) = self.agent_group() else {
|
||||
self.relay_scoped = false;
|
||||
return false;
|
||||
};
|
||||
let ok = self
|
||||
.client
|
||||
.send_command_typed::<_, Value>("ABRelay.setGroup", &json!({ "group": group }), None)
|
||||
.await
|
||||
.is_ok();
|
||||
self.relay_scoped = ok;
|
||||
ok
|
||||
}
|
||||
|
||||
pub async fn tab_new(
|
||||
&mut self,
|
||||
url: Option<&str>,
|
||||
@@ -1687,6 +2070,46 @@ impl BrowserManager {
|
||||
self.active_page_index = index;
|
||||
self.pin_active_target();
|
||||
|
||||
// Close the daemon's leftover initial `about:blank` scratch tab once this
|
||||
// real tab exists, so the session's tab group isn't left showing a stray
|
||||
// blank page beside the work tab (every group otherwise carried one). Only
|
||||
// on the RELAY — there the about:blank is a tab WE created as scratch; on a
|
||||
// launched browser the initial about:blank is the browser's own first tab,
|
||||
// which we must not close. Only when opening a real url, OWNED, still-blank.
|
||||
if target_url != "about:blank" && self.agent_group().is_some() {
|
||||
if let Some(new_tid) = self.pages.get(index).map(|p| p.target_id.clone()) {
|
||||
let blanks: Vec<String> = self
|
||||
.pages
|
||||
.iter()
|
||||
.filter(|p| {
|
||||
p.target_id != new_tid
|
||||
&& self.created_targets.contains(&p.target_id)
|
||||
&& (p.url == "about:blank" || p.url.is_empty())
|
||||
})
|
||||
.map(|p| p.target_id.clone())
|
||||
.collect();
|
||||
for tid in blanks {
|
||||
let _ = self
|
||||
.client
|
||||
.send_command_typed::<_, Value>(
|
||||
"Target.closeTarget",
|
||||
&CloseTargetParams {
|
||||
target_id: tid.clone(),
|
||||
},
|
||||
None,
|
||||
)
|
||||
.await;
|
||||
self.created_targets.remove(&tid);
|
||||
self.remove_page_by_target_id(&tid);
|
||||
}
|
||||
// Removing earlier pages shifts indices — re-pin the new tab.
|
||||
if let Some(i) = self.pages.iter().position(|p| p.target_id == new_tid) {
|
||||
self.active_page_index = i;
|
||||
self.pin_active_target();
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
Ok(json!({
|
||||
"tabId": format_tab_id(tab_id),
|
||||
"label": label,
|
||||
@@ -1793,32 +2216,39 @@ impl BrowserManager {
|
||||
.await?;
|
||||
|
||||
// Screencast captures the actual content area, not the emulated CSS
|
||||
// viewport, so resize the content area to match.
|
||||
if let Ok(target_id) = self.active_target_id() {
|
||||
if let Ok(window_info) = self
|
||||
.client
|
||||
.send_command(
|
||||
"Browser.getWindowForTarget",
|
||||
Some(json!({ "targetId": target_id })),
|
||||
None,
|
||||
)
|
||||
.await
|
||||
{
|
||||
if let Some(window_id) = window_info.get("windowId").and_then(|v| v.as_i64()) {
|
||||
if let Err(e) = self
|
||||
.client
|
||||
.send_command(
|
||||
"Browser.setContentsSize",
|
||||
Some(json!({
|
||||
"windowId": window_id,
|
||||
"width": width,
|
||||
"height": height,
|
||||
})),
|
||||
None,
|
||||
)
|
||||
.await
|
||||
{
|
||||
eprintln!("Browser.setContentsSize failed (experimental CDP): {e}");
|
||||
// viewport, so resize the content area to match — but ONLY for a browser
|
||||
// we launched. Over the ab-connect relay the "window" is the user's real
|
||||
// Chrome window, and Browser.setContentsSize would physically resize it
|
||||
// (issue #47) — the exact thing the CDP device-metrics override exists to
|
||||
// avoid. The Emulation override above already gives the tab the requested
|
||||
// CSS viewport without touching the OS window, so skip the resize there.
|
||||
if !self.via_relay() {
|
||||
if let Ok(target_id) = self.active_target_id() {
|
||||
if let Ok(window_info) = self
|
||||
.client
|
||||
.send_command(
|
||||
"Browser.getWindowForTarget",
|
||||
Some(json!({ "targetId": target_id })),
|
||||
None,
|
||||
)
|
||||
.await
|
||||
{
|
||||
if let Some(window_id) = window_info.get("windowId").and_then(|v| v.as_i64()) {
|
||||
if let Err(e) = self
|
||||
.client
|
||||
.send_command(
|
||||
"Browser.setContentsSize",
|
||||
Some(json!({
|
||||
"windowId": window_id,
|
||||
"width": width,
|
||||
"height": height,
|
||||
})),
|
||||
None,
|
||||
)
|
||||
.await
|
||||
{
|
||||
eprintln!("Browser.setContentsSize failed (experimental CDP): {e}");
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -1827,6 +2257,22 @@ impl BrowserManager {
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Clear the CDP device-metrics override (`viewport reset`), restoring the
|
||||
/// tab's real layout viewport. Never touches the OS window, so it is safe on
|
||||
/// the relay (we never physically resized the user's window — see
|
||||
/// `set_viewport`).
|
||||
pub async fn clear_viewport(&self) -> Result<(), String> {
|
||||
let session_id = self.active_session_id()?;
|
||||
self.client
|
||||
.send_command(
|
||||
"Emulation.clearDeviceMetricsOverride",
|
||||
Some(json!({})),
|
||||
Some(session_id),
|
||||
)
|
||||
.await?;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
pub async fn set_user_agent(&self, user_agent: &str) -> Result<(), String> {
|
||||
let session_id = self.active_session_id()?;
|
||||
self.client
|
||||
@@ -2409,6 +2855,8 @@ async fn initialize_lightpanda_manager(
|
||||
visited_origins: HashSet::new(),
|
||||
created_targets: HashSet::new(),
|
||||
active_target_id: None,
|
||||
relay_target_misses: HashMap::new(),
|
||||
relay_scoped: false,
|
||||
next_tab_id: 1,
|
||||
capture_console: console_capture_enabled(),
|
||||
};
|
||||
@@ -2693,6 +3141,33 @@ mod tests {
|
||||
assert_eq!(active_page_index_after_removal(0, 0, 0), 0);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn stale_target_error_matches_relay_signatures() {
|
||||
// The exact relay error `open` must recover from (issue #35), as wrapped
|
||||
// by send_command's `CDP error (Page.navigate): …` prefix.
|
||||
assert!(is_stale_target_error(
|
||||
"CDP error (Page.navigate): stale sessionId cb-tab-1655244623 for Page.navigate: \
|
||||
its tab is gone (closed, navigated across processes, or lost after an extension \
|
||||
restart). Re-attach by re-opening your target URL before retrying."
|
||||
));
|
||||
assert!(is_stale_target_error(
|
||||
"unknown sessionId cb-tab-7 for Page.navigate"
|
||||
));
|
||||
assert!(is_stale_target_error("no attached tab for Page.navigate"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn stale_target_error_ignores_unrelated_failures() {
|
||||
// A genuine navigation failure (bad URL, DNS, blocked) must NOT trigger
|
||||
// the open-a-fresh-tab recovery — that would mask the real error.
|
||||
assert!(!is_stale_target_error(
|
||||
"Navigation failed: net::ERR_NAME_NOT_RESOLVED"
|
||||
));
|
||||
assert!(!is_stale_target_error(
|
||||
"CDP command timed out: Page.navigate"
|
||||
));
|
||||
}
|
||||
|
||||
fn page(target_id: &str) -> PageInfo {
|
||||
PageInfo {
|
||||
tab_id: 1,
|
||||
@@ -2797,7 +3272,10 @@ mod tests {
|
||||
let dirty = "\u{200d}\u{2061}\u{200d}\u{2063}\u{200b}\u{2062}\u{feff}GitHub";
|
||||
assert_eq!(sanitize_title(dirty), "GitHub");
|
||||
// Clean titles (incl. CJK + normal punctuation) pass through untouched.
|
||||
assert_eq!(sanitize_title("購入手続きへ - メルカリ"), "購入手続きへ - メルカリ");
|
||||
assert_eq!(
|
||||
sanitize_title("購入手続きへ - メルカリ"),
|
||||
"購入手続きへ - メルカリ"
|
||||
);
|
||||
assert_eq!(sanitize_title(" Hello World "), "Hello World");
|
||||
// Emoji and real content survive; only the invisibles are dropped.
|
||||
assert_eq!(sanitize_title("✓ Done\u{200b}"), "✓ Done");
|
||||
@@ -2829,7 +3307,47 @@ mod tests {
|
||||
// A pinned target that IS in the live set is simply not prunable anyway.
|
||||
let mut live2 = HashSet::new();
|
||||
live2.insert("A".to_string());
|
||||
assert_eq!(prunable_target_ids(&pages, &live2, Some("A")), vec!["B".to_string()]);
|
||||
assert_eq!(
|
||||
prunable_target_ids(&pages, &live2, Some("A")),
|
||||
vec!["B".to_string()]
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn debounced_prune_tolerates_transient_churn() {
|
||||
// Multi-agent churn: a single getTargets snapshot omits our owned tab "B"
|
||||
// (another agent opened/closed tabs). It must NOT be pruned on one miss.
|
||||
let pages = vec![page("A"), page("B")];
|
||||
let mut misses = HashMap::new();
|
||||
let empty: HashSet<String> = HashSet::new();
|
||||
// Misses 1 and 2: B absent but under threshold → not pruned.
|
||||
assert!(debounced_prune_ids(&pages, &empty, Some("A"), &mut misses).is_empty());
|
||||
assert!(debounced_prune_ids(&pages, &empty, Some("A"), &mut misses).is_empty());
|
||||
// Miss 3 (== RELAY_PRUNE_MISSES): genuinely gone → pruned.
|
||||
assert_eq!(
|
||||
debounced_prune_ids(&pages, &empty, Some("A"), &mut misses),
|
||||
vec!["B".to_string()]
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn debounced_prune_resets_on_reappearance_and_protects_pin() {
|
||||
let pages = vec![page("A"), page("B")];
|
||||
let mut misses = HashMap::new();
|
||||
let empty: HashSet<String> = HashSet::new();
|
||||
let mut live_b: HashSet<String> = HashSet::new();
|
||||
live_b.insert("B".to_string());
|
||||
// Two misses for B, then it reappears → counter resets, so it survives
|
||||
// indefinitely under intermittent churn.
|
||||
debounced_prune_ids(&pages, &empty, Some("A"), &mut misses);
|
||||
debounced_prune_ids(&pages, &empty, Some("A"), &mut misses);
|
||||
assert!(debounced_prune_ids(&pages, &live_b, Some("A"), &mut misses).is_empty());
|
||||
assert!(debounced_prune_ids(&pages, &empty, Some("A"), &mut misses).is_empty()); // back to miss 1
|
||||
// The pinned active "A" is never pruned no matter how many misses.
|
||||
for _ in 0..5 {
|
||||
let gone = debounced_prune_ids(&pages, &empty, Some("A"), &mut misses);
|
||||
assert!(!gone.contains(&"A".to_string()));
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
|
||||
@@ -21,6 +21,16 @@ pub async fn run_daemon(session: &str) {
|
||||
// (via the ab-connect extension) land in a per-session Chrome tab group.
|
||||
let _ = super::browser::DAEMON_SESSION.set(session.to_string());
|
||||
|
||||
// Bootstrap / refresh the site-adapter pack in the background (first-run +
|
||||
// periodic TTL). This populates ~/.chrome-use/sites/.index.json so navigation
|
||||
// can auto-suggest `site` commands for the page you land on, with zero added
|
||||
// latency to any command. Best-effort; offline is a no-op.
|
||||
if crate::site::needs_refresh() {
|
||||
tokio::spawn(async {
|
||||
let _ = crate::site::update().await;
|
||||
});
|
||||
}
|
||||
|
||||
let socket_dir = get_daemon_socket_dir();
|
||||
if !socket_dir.exists() {
|
||||
let _ = fs::create_dir_all(&socket_dir);
|
||||
@@ -120,12 +130,21 @@ pub async fn run_daemon(session: &str) {
|
||||
}
|
||||
}
|
||||
|
||||
// Auto-shutdown the daemon after this many ms of inactivity (no commands received).
|
||||
// Disabled when unset or 0.
|
||||
let idle_timeout_ms = env::var("AGENT_BROWSER_IDLE_TIMEOUT_MS")
|
||||
.ok()
|
||||
.and_then(|s| s.parse::<u64>().ok())
|
||||
.filter(|&ms| ms > 0);
|
||||
// Auto-shutdown the daemon after this many ms of inactivity (no commands
|
||||
// received). On shutdown the daemon closes the tabs IT created (its per-session
|
||||
// tab group), so an agent that finishes a task and just stops — without ever
|
||||
// calling `close` — no longer leaves a pile of scratch tabs and a lingering
|
||||
// tab group in the user's Chrome. The timer resets on every command, so active
|
||||
// sessions are never interrupted; only genuinely-idle ones clean up.
|
||||
//
|
||||
// Defaults to 10 minutes. Set AGENT_BROWSER_IDLE_TIMEOUT_MS to override, or 0
|
||||
// to disable (keep the daemon alive forever — the old behaviour). Adopted
|
||||
// tabs (the user's own, via `adopt`) are never closed: only `created_targets`.
|
||||
const DEFAULT_IDLE_TIMEOUT_MS: u64 = 600_000;
|
||||
let idle_timeout_ms = match env::var("AGENT_BROWSER_IDLE_TIMEOUT_MS") {
|
||||
Ok(s) => s.trim().parse::<u64>().ok().filter(|&ms| ms > 0),
|
||||
Err(_) => Some(DEFAULT_IDLE_TIMEOUT_MS),
|
||||
};
|
||||
|
||||
let result = run_socket_server(
|
||||
&socket_path,
|
||||
|
||||
@@ -35,6 +35,7 @@ fn native_test_fixture_html(name: &str) -> &'static str {
|
||||
"html5_drag_probe" => include_str!("test_fixtures/html5_drag_probe.html"),
|
||||
"pointer_capture_probe" => include_str!("test_fixtures/pointer_capture_probe.html"),
|
||||
"upload_probe" => include_str!("test_fixtures/upload_probe.html"),
|
||||
"iframe_button_probe" => include_str!("test_fixtures/iframe_button_probe.html"),
|
||||
_ => panic!("Unknown native test fixture: {}", name),
|
||||
}
|
||||
}
|
||||
@@ -573,6 +574,76 @@ async fn e2e_snapshot_and_click_ref() {
|
||||
assert_success(&resp);
|
||||
}
|
||||
|
||||
/// Clicking a button INSIDE an iframe by `@ref` must deliver a TRUSTED activation
|
||||
/// (`event.isTrusted === true`), not a synthetic DOM `.click()`. Security-sensitive
|
||||
/// embedded forms (Google Payments' `保存`) reject `isTrusted:false` clicks, so an
|
||||
/// enabled submit button silently no-op'd (issue #39). The fix routes iframe-ref
|
||||
/// clicks to a real `Input.dispatchMouseEvent` on the element's own frame session.
|
||||
/// The fixture's iframe button writes `clicked:<isTrusted>` into its own text on
|
||||
/// click, which the cross-frame snapshot reads back.
|
||||
#[tokio::test]
|
||||
#[ignore]
|
||||
async fn e2e_iframe_button_click_is_trusted() {
|
||||
let mut state = DaemonState::new();
|
||||
|
||||
let resp = execute_command(
|
||||
&json!({ "id": "1", "action": "launch", "headless": true }),
|
||||
&mut state,
|
||||
)
|
||||
.await;
|
||||
assert_success(&resp);
|
||||
|
||||
let resp = execute_command(
|
||||
&json!({ "id": "2", "action": "navigate", "url": native_test_fixture_url("iframe_button_probe") }),
|
||||
&mut state,
|
||||
)
|
||||
.await;
|
||||
assert_success(&resp);
|
||||
|
||||
// Snapshot (interactive) — the button lives in the iframe and must appear with
|
||||
// a ref; that ref carries the frame_id so the click resolves into the frame.
|
||||
let resp = execute_command(
|
||||
&json!({ "id": "3", "action": "snapshot", "interactive": true }),
|
||||
&mut state,
|
||||
)
|
||||
.await;
|
||||
assert_success(&resp);
|
||||
let snapshot = get_data(&resp)["snapshot"].as_str().unwrap_or("");
|
||||
let ref_id = snapshot
|
||||
.lines()
|
||||
.find(|l| l.contains("button \"save\""))
|
||||
.and_then(|l| l.split("ref=").nth(1))
|
||||
.map(|r| r.trim_end_matches(']').trim())
|
||||
.unwrap_or_else(|| panic!("iframe button not found in snapshot:\n{snapshot}"));
|
||||
|
||||
// Click it by ref.
|
||||
let resp = execute_command(
|
||||
&json!({ "id": "4", "action": "click", "selector": ref_id }),
|
||||
&mut state,
|
||||
)
|
||||
.await;
|
||||
assert_success(&resp);
|
||||
|
||||
tokio::time::sleep(tokio::time::Duration::from_millis(300)).await;
|
||||
|
||||
// The button rewrote its own text with the click's isTrusted flag; read it
|
||||
// back across frames.
|
||||
let resp = execute_command(
|
||||
&json!({ "id": "5", "action": "snapshot", "interactive": true }),
|
||||
&mut state,
|
||||
)
|
||||
.await;
|
||||
assert_success(&resp);
|
||||
let after = get_data(&resp)["snapshot"].as_str().unwrap_or("");
|
||||
assert!(
|
||||
after.contains("clicked:true"),
|
||||
"iframe button click must be trusted (isTrusted:true); snapshot:\n{after}"
|
||||
);
|
||||
|
||||
let resp = execute_command(&json!({ "id": "99", "action": "close" }), &mut state).await;
|
||||
assert_success(&resp);
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Screenshot
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
@@ -100,6 +100,15 @@ impl RefMap {
|
||||
self.map.get(ref_id)
|
||||
}
|
||||
|
||||
/// Whether `selector_or_ref` is a `@ref` whose snapshot entry lives inside an
|
||||
/// iframe (has a `frame_id`). Pointer interactions use this to choose
|
||||
/// DOM-dispatch over coordinates for OOPIF elements (issue #36).
|
||||
pub fn ref_is_in_iframe(&self, selector_or_ref: &str) -> bool {
|
||||
parse_ref(selector_or_ref)
|
||||
.and_then(|r| self.map.get(&r).map(|e| e.frame_id.is_some()))
|
||||
.unwrap_or(false)
|
||||
}
|
||||
|
||||
pub fn entries_sorted(&self) -> Vec<(String, RefEntry)> {
|
||||
let mut entries = self
|
||||
.map
|
||||
@@ -1019,7 +1028,9 @@ async fn eval_text_in_frame(client: &CdpClient, session_id: &str, frame_id: &str
|
||||
.await
|
||||
.ok()
|
||||
.and_then(|v| v.get("executionContextId").and_then(|c| c.as_i64()));
|
||||
let Some(ctx_id) = ctx else { return String::new() };
|
||||
let Some(ctx_id) = ctx else {
|
||||
return String::new();
|
||||
};
|
||||
let res = client
|
||||
.send_command(
|
||||
"Runtime.evaluate",
|
||||
@@ -1090,7 +1101,10 @@ pub async fn collect_all_frames_text(
|
||||
let (kind, text) = if is_top {
|
||||
("top", eval_text_default(client, top_session).await)
|
||||
} else {
|
||||
("inline", eval_text_in_frame(client, top_session, &fid).await)
|
||||
(
|
||||
"inline",
|
||||
eval_text_in_frame(client, top_session, &fid).await,
|
||||
)
|
||||
};
|
||||
out.push(FrameText {
|
||||
frame_id: fid,
|
||||
@@ -1515,9 +1529,27 @@ pub async fn get_element_input_value(
|
||||
.send_command_typed(
|
||||
"Runtime.callFunctionOn",
|
||||
&CallFunctionOnParams {
|
||||
function_declaration:
|
||||
"function() { return typeof this.value === 'string' ? this.value : ''; }"
|
||||
.to_string(),
|
||||
// Read rich-editor content too (issue #41): CodeMirror 5 / Monaco
|
||||
// keep their text in a model, not `.value`; contenteditable keeps
|
||||
// it as innerText. Falls back to `.value` for plain inputs.
|
||||
function_declaration: r#"function() {
|
||||
const el = this;
|
||||
const cm5 = el.closest && el.closest('.CodeMirror');
|
||||
if (cm5 && cm5.CodeMirror) return cm5.CodeMirror.getValue();
|
||||
if (window.monaco && monaco.editor) {
|
||||
try {
|
||||
const eds = monaco.editor.getEditors ? monaco.editor.getEditors() : [];
|
||||
const ed = eds.find(e => e.getDomNode && e.getDomNode().contains(el)) || eds[0];
|
||||
if (ed) return ed.getValue();
|
||||
const m = monaco.editor.getModels ? monaco.editor.getModels() : [];
|
||||
if (m[0]) return m[0].getValue();
|
||||
} catch (e) {}
|
||||
}
|
||||
if (typeof el.value === 'string') return el.value;
|
||||
if (el.isContentEditable) return el.innerText;
|
||||
return '';
|
||||
}"#
|
||||
.to_string(),
|
||||
object_id: Some(object_id),
|
||||
arguments: None,
|
||||
return_by_value: Some(true),
|
||||
@@ -1595,7 +1627,15 @@ pub async fn get_element_bounding_box(
|
||||
&CallFunctionOnParams {
|
||||
function_declaration: r#"function() {
|
||||
const r = this.getBoundingClientRect();
|
||||
return { x: r.x, y: r.y, width: r.width, height: r.height };
|
||||
const inViewport = r.bottom > 0 && r.right > 0
|
||||
&& r.top < (innerHeight || document.documentElement.clientHeight)
|
||||
&& r.left < (innerWidth || document.documentElement.clientWidth);
|
||||
return {
|
||||
x: r.x, y: r.y, width: r.width, height: r.height,
|
||||
centerX: Math.round(r.x + r.width / 2),
|
||||
centerY: Math.round(r.y + r.height / 2),
|
||||
inViewport,
|
||||
};
|
||||
}"#
|
||||
.to_string(),
|
||||
object_id: Some(object_id),
|
||||
|
||||
+378
-18
@@ -7,6 +7,17 @@ use super::cdp::types::*;
|
||||
use super::element::{parse_ref, resolve_element_center, resolve_element_object_id, RefMap};
|
||||
use super::humanize;
|
||||
|
||||
/// Whether a pointer interaction should be DOM-dispatched (invoke the event on
|
||||
/// the element in its own session) rather than dispatched at a viewport
|
||||
/// coordinate via `Input.dispatchMouseEvent`. True when the target is inside an
|
||||
/// iframe (an OOPIF element's box can't be mapped to a top-viewport point) or we
|
||||
/// drive over the extension relay (a coordinate Input event isn't confined to the
|
||||
/// target tab on a busy real Chrome — it drifts onto the foreground tab; issues
|
||||
/// #31/#36). DOM-dispatch always hits the right element in the right tab.
|
||||
fn prefer_dom_dispatch(ref_map: &RefMap, selector_or_ref: &str) -> bool {
|
||||
ref_map.ref_is_in_iframe(selector_or_ref) || crate::connect::relay_url().is_some()
|
||||
}
|
||||
|
||||
pub async fn click(
|
||||
client: &CdpClient,
|
||||
session_id: &str,
|
||||
@@ -45,6 +56,45 @@ pub async fn click(
|
||||
.await;
|
||||
}
|
||||
|
||||
// An element INSIDE an iframe needs a TRUSTED activation: a DOM `.click()` is
|
||||
// `isTrusted:false`, which security-sensitive embedded forms reject — Google
|
||||
// Payments' enabled `保存` button silently no-ops on a synthetic click (issue
|
||||
// #39). A coordinate `Input.dispatchMouseEvent` can't help either: `getBoxModel`
|
||||
// for a sub-frame node returns frame-local coordinates that don't compose the
|
||||
// iframe's offset, so the click lands in the wrong place. The frame-agnostic
|
||||
// trusted path is keyboard activation — focus the element in its own frame, then
|
||||
// dispatch a real Enter on the page session; Chrome routes the key to the
|
||||
// focused element regardless of frame (same as `type --focused`), and Enter on a
|
||||
// focused button/link fires a trusted `click`. `coord` mode opts out.
|
||||
let in_iframe = ref_map.ref_is_in_iframe(selector_or_ref);
|
||||
if mode != "coord" && button == "left" && click_count == 1 && in_iframe {
|
||||
return dom_activate(
|
||||
client,
|
||||
session_id,
|
||||
ref_map,
|
||||
selector_or_ref,
|
||||
iframe_sessions,
|
||||
)
|
||||
.await;
|
||||
}
|
||||
// On the relay (the user's real Chrome) a TOP-document coordinate click used to
|
||||
// drift onto the foreground tab; that root cause is fixed (#5: the agent drives
|
||||
// its own pinned tab), but DOM-dispatch stays the conservative default here.
|
||||
if mode != "coord"
|
||||
&& button == "left"
|
||||
&& click_count == 1
|
||||
&& crate::connect::relay_url().is_some()
|
||||
{
|
||||
return dom_click(
|
||||
client,
|
||||
session_id,
|
||||
ref_map,
|
||||
selector_or_ref,
|
||||
iframe_sessions,
|
||||
)
|
||||
.await;
|
||||
}
|
||||
|
||||
let resolved = resolve_element_center(
|
||||
client,
|
||||
session_id,
|
||||
@@ -235,6 +285,112 @@ async fn dom_click(
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Trusted activation of an element inside an iframe (issue #39). Focuses the
|
||||
/// element in its own frame session, then dispatches a real Enter/Space on the
|
||||
/// page session — Chrome routes the key to the focused element across frames, and
|
||||
/// Enter/Space on a focused button/link/checkbox fires a `click` with
|
||||
/// `isTrusted: true`, which security-sensitive embedded forms (Google Payments
|
||||
/// `保存`) require. Non-activatable roles (a `div[onclick]`) can't be keyboard-
|
||||
/// activated, so they fall back to a DOM `.click()`.
|
||||
async fn dom_activate(
|
||||
client: &CdpClient,
|
||||
session_id: &str,
|
||||
ref_map: &RefMap,
|
||||
selector_or_ref: &str,
|
||||
iframe_sessions: &HashMap<String, String>,
|
||||
) -> Result<(), String> {
|
||||
let role = parse_ref(selector_or_ref)
|
||||
.and_then(|r| ref_map.get(&r).map(|e| e.role.clone()))
|
||||
.unwrap_or_default();
|
||||
// Space toggles checkbox-like controls; Enter activates buttons/links/menus.
|
||||
let key = match role.as_str() {
|
||||
"checkbox" | "radio" | "switch" | "option" | "menuitemcheckbox" | "menuitemradio" => {
|
||||
Some("space")
|
||||
}
|
||||
"button" | "link" | "menuitem" | "tab" | "treeitem" => Some("enter"),
|
||||
_ => None,
|
||||
};
|
||||
let Some(key) = key else {
|
||||
// Not keyboard-activatable — best effort via DOM .click() (untrusted).
|
||||
return dom_click(
|
||||
client,
|
||||
session_id,
|
||||
ref_map,
|
||||
selector_or_ref,
|
||||
iframe_sessions,
|
||||
)
|
||||
.await;
|
||||
};
|
||||
|
||||
let (object_id, effective_session_id) = resolve_element_object_id(
|
||||
client,
|
||||
session_id,
|
||||
ref_map,
|
||||
selector_or_ref,
|
||||
iframe_sessions,
|
||||
)
|
||||
.await?;
|
||||
// Focus the element in its OWN frame session so the keystroke lands on it.
|
||||
client
|
||||
.send_command_typed::<_, Value>(
|
||||
"Runtime.callFunctionOn",
|
||||
&CallFunctionOnParams {
|
||||
function_declaration: "function() { this.focus(); }".to_string(),
|
||||
object_id: Some(object_id),
|
||||
arguments: None,
|
||||
return_by_value: Some(true),
|
||||
await_promise: Some(false),
|
||||
},
|
||||
Some(&effective_session_id),
|
||||
)
|
||||
.await?;
|
||||
// Trusted key on the page session — routed to the focused (in-frame) element.
|
||||
press_key(client, session_id, key).await?;
|
||||
wait_for_paint_settled(client, &effective_session_id).await;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// DOM-dispatch a double-click on the element in its own session (no coordinates)
|
||||
/// — the relay/iframe-safe counterpart to a coordinate dblclick. Fires the full
|
||||
/// click,click,dblclick sequence so handlers bound to any of them respond.
|
||||
async fn dom_dblclick(
|
||||
client: &CdpClient,
|
||||
session_id: &str,
|
||||
ref_map: &RefMap,
|
||||
selector_or_ref: &str,
|
||||
iframe_sessions: &HashMap<String, String>,
|
||||
) -> Result<(), String> {
|
||||
let (object_id, effective_session_id) = resolve_element_object_id(
|
||||
client,
|
||||
session_id,
|
||||
ref_map,
|
||||
selector_or_ref,
|
||||
iframe_sessions,
|
||||
)
|
||||
.await?;
|
||||
client
|
||||
.send_command_typed::<_, Value>(
|
||||
"Runtime.callFunctionOn",
|
||||
&CallFunctionOnParams {
|
||||
function_declaration: r#"function() {
|
||||
const opts = { bubbles: true, cancelable: true, view: window };
|
||||
this.dispatchEvent(new MouseEvent('click', opts));
|
||||
this.dispatchEvent(new MouseEvent('click', { ...opts, detail: 2 }));
|
||||
this.dispatchEvent(new MouseEvent('dblclick', opts));
|
||||
}"#
|
||||
.to_string(),
|
||||
object_id: Some(object_id),
|
||||
arguments: None,
|
||||
return_by_value: Some(true),
|
||||
await_promise: Some(false),
|
||||
},
|
||||
Some(&effective_session_id),
|
||||
)
|
||||
.await?;
|
||||
wait_for_paint_settled(client, &effective_session_id).await;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
pub async fn dblclick(
|
||||
client: &CdpClient,
|
||||
session_id: &str,
|
||||
@@ -242,6 +398,20 @@ pub async fn dblclick(
|
||||
selector_or_ref: &str,
|
||||
iframe_sessions: &HashMap<String, String>,
|
||||
) -> Result<(), String> {
|
||||
// Same relay/iframe drift hazard as a single click — DOM-dispatch the
|
||||
// double-click there instead of a coordinate one (issues #31/#36).
|
||||
if std::env::var("AGENT_BROWSER_CLICK_MODE").as_deref() != Ok("coord")
|
||||
&& prefer_dom_dispatch(ref_map, selector_or_ref)
|
||||
{
|
||||
return dom_dblclick(
|
||||
client,
|
||||
session_id,
|
||||
ref_map,
|
||||
selector_or_ref,
|
||||
iframe_sessions,
|
||||
)
|
||||
.await;
|
||||
}
|
||||
click(
|
||||
client,
|
||||
session_id,
|
||||
@@ -254,6 +424,50 @@ pub async fn dblclick(
|
||||
.await
|
||||
}
|
||||
|
||||
/// DOM-dispatch a hover (pointer/mouse enter+move) on the element in its own
|
||||
/// session — reaches OOPIF elements and never drifts to the foreground tab over
|
||||
/// the relay, unlike a coordinate `mouseMoved` (issues #31/#36).
|
||||
async fn dom_hover(
|
||||
client: &CdpClient,
|
||||
session_id: &str,
|
||||
ref_map: &RefMap,
|
||||
selector_or_ref: &str,
|
||||
iframe_sessions: &HashMap<String, String>,
|
||||
) -> Result<(), String> {
|
||||
let (object_id, effective_session_id) = resolve_element_object_id(
|
||||
client,
|
||||
session_id,
|
||||
ref_map,
|
||||
selector_or_ref,
|
||||
iframe_sessions,
|
||||
)
|
||||
.await?;
|
||||
client
|
||||
.send_command_typed::<_, Value>(
|
||||
"Runtime.callFunctionOn",
|
||||
&CallFunctionOnParams {
|
||||
function_declaration: r#"function() {
|
||||
const r = this.getBoundingClientRect();
|
||||
const cx = r.left + r.width / 2, cy = r.top + r.height / 2;
|
||||
const base = { bubbles: true, cancelable: true, view: window, clientX: cx, clientY: cy };
|
||||
this.dispatchEvent(new PointerEvent('pointerover', base));
|
||||
this.dispatchEvent(new PointerEvent('pointerenter', { ...base, bubbles: false }));
|
||||
this.dispatchEvent(new MouseEvent('mouseover', base));
|
||||
this.dispatchEvent(new MouseEvent('mouseenter', { ...base, bubbles: false }));
|
||||
this.dispatchEvent(new MouseEvent('mousemove', base));
|
||||
}"#
|
||||
.to_string(),
|
||||
object_id: Some(object_id),
|
||||
arguments: None,
|
||||
return_by_value: Some(true),
|
||||
await_promise: Some(false),
|
||||
},
|
||||
Some(&effective_session_id),
|
||||
)
|
||||
.await?;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
pub async fn hover(
|
||||
client: &CdpClient,
|
||||
session_id: &str,
|
||||
@@ -261,6 +475,18 @@ pub async fn hover(
|
||||
selector_or_ref: &str,
|
||||
iframe_sessions: &HashMap<String, String>,
|
||||
) -> Result<(), String> {
|
||||
// Coordinate `mouseMoved` drifts to the foreground tab over the relay and
|
||||
// can't reach an OOPIF — DOM-dispatch the hover there (issues #31/#36).
|
||||
if prefer_dom_dispatch(ref_map, selector_or_ref) {
|
||||
return dom_hover(
|
||||
client,
|
||||
session_id,
|
||||
ref_map,
|
||||
selector_or_ref,
|
||||
iframe_sessions,
|
||||
)
|
||||
.await;
|
||||
}
|
||||
let (x, y, _w, _h, effective_session_id) = resolve_element_center(
|
||||
client,
|
||||
session_id,
|
||||
@@ -289,6 +515,63 @@ pub async fn hover(
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// DOM-dispatch an HTML5 drag-and-drop from `source` to `target` in their shared
|
||||
/// session — the relay/iframe-safe counterpart to the coordinate drag, which
|
||||
/// drifts to the foreground tab over the relay and can't reach an OOPIF (issues
|
||||
/// #31/#36). Covers HTML5 DnD (sortable lists, file/card boards); pointer-driven
|
||||
/// drag (canvas, sliders) still needs the coordinate path. Errors if source and
|
||||
/// target live in different frames — a synthetic cross-frame DnD isn't reliable.
|
||||
pub async fn dom_drag(
|
||||
client: &CdpClient,
|
||||
session_id: &str,
|
||||
ref_map: &RefMap,
|
||||
source: &str,
|
||||
target: &str,
|
||||
iframe_sessions: &HashMap<String, String>,
|
||||
) -> Result<(), String> {
|
||||
let (src_obj, src_session) =
|
||||
resolve_element_object_id(client, session_id, ref_map, source, iframe_sessions).await?;
|
||||
let (tgt_obj, tgt_session) =
|
||||
resolve_element_object_id(client, session_id, ref_map, target, iframe_sessions).await?;
|
||||
if src_session != tgt_session {
|
||||
return Err(
|
||||
"drag source and target are in different frames; cross-frame drag-and-drop over the \
|
||||
relay isn't supported — drag within a single frame, or use a launched browser with \
|
||||
AGENT_BROWSER_CLICK_MODE=coord"
|
||||
.to_string(),
|
||||
);
|
||||
}
|
||||
client
|
||||
.send_command_typed::<_, Value>(
|
||||
"Runtime.callFunctionOn",
|
||||
&CallFunctionOnParams {
|
||||
function_declaration: r#"function(target) {
|
||||
const dt = new DataTransfer();
|
||||
const ev = (type, el) => el.dispatchEvent(
|
||||
new DragEvent(type, { bubbles: true, cancelable: true, dataTransfer: dt }));
|
||||
ev('dragstart', this);
|
||||
ev('drag', this);
|
||||
ev('dragenter', target);
|
||||
ev('dragover', target);
|
||||
ev('drop', target);
|
||||
ev('dragend', this);
|
||||
}"#
|
||||
.to_string(),
|
||||
object_id: Some(src_obj),
|
||||
arguments: Some(vec![CallArgument {
|
||||
value: None,
|
||||
object_id: Some(tgt_obj),
|
||||
}]),
|
||||
return_by_value: Some(true),
|
||||
await_promise: Some(false),
|
||||
},
|
||||
Some(&src_session),
|
||||
)
|
||||
.await?;
|
||||
wait_for_paint_settled(client, &src_session).await;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
pub async fn fill(
|
||||
client: &CdpClient,
|
||||
session_id: &str,
|
||||
@@ -296,7 +579,7 @@ pub async fn fill(
|
||||
selector_or_ref: &str,
|
||||
value: &str,
|
||||
iframe_sessions: &HashMap<String, String>,
|
||||
) -> Result<(), String> {
|
||||
) -> Result<String, String> {
|
||||
let (object_id, effective_session_id) = resolve_element_object_id(
|
||||
client,
|
||||
session_id,
|
||||
@@ -307,14 +590,15 @@ pub async fn fill(
|
||||
.await?;
|
||||
|
||||
// Emulate a real edit so framework-controlled inputs (React/Vue) and
|
||||
// site-side listeners actually see the change (issue #25): the old path set
|
||||
// `this.value` directly and used Input.insertText, which left React's
|
||||
// internal value-tracker out of sync and never fired change/blur — so
|
||||
// dependent logic (e.g. Mercari's postal-code → 都道府県 autocomplete) never
|
||||
// ran even though the value was visible. Set the value through the element's
|
||||
// PROTOTYPE setter (which React's _valueTracker hooks), then dispatch
|
||||
// input → change → blur/focusout. `type <sel> <text>` remains for sites that
|
||||
// need per-keystroke events.
|
||||
// site-side listeners actually see the change (issue #25): set the value
|
||||
// through the element's PROTOTYPE setter (which React's _valueTracker hooks),
|
||||
// then dispatch input → change → blur/focusout. Beyond plain inputs, detect
|
||||
// rich editors and use their own API/events (issue #41): CodeMirror 5 and
|
||||
// Monaco have a model that `.value`/`textContent` can't touch; ProseMirror /
|
||||
// contenteditable need `execCommand('insertText')` so beforeinput/input fire
|
||||
// (a raw `textContent =` corrupts PM's doc and skips React composers).
|
||||
// Returns the engine used so the caller can report it. `type <sel> <text>`
|
||||
// remains for sites that need per-keystroke events.
|
||||
let fill_js = format!(
|
||||
r#"function() {{
|
||||
const el = this;
|
||||
@@ -322,13 +606,43 @@ pub async fn fill(
|
||||
try {{ el.focus(); }} catch (e) {{}}
|
||||
const tag = el.tagName;
|
||||
const fire = (type, ctor) => el.dispatchEvent(new (ctor || Event)(type, {{ bubbles: true }}));
|
||||
if (tag === 'SELECT') {{
|
||||
el.value = v; fire('input'); fire('change'); return true;
|
||||
|
||||
// CodeMirror 5: a hidden <textarea> inside .CodeMirror with a live instance.
|
||||
const cm5 = el.closest && el.closest('.CodeMirror');
|
||||
if (cm5 && cm5.CodeMirror) {{ cm5.CodeMirror.setValue(v); return 'codemirror5'; }}
|
||||
|
||||
// Monaco: global `monaco`; prefer the editor whose DOM contains el.
|
||||
if (window.monaco && monaco.editor) {{
|
||||
try {{
|
||||
const eds = monaco.editor.getEditors ? monaco.editor.getEditors() : [];
|
||||
const ed = eds.find(e => e.getDomNode && e.getDomNode().contains(el)) || eds[0];
|
||||
if (ed) {{ ed.setValue(v); return 'monaco'; }}
|
||||
const models = monaco.editor.getModels ? monaco.editor.getModels() : [];
|
||||
if (models[0]) {{ models[0].setValue(v); return 'monaco'; }}
|
||||
}} catch (e) {{}}
|
||||
}}
|
||||
|
||||
if (tag === 'SELECT') {{ el.value = v; fire('input'); fire('change'); return 'select'; }}
|
||||
|
||||
if (el.isContentEditable) {{
|
||||
el.textContent = v; fire('input', window.InputEvent || Event); fire('change');
|
||||
try {{ el.blur(); }} catch (e) {{}} fire('focusout'); return true;
|
||||
// ProseMirror / contenteditable: select-all then insertText fires
|
||||
// beforeinput/input that PM and React composers listen for.
|
||||
let ok = false;
|
||||
try {{
|
||||
const sel = window.getSelection();
|
||||
const range = document.createRange();
|
||||
range.selectNodeContents(el);
|
||||
sel.removeAllRanges();
|
||||
sel.addRange(range);
|
||||
ok = document.execCommand('insertText', false, v);
|
||||
}} catch (e) {{}}
|
||||
if (!ok) {{ el.textContent = v; fire('input', window.InputEvent || Event); }}
|
||||
fire('change');
|
||||
try {{ el.blur(); }} catch (e) {{}}
|
||||
fire('focusout');
|
||||
return ok ? 'contenteditable' : 'contenteditable-fallback';
|
||||
}}
|
||||
|
||||
const proto = tag === 'TEXTAREA' ? window.HTMLTextAreaElement.prototype
|
||||
: window.HTMLInputElement.prototype;
|
||||
const desc = Object.getOwnPropertyDescriptor(proto, 'value');
|
||||
@@ -340,13 +654,13 @@ pub async fn fill(
|
||||
fire('change');
|
||||
try {{ el.blur(); }} catch (e) {{}}
|
||||
fire('focusout'); // blur-triggered lookups/validation
|
||||
return true;
|
||||
return 'input';
|
||||
}}"#,
|
||||
val = serde_json::to_string(value).unwrap_or_default()
|
||||
);
|
||||
|
||||
client
|
||||
.send_command_typed::<_, Value>(
|
||||
let result: EvaluateResult = client
|
||||
.send_command_typed(
|
||||
"Runtime.callFunctionOn",
|
||||
&CallFunctionOnParams {
|
||||
function_declaration: fill_js,
|
||||
@@ -359,7 +673,11 @@ pub async fn fill(
|
||||
)
|
||||
.await?;
|
||||
|
||||
Ok(())
|
||||
Ok(result
|
||||
.result
|
||||
.value
|
||||
.and_then(|v| v.as_str().map(String::from))
|
||||
.unwrap_or_else(|| "input".to_string()))
|
||||
}
|
||||
|
||||
#[allow(clippy::too_many_arguments)]
|
||||
@@ -372,6 +690,7 @@ pub async fn type_text(
|
||||
clear: bool,
|
||||
delay_ms: Option<u64>,
|
||||
iframe_sessions: &HashMap<String, String>,
|
||||
key_events: bool,
|
||||
) -> Result<(), String> {
|
||||
let (object_id, effective_session_id) = resolve_element_object_id(
|
||||
client,
|
||||
@@ -418,7 +737,7 @@ pub async fn type_text(
|
||||
.await?;
|
||||
}
|
||||
|
||||
type_text_into_active_context(client, session_id, text, delay_ms).await
|
||||
type_text_into_active_context(client, session_id, text, delay_ms, key_events).await
|
||||
}
|
||||
|
||||
pub async fn type_text_into_active_context(
|
||||
@@ -426,6 +745,7 @@ pub async fn type_text_into_active_context(
|
||||
session_id: &str,
|
||||
text: &str,
|
||||
delay_ms: Option<u64>,
|
||||
key_events: bool,
|
||||
) -> Result<(), String> {
|
||||
// Per-character timing: an explicit `delay_ms` wins (caller asked for a
|
||||
// fixed cadence); otherwise fall back to humanize — variable, human-like
|
||||
@@ -475,6 +795,46 @@ pub async fn type_text_into_active_context(
|
||||
Some(session_id),
|
||||
)
|
||||
.await?;
|
||||
} else if key_events {
|
||||
// Real keystrokes (keyDown+keyUp carrying `text`) for autocomplete /
|
||||
// combobox widgets that only react to key events and ignore the
|
||||
// `input` that `Input.insertText` fires — e.g. Google's address
|
||||
// postal-code → city/prefecture lookup (issue #36 / #4). The keyDown's
|
||||
// `text` still inserts the character, so the field also fills.
|
||||
let (key, code, key_code) = char_to_key_info(ch);
|
||||
let s = ch.to_string();
|
||||
client
|
||||
.send_command_typed::<_, Value>(
|
||||
"Input.dispatchKeyEvent",
|
||||
&DispatchKeyEventParams {
|
||||
event_type: "keyDown".to_string(),
|
||||
key: Some(key.clone()),
|
||||
code: Some(code.clone()),
|
||||
text: Some(s.clone()),
|
||||
unmodified_text: Some(s),
|
||||
windows_virtual_key_code: Some(key_code),
|
||||
native_virtual_key_code: Some(key_code),
|
||||
modifiers: None,
|
||||
},
|
||||
Some(session_id),
|
||||
)
|
||||
.await?;
|
||||
client
|
||||
.send_command_typed::<_, Value>(
|
||||
"Input.dispatchKeyEvent",
|
||||
&DispatchKeyEventParams {
|
||||
event_type: "keyUp".to_string(),
|
||||
key: Some(key),
|
||||
code: Some(code),
|
||||
text: None,
|
||||
unmodified_text: None,
|
||||
windows_virtual_key_code: Some(key_code),
|
||||
native_virtual_key_code: Some(key_code),
|
||||
modifiers: None,
|
||||
},
|
||||
Some(session_id),
|
||||
)
|
||||
.await?;
|
||||
} else {
|
||||
// VS Code/Electron webviews reject repeated dispatchKeyEvent calls
|
||||
// carrying printable `text`. Insert printable characters directly
|
||||
|
||||
+273
-4
@@ -52,6 +52,22 @@ pub struct RelayState {
|
||||
pending: HashMap<i64, (ClientId, Value)>,
|
||||
/// monotonic source of relay-global command ids
|
||||
next_global_id: i64,
|
||||
/// Group-scoped isolation (issue #40). A tab group belongs to exactly one
|
||||
/// agent/session; the relay scopes `Target.getTargets` per client to its own
|
||||
/// group so the daemon can safely adopt new tabs (follow-popup, cross-session
|
||||
/// adopt) without ever seeing the user's or another agent's tabs.
|
||||
///
|
||||
/// clientId -> group name. A client that never announced a group (older
|
||||
/// daemon) is absent here and gets the full, UNSCOPED target list — so this
|
||||
/// is fully backward-compatible.
|
||||
client_groups: HashMap<ClientId, String>,
|
||||
/// targetId -> group name. Created tabs are tagged from `Target.createTarget`'s
|
||||
/// `agentGroup`; an explicitly adopted tab is tagged to the adopter; a pop-up
|
||||
/// inherits its opener's group (needs the extension to report `openerTargetId`).
|
||||
target_group: HashMap<String, String>,
|
||||
/// relay-global id of an in-flight `Target.createTarget` -> the `agentGroup`
|
||||
/// it carried, so the reply's `targetId` can be tagged with that group.
|
||||
pending_create: HashMap<i64, String>,
|
||||
}
|
||||
|
||||
/// What to do with a raw CDP command received from a `CdpClient`.
|
||||
@@ -95,6 +111,7 @@ impl RelayState {
|
||||
/// `pending` entries don't leak.
|
||||
pub fn drop_client(&mut self, client_id: ClientId) {
|
||||
self.pending.retain(|_, (cid, _)| *cid != client_id);
|
||||
self.client_groups.remove(&client_id);
|
||||
}
|
||||
|
||||
/// Route a raw CDP command `{id, method, params?, sessionId?}` from a
|
||||
@@ -123,12 +140,28 @@ impl RelayState {
|
||||
"jsVersion": ""
|
||||
}
|
||||
})),
|
||||
// Non-CDP control message: a daemon announces which tab group
|
||||
// (session) it owns, so getTargets can be scoped to it (issue #40).
|
||||
"ABRelay.setGroup" => {
|
||||
if let Some(g) = params.get("group").and_then(|g| g.as_str()) {
|
||||
if !g.is_empty() {
|
||||
self.client_groups.insert(client_id, g.to_string());
|
||||
}
|
||||
}
|
||||
ClientRoute::Local(json!({ "id": id, "result": {} }))
|
||||
}
|
||||
// Discovery is best-effort and event-driven in real CDP; abs only
|
||||
// reads the getTargets result, so an empty ack is enough here.
|
||||
"Target.setDiscoverTargets" | "Target.setAutoAttach" => {
|
||||
ClientRoute::Local(json!({ "id": id, "result": {} }))
|
||||
}
|
||||
"Target.getTargets" => {
|
||||
// Unscoped discovery for EXPLICIT cross-group adoption (`chrome-use
|
||||
// adopt`): returns every target the extension has attached, ignoring
|
||||
// group scoping, so an agent can find a specific pre-existing tab (the
|
||||
// user's, another session's) by URL/targetId and adopt it. Isolation
|
||||
// is preserved because the daemon only acts on the one tab it then
|
||||
// attaches (which the relay re-tags into the adopter's group).
|
||||
"ABRelay.getAllTargets" => {
|
||||
let infos: Vec<Value> = self
|
||||
.targets
|
||||
.values()
|
||||
@@ -136,15 +169,44 @@ impl RelayState {
|
||||
.collect();
|
||||
ClientRoute::Local(json!({ "id": id, "result": { "targetInfos": infos } }))
|
||||
}
|
||||
"Target.getTargets" => {
|
||||
// Scope to the client's own group when it announced one; an
|
||||
// un-announced (legacy) client gets the full list (back-compat).
|
||||
let scoped = self.client_groups.get(&client_id).cloned();
|
||||
let infos: Vec<Value> = self
|
||||
.targets
|
||||
.iter()
|
||||
.filter(|(tid, _)| match &scoped {
|
||||
Some(g) => self
|
||||
.target_group
|
||||
.get(*tid)
|
||||
.map(|tg| tg == g)
|
||||
.unwrap_or(false),
|
||||
None => true,
|
||||
})
|
||||
.map(|(_, t)| t.target_info.clone())
|
||||
.collect();
|
||||
ClientRoute::Local(json!({ "id": id, "result": { "targetInfos": infos } }))
|
||||
}
|
||||
"Target.attachToTarget" => {
|
||||
let target_id = params
|
||||
.get("targetId")
|
||||
.and_then(|t| t.as_str())
|
||||
.unwrap_or("");
|
||||
match self.targets.get(target_id) {
|
||||
Some(entry) => ClientRoute::Local(
|
||||
json!({ "id": id, "result": { "sessionId": entry.session_id } }),
|
||||
),
|
||||
Some(entry) => {
|
||||
let session_id = entry.session_id.clone();
|
||||
// Explicitly adopting a target makes it this client's
|
||||
// (cross-session adopt, #21) — tag it into the adopter's
|
||||
// group so it stays in that client's scoped getTargets and
|
||||
// isn't churn-pruned.
|
||||
if let Some(g) = self.client_groups.get(&client_id).cloned() {
|
||||
self.target_group.insert(target_id.to_string(), g);
|
||||
}
|
||||
ClientRoute::Local(
|
||||
json!({ "id": id, "result": { "sessionId": session_id } }),
|
||||
)
|
||||
}
|
||||
None => ClientRoute::Local(json!({
|
||||
"id": id,
|
||||
"error": { "code": -32602, "message": format!("No such target {target_id}") }
|
||||
@@ -157,6 +219,18 @@ impl RelayState {
|
||||
self.next_global_id += 1;
|
||||
let gid = self.next_global_id;
|
||||
self.pending.insert(gid, (client_id, id));
|
||||
// Remember the group a createTarget carries so the reply's
|
||||
// targetId can be tagged to the creating session (issue #40).
|
||||
if method == "Target.createTarget" {
|
||||
if let Some(g) = params.get("agentGroup").and_then(|g| g.as_str()) {
|
||||
if !g.is_empty() {
|
||||
self.pending_create.insert(gid, g.to_string());
|
||||
self.client_groups
|
||||
.entry(client_id)
|
||||
.or_insert_with(|| g.to_string());
|
||||
}
|
||||
}
|
||||
}
|
||||
ClientRoute::Forward(json!({
|
||||
"id": gid,
|
||||
"method": "forwardCDPCommand",
|
||||
@@ -201,6 +275,17 @@ impl RelayState {
|
||||
&& msg.get("method").is_none()
|
||||
{
|
||||
let gid = msg.get("id").and_then(|i| i.as_i64());
|
||||
// A createTarget reply: tag the new tab's targetId with the group the
|
||||
// command carried, so it lands in the creating session's scope (#40).
|
||||
if let Some(g) = gid.and_then(|g| self.pending_create.remove(&g)) {
|
||||
if let Some(tid) = msg
|
||||
.get("result")
|
||||
.and_then(|r| r.get("targetId"))
|
||||
.and_then(|t| t.as_str())
|
||||
{
|
||||
self.target_group.insert(tid.to_string(), g);
|
||||
}
|
||||
}
|
||||
let (to, orig_id) = match gid.and_then(|g| self.pending.remove(&g)) {
|
||||
Some((client_id, orig)) => (Some(client_id), orig),
|
||||
// No mapping (stale/unknown id) — fall back to broadcasting with
|
||||
@@ -241,6 +326,27 @@ impl RelayState {
|
||||
.and_then(|s| s.as_str())
|
||||
.unwrap_or("")
|
||||
.to_string();
|
||||
// Attribute the tab to a group for scoping (issue #40),
|
||||
// unless we already know it (createTarget tag). An
|
||||
// explicit `abGroup` from the extension wins; otherwise a
|
||||
// pop-up inherits its opener's group via `openerTargetId`.
|
||||
if !self.target_group.contains_key(tid) {
|
||||
if let Some(g) = info
|
||||
.get("abGroup")
|
||||
.and_then(|g| g.as_str())
|
||||
.filter(|g| !g.is_empty())
|
||||
{
|
||||
self.target_group.insert(tid.to_string(), g.to_string());
|
||||
} else if let Some(opener) = info
|
||||
.get("openerTargetId")
|
||||
.and_then(|o| o.as_str())
|
||||
.filter(|o| !o.is_empty())
|
||||
{
|
||||
if let Some(g) = self.target_group.get(opener).cloned() {
|
||||
self.target_group.insert(tid.to_string(), g);
|
||||
}
|
||||
}
|
||||
}
|
||||
self.targets.insert(
|
||||
tid.to_string(),
|
||||
TargetEntry {
|
||||
@@ -255,6 +361,15 @@ impl RelayState {
|
||||
"Target.detachedFromTarget" => {
|
||||
let gone = inner_params.get("sessionId").and_then(|s| s.as_str());
|
||||
if let Some(gone) = gone {
|
||||
let gone_tids: Vec<String> = self
|
||||
.targets
|
||||
.iter()
|
||||
.filter(|(_, e)| e.session_id == gone)
|
||||
.map(|(tid, _)| tid.clone())
|
||||
.collect();
|
||||
for tid in gone_tids {
|
||||
self.target_group.remove(&tid);
|
||||
}
|
||||
self.targets.retain(|_, e| e.session_id != gone);
|
||||
}
|
||||
return vec![];
|
||||
@@ -557,4 +672,158 @@ mod tests {
|
||||
_ => panic!("expected ToExt"),
|
||||
}
|
||||
}
|
||||
|
||||
// === Group-scoped isolation (issue #40) ===
|
||||
|
||||
/// Drive the real create path: announce group, createTarget(agentGroup), feed
|
||||
/// the ext reply (tags target→group) + the attachedToTarget event (creates the
|
||||
/// entry). Returns nothing; mutates `s`.
|
||||
fn create_in_group(s: &mut RelayState, client: ClientId, group: &str, tid: &str, sid: &str) {
|
||||
s.route_client_command(
|
||||
client,
|
||||
&json!({ "id": 1, "method": "ABRelay.setGroup", "params": { "group": group } }),
|
||||
);
|
||||
let route = s.route_client_command(
|
||||
client,
|
||||
&json!({ "id": 2, "method": "Target.createTarget",
|
||||
"params": { "url": "about:blank", "agentGroup": group } }),
|
||||
);
|
||||
let gid = match route {
|
||||
ClientRoute::Forward(env) => env["id"].as_i64().unwrap(),
|
||||
_ => panic!("createTarget must forward"),
|
||||
};
|
||||
s.handle_ext_message(&json!({ "id": gid, "result": { "targetId": tid } }), "");
|
||||
s.handle_ext_message(
|
||||
&json!({ "method": "forwardCDPEvent", "params": {
|
||||
"method": "Target.attachedToTarget",
|
||||
"params": { "sessionId": sid, "targetInfo": {
|
||||
"targetId": tid, "type": "page", "url": "about:blank", "attached": true } } } }),
|
||||
"",
|
||||
);
|
||||
}
|
||||
|
||||
fn get_target_ids(s: &mut RelayState, client: ClientId) -> Vec<String> {
|
||||
match s.route_client_command(client, &json!({ "id": 9, "method": "Target.getTargets" })) {
|
||||
ClientRoute::Local(v) => v["result"]["targetInfos"]
|
||||
.as_array()
|
||||
.unwrap()
|
||||
.iter()
|
||||
.map(|t| t["targetId"].as_str().unwrap().to_string())
|
||||
.collect(),
|
||||
_ => panic!("getTargets must be local"),
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn get_targets_is_scoped_to_each_clients_group() {
|
||||
let mut s = RelayState::new();
|
||||
create_in_group(&mut s, 1, "agent-a", "ta", "sa");
|
||||
create_in_group(&mut s, 2, "agent-b", "tb", "sb");
|
||||
// Each client sees ONLY its own group's tab — never the other agent's.
|
||||
assert_eq!(get_target_ids(&mut s, 1), vec!["ta"]);
|
||||
assert_eq!(get_target_ids(&mut s, 2), vec!["tb"]);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn legacy_client_without_group_sees_all_targets() {
|
||||
let mut s = RelayState::new();
|
||||
create_in_group(&mut s, 1, "agent-a", "ta", "sa");
|
||||
create_in_group(&mut s, 2, "agent-b", "tb", "sb");
|
||||
// Client 3 never announced a group → full, unscoped list (back-compat).
|
||||
let mut all = get_target_ids(&mut s, 3);
|
||||
all.sort();
|
||||
assert_eq!(all, vec!["ta", "tb"]);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn popup_inherits_opener_group_and_is_visible_to_that_client_only() {
|
||||
let mut s = RelayState::new();
|
||||
create_in_group(&mut s, 1, "agent-a", "ta", "sa");
|
||||
create_in_group(&mut s, 2, "agent-b", "tb", "sb");
|
||||
// A pop-up that agent-a's tab opened: extension reports openerTargetId=ta.
|
||||
s.handle_ext_message(
|
||||
&json!({ "method": "forwardCDPEvent", "params": {
|
||||
"method": "Target.attachedToTarget",
|
||||
"params": { "sessionId": "sp", "targetInfo": {
|
||||
"targetId": "tp", "type": "page", "url": "https://oauth.example/",
|
||||
"attached": true, "openerTargetId": "ta" } } } }),
|
||||
"",
|
||||
);
|
||||
// Only agent-a sees the pop-up; agent-b never does.
|
||||
let mut a = get_target_ids(&mut s, 1);
|
||||
a.sort();
|
||||
assert_eq!(a, vec!["ta", "tp"]);
|
||||
assert_eq!(get_target_ids(&mut s, 2), vec!["tb"]);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn explicit_attach_tags_target_into_adopter_group() {
|
||||
let mut s = RelayState::new();
|
||||
// A pre-existing, ungrouped tab the extension reported (e.g. user's tab).
|
||||
s.handle_ext_message(
|
||||
&json!({ "method": "forwardCDPEvent", "params": {
|
||||
"method": "Target.attachedToTarget",
|
||||
"params": { "sessionId": "su", "targetInfo": {
|
||||
"targetId": "tu", "type": "page", "url": "https://user.example/", "attached": true } } } }),
|
||||
"",
|
||||
);
|
||||
// Client 1 (group agent-a) explicitly adopts it by targetId (#21).
|
||||
s.route_client_command(
|
||||
1,
|
||||
&json!({ "id": 1, "method": "ABRelay.setGroup", "params": { "group": "agent-a" } }),
|
||||
);
|
||||
s.route_client_command(
|
||||
1,
|
||||
&json!({ "id": 2, "method": "Target.attachToTarget", "params": { "targetId": "tu" } }),
|
||||
);
|
||||
// Now it's in agent-a's scope and survives the scoped getTargets.
|
||||
assert_eq!(get_target_ids(&mut s, 1), vec!["tu"]);
|
||||
// A different agent still doesn't see it.
|
||||
s.route_client_command(
|
||||
2,
|
||||
&json!({ "id": 1, "method": "ABRelay.setGroup", "params": { "group": "agent-b" } }),
|
||||
);
|
||||
assert!(get_target_ids(&mut s, 2).is_empty());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn get_all_targets_is_unscoped() {
|
||||
let mut s = RelayState::new();
|
||||
create_in_group(&mut s, 1, "agent-a", "ta", "sa");
|
||||
create_in_group(&mut s, 2, "agent-b", "tb", "sb");
|
||||
// Client 1's scoped getTargets sees only its own group...
|
||||
assert_eq!(get_target_ids(&mut s, 1), vec!["ta"]);
|
||||
// ...but ABRelay.getAllTargets returns EVERY target regardless of group
|
||||
// (for explicit cross-group adoption).
|
||||
let all = match s
|
||||
.route_client_command(1, &json!({ "id": 1, "method": "ABRelay.getAllTargets" }))
|
||||
{
|
||||
ClientRoute::Local(v) => {
|
||||
let mut ids: Vec<String> = v["result"]["targetInfos"]
|
||||
.as_array()
|
||||
.unwrap()
|
||||
.iter()
|
||||
.map(|t| t["targetId"].as_str().unwrap().to_string())
|
||||
.collect();
|
||||
ids.sort();
|
||||
ids
|
||||
}
|
||||
_ => panic!("getAllTargets must be local"),
|
||||
};
|
||||
assert_eq!(all, vec!["ta", "tb"]);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn detach_clears_target_group() {
|
||||
let mut s = RelayState::new();
|
||||
create_in_group(&mut s, 1, "agent-a", "ta", "sa");
|
||||
assert_eq!(get_target_ids(&mut s, 1), vec!["ta"]);
|
||||
s.handle_ext_message(
|
||||
&json!({ "method": "forwardCDPEvent", "params": {
|
||||
"method": "Target.detachedFromTarget", "params": { "sessionId": "sa" } } }),
|
||||
"",
|
||||
);
|
||||
assert!(get_target_ids(&mut s, 1).is_empty());
|
||||
assert!(!s.target_group.contains_key("ta"));
|
||||
}
|
||||
}
|
||||
|
||||
@@ -330,6 +330,13 @@ impl RoleNameTracker {
|
||||
}
|
||||
}
|
||||
|
||||
/// Max iframe nesting depth `take_snapshot` expands. Embedded payment/checkout
|
||||
/// widgets nest a few frames deep (e.g. AdSense → payments.google.com → an inner
|
||||
/// form frame); expanding past the first level is what gives those inner refs a
|
||||
/// `frame_id` so clicks resolve into the right frame (issue #36). Capped to keep
|
||||
/// a pathological frame tree from blowing up the snapshot.
|
||||
const MAX_IFRAME_DEPTH: usize = 3;
|
||||
|
||||
pub async fn take_snapshot(
|
||||
client: &CdpClient,
|
||||
session_id: &str,
|
||||
@@ -337,6 +344,28 @@ pub async fn take_snapshot(
|
||||
ref_map: &mut RefMap,
|
||||
frame_id: Option<&str>,
|
||||
iframe_sessions: &HashMap<String, String>,
|
||||
) -> Result<String, String> {
|
||||
take_snapshot_at_depth(
|
||||
client,
|
||||
session_id,
|
||||
options,
|
||||
ref_map,
|
||||
frame_id,
|
||||
iframe_sessions,
|
||||
0,
|
||||
)
|
||||
.await
|
||||
}
|
||||
|
||||
#[allow(clippy::too_many_arguments)]
|
||||
async fn take_snapshot_at_depth(
|
||||
client: &CdpClient,
|
||||
session_id: &str,
|
||||
options: &SnapshotOptions,
|
||||
ref_map: &mut RefMap,
|
||||
frame_id: Option<&str>,
|
||||
iframe_sessions: &HashMap<String, String>,
|
||||
depth: usize,
|
||||
) -> Result<String, String> {
|
||||
client
|
||||
.send_command_no_params("DOM.enable", Some(session_id))
|
||||
@@ -606,10 +635,11 @@ pub async fn take_snapshot(
|
||||
}
|
||||
|
||||
// Recurse into child iframes: for each Iframe node with a backend_node_id,
|
||||
// resolve the child frame ID and take a snapshot of its content.
|
||||
// We only recurse from the main frame (frame_id == None) to avoid
|
||||
// unbounded depth; nested iframes within iframes are not expanded.
|
||||
if frame_id.is_none() {
|
||||
// resolve the child frame ID and snapshot its content. Recurse to
|
||||
// MAX_IFRAME_DEPTH (not just the main frame) so refs inside nested
|
||||
// payment/checkout widgets get a `frame_id` and clicks resolve into the right
|
||||
// frame (issue #36); the cap bounds a pathological frame tree.
|
||||
if depth < MAX_IFRAME_DEPTH {
|
||||
let mut iframe_snapshots: Vec<(String, String)> = Vec::new(); // (ref_id, child_snapshot)
|
||||
for node in tree_nodes.iter() {
|
||||
if node.role != "Iframe" || !node.has_ref {
|
||||
@@ -622,13 +652,14 @@ pub async fn take_snapshot(
|
||||
if let Ok(child_fid) = resolve_iframe_frame_id(client, session_id, bid).await {
|
||||
// Snapshot the child frame; errors are silently ignored
|
||||
// (e.g. cross-origin iframes)
|
||||
if let Ok(child_text) = Box::pin(take_snapshot(
|
||||
if let Ok(child_text) = Box::pin(take_snapshot_at_depth(
|
||||
client,
|
||||
session_id,
|
||||
options,
|
||||
ref_map,
|
||||
Some(&child_fid),
|
||||
iframe_sessions,
|
||||
depth + 1,
|
||||
))
|
||||
.await
|
||||
{
|
||||
|
||||
@@ -0,0 +1,28 @@
|
||||
<!doctype html>
|
||||
<html>
|
||||
<head>
|
||||
<meta charset="utf-8" />
|
||||
<title>iframe button probe</title>
|
||||
</head>
|
||||
<body>
|
||||
<h1>iframe button probe</h1>
|
||||
<iframe
|
||||
id="frame"
|
||||
width="320"
|
||||
height="140"
|
||||
srcdoc="
|
||||
<!doctype html>
|
||||
<html>
|
||||
<body style='margin:24px'>
|
||||
<button id='b' style='padding:24px;font-size:22px'>save</button>
|
||||
<script>
|
||||
document.getElementById('b').addEventListener('click', function (e) {
|
||||
this.textContent = 'clicked:' + e.isTrusted;
|
||||
});
|
||||
</script>
|
||||
</body>
|
||||
</html>
|
||||
"
|
||||
></iframe>
|
||||
</body>
|
||||
</html>
|
||||
+152
-14
@@ -186,6 +186,26 @@ pub fn print_response_with_opts(resp: &Response, action: Option<&str>, opts: &Ou
|
||||
}
|
||||
|
||||
if let Some(data) = &resp.data {
|
||||
// Auto-trigger: when you land on / read a page whose domain has site
|
||||
// adapters, surface them so the agent pulls structured data via
|
||||
// `chrome-use site <name>/<cmd>` instead of scraping the DOM. (In --json
|
||||
// mode this same info rides along in the `siteAdapters` field above.)
|
||||
if let Some(hint) = data.get("siteAdapters") {
|
||||
let domain = hint.get("domain").and_then(|v| v.as_str()).unwrap_or("");
|
||||
let cmds: Vec<&str> = hint
|
||||
.get("commands")
|
||||
.and_then(|v| v.as_array())
|
||||
.map(|a| a.iter().filter_map(|v| v.as_str()).collect())
|
||||
.unwrap_or_default();
|
||||
if !cmds.is_empty() {
|
||||
eprintln!("💡 site adapters for {domain} — prefer these for structured data:");
|
||||
eprintln!(" {}", color::dim(&cmds.join(", ")));
|
||||
eprintln!(
|
||||
" {}",
|
||||
color::dim(&format!("e.g. chrome-use site {} --json", cmds[0]))
|
||||
);
|
||||
}
|
||||
}
|
||||
// A click that opened a new tab: surface it so the agent doesn't read the
|
||||
// unchanged old page as a failed click (issue #24-A).
|
||||
if let Some(opened) = data.get("openedTab") {
|
||||
@@ -228,13 +248,28 @@ pub fn print_response_with_opts(resp: &Response, action: Option<&str>, opts: &Ou
|
||||
// because its response carries `url`/`title`, which later generic
|
||||
// renderers would otherwise swallow.
|
||||
if action == Some("cf_status") {
|
||||
let challenged = data.get("challenged").and_then(|v| v.as_bool()).unwrap_or(false);
|
||||
let rec = data.get("recommendation").and_then(|v| v.as_str()).unwrap_or("?");
|
||||
let challenged = data
|
||||
.get("challenged")
|
||||
.and_then(|v| v.as_bool())
|
||||
.unwrap_or(false);
|
||||
let rec = data
|
||||
.get("recommendation")
|
||||
.and_then(|v| v.as_str())
|
||||
.unwrap_or("?");
|
||||
let cl = data.get("clearance");
|
||||
let present = cl.and_then(|c| c.get("present")).and_then(|v| v.as_bool()).unwrap_or(false);
|
||||
let expired = cl.and_then(|c| c.get("expired")).and_then(|v| v.as_bool()).unwrap_or(false);
|
||||
let present = cl
|
||||
.and_then(|c| c.get("present"))
|
||||
.and_then(|v| v.as_bool())
|
||||
.unwrap_or(false);
|
||||
let expired = cl
|
||||
.and_then(|c| c.get("expired"))
|
||||
.and_then(|v| v.as_bool())
|
||||
.unwrap_or(false);
|
||||
let expires_in = cl.and_then(|c| c.get("expiresIn")).and_then(|v| v.as_i64());
|
||||
let device = data.get("deviceVerified").and_then(|v| v.as_bool()).unwrap_or(false);
|
||||
let device = data
|
||||
.get("deviceVerified")
|
||||
.and_then(|v| v.as_bool())
|
||||
.unwrap_or(false);
|
||||
|
||||
let (icon, headline) = match rec {
|
||||
"proceed" => (color::success_indicator().to_string(), "cleared — no challenge, proceed"),
|
||||
@@ -243,7 +278,10 @@ pub fn print_response_with_opts(resp: &Response, action: Option<&str>, opts: &Ou
|
||||
_ => (color::cyan("•").to_string(), "unknown"),
|
||||
};
|
||||
println!("{} {}", icon, headline);
|
||||
println!(" challenged: {}", if challenged { "yes" } else { "no" });
|
||||
println!(
|
||||
" challenged: {}",
|
||||
if challenged { "yes" } else { "no" }
|
||||
);
|
||||
let cl_desc = if !present {
|
||||
"absent".to_string()
|
||||
} else if expired {
|
||||
@@ -254,7 +292,14 @@ pub fn print_response_with_opts(resp: &Response, action: Option<&str>, opts: &Ou
|
||||
"present (session)".to_string()
|
||||
};
|
||||
println!(" cf_clearance: {}", cl_desc);
|
||||
println!(" device trusted: {}", if device { "yes (CF_VERIFIED_DEVICE)" } else { "no" });
|
||||
println!(
|
||||
" device trusted: {}",
|
||||
if device {
|
||||
"yes (CF_VERIFIED_DEVICE)"
|
||||
} else {
|
||||
"no"
|
||||
}
|
||||
);
|
||||
return;
|
||||
}
|
||||
|
||||
@@ -382,7 +427,11 @@ pub fn print_response_with_opts(resp: &Response, action: Option<&str>, opts: &Ou
|
||||
let count = list.len();
|
||||
println!(
|
||||
"{}",
|
||||
color::bold(&format!("{} frame{}", count, if count == 1 { "" } else { "s" }))
|
||||
color::bold(&format!(
|
||||
"{} frame{}",
|
||||
count,
|
||||
if count == 1 { "" } else { "s" }
|
||||
))
|
||||
);
|
||||
for f in list {
|
||||
let idx = f.get("index").and_then(|v| v.as_i64()).unwrap_or(0);
|
||||
@@ -440,6 +489,17 @@ pub fn print_response_with_opts(resp: &Response, action: Option<&str>, opts: &Ou
|
||||
println!("y: {}", y);
|
||||
println!("width: {}", w);
|
||||
println!("height: {}", h);
|
||||
if let (Some(cx), Some(cy)) = (
|
||||
obj.get("centerX").and_then(|v| v.as_i64()),
|
||||
obj.get("centerY").and_then(|v| v.as_i64()),
|
||||
) {
|
||||
// Echoed in click-ready CSS px so the agent can paste straight
|
||||
// into `click <centerX> <centerY>` (issue #43).
|
||||
println!("center: {} {}", cx, cy);
|
||||
}
|
||||
if let Some(iv) = obj.get("inViewport").and_then(|v| v.as_bool()) {
|
||||
println!("inViewport: {}", iv);
|
||||
}
|
||||
}
|
||||
return;
|
||||
}
|
||||
@@ -1441,9 +1501,20 @@ Examples:
|
||||
chrome-use fill - Clear and fill an input field
|
||||
|
||||
Usage: chrome-use fill <selector> <text>
|
||||
chrome-use fill <selector> --file <path>
|
||||
chrome-use fill <selector> --stdin
|
||||
|
||||
Clears the input field and fills it with the specified text.
|
||||
This replaces any existing content in the field.
|
||||
Clears the field and fills it with the text, replacing existing content.
|
||||
Works on rich editors too (issue #41): CodeMirror 5, Monaco, ProseMirror and
|
||||
plain contenteditable are detected and set via their own API / input events,
|
||||
not a raw `.value` write — and the response echoes which `engine` was used.
|
||||
For framework inputs (React/Vue/Angular) the value goes through the native
|
||||
setter so the form registers it (no more "pristine" Save no-ops).
|
||||
|
||||
Options:
|
||||
--file <path> Read the value from a UTF-8 file (large/multiline text,
|
||||
backticks/quotes/newlines/non-ASCII — no shell escaping)
|
||||
--stdin Read the value from stdin
|
||||
|
||||
Global Options:
|
||||
--json Output as JSON
|
||||
@@ -1452,7 +1523,8 @@ Global Options:
|
||||
Examples:
|
||||
chrome-use fill "#email" "user@example.com"
|
||||
chrome-use fill @e3 "Hello World"
|
||||
chrome-use fill "input[name='search']" "query"
|
||||
chrome-use fill ".CodeMirror" --file ./article.md # set a CodeMirror editor
|
||||
cat post.md | chrome-use fill @e7 --stdin
|
||||
"##
|
||||
}
|
||||
"type" => {
|
||||
@@ -1464,6 +1536,12 @@ Usage: chrome-use type <selector> <text>
|
||||
Types text into the specified element character by character.
|
||||
Unlike fill, this does not clear existing content first.
|
||||
|
||||
Options:
|
||||
--key-events Send real per-character keyDown/keyUp instead of
|
||||
(alias --keys) Input.insertText. Use for autocomplete / combobox fields
|
||||
that only react to key events — e.g. a postal-code box
|
||||
that auto-fills city/prefecture, or Google Places.
|
||||
|
||||
Global Options:
|
||||
--json Output as JSON
|
||||
--session <name> Use specific session
|
||||
@@ -1471,6 +1549,7 @@ Global Options:
|
||||
Examples:
|
||||
chrome-use type "#search" "hello"
|
||||
chrome-use type @e2 "additional text"
|
||||
chrome-use type @e5 "201-0001" --key-events # trigger the address autocomplete
|
||||
|
||||
See Also:
|
||||
For typing into contenteditable editors (Lexical, ProseMirror, etc.)
|
||||
@@ -1735,12 +1814,23 @@ Usage: chrome-use scroll [direction] [amount] [options]
|
||||
|
||||
Scrolls the page or a specific element in the specified direction.
|
||||
|
||||
Without --selector, scroll dispatches a real (isTrusted) mouse wheel at a
|
||||
viewport coordinate, so it scrolls whatever container is under the pointer —
|
||||
including cross-origin iframes (Google Payments, Stripe, embedded checkout/KYC)
|
||||
that plain page scroll can't reach.
|
||||
|
||||
Arguments:
|
||||
direction up, down, left, right (default: down)
|
||||
amount Pixels to scroll (default: 300)
|
||||
|
||||
Options:
|
||||
-s, --selector <sel> CSS selector for a scrollable container
|
||||
-s, --selector <sel> CSS selector for a scrollable container (same-origin)
|
||||
--at <x,y> Dispatch the wheel at this viewport pixel (read it from a
|
||||
screenshot) — precise way into a cross-origin iframe
|
||||
--frame <n> Scroll the n-th frame from `chrome-use frames` (wheel at
|
||||
that frame's center)
|
||||
|
||||
Without --selector/--at/--frame the wheel lands at the viewport center.
|
||||
|
||||
Global Options:
|
||||
--json Output as JSON
|
||||
@@ -1752,6 +1842,8 @@ Examples:
|
||||
chrome-use scroll up 200
|
||||
chrome-use scroll left 100
|
||||
chrome-use scroll down 500 --selector "div.scroll-container"
|
||||
chrome-use scroll down 700 --at 640,400 # wheel at a pixel over an iframe
|
||||
chrome-use scroll down 700 --frame 2 # scroll frame 2 from `frames`
|
||||
"##
|
||||
}
|
||||
"scrollintoview" | "scrollinto" => {
|
||||
@@ -1834,6 +1926,13 @@ Options:
|
||||
--full, -f Capture full page (not just viewport)
|
||||
[selector] Capture just an element (CSS or @ref), e.g. `screenshot ".header" h.png`
|
||||
--clip <x,y,w,h> Capture a pixel region, e.g. `screenshot --clip 0,0,200,40 corner.png`
|
||||
--max-width <px> Downscale so the image's width ≤ px (preserves aspect)
|
||||
--max-height <px> Downscale so the image's height ≤ px
|
||||
--scale <0..1> Downscale by a factor, e.g. 0.5 (DPR-1, so screenshot px
|
||||
line up 1:1 with `click x y`)
|
||||
Default: capped at 2000px longest edge unless overridden
|
||||
(AGENT_BROWSER_SCREENSHOT_MAX_EDGE; 0 disables). Annotated
|
||||
shots are never downscaled, so ref overlays stay aligned.
|
||||
--annotate Overlay numbered labels on interactive elements.
|
||||
Each label [N] corresponds to ref @eN from snapshot.
|
||||
Prints a legend mapping labels to element roles/names.
|
||||
@@ -1856,6 +1955,8 @@ Examples:
|
||||
chrome-use screenshot --full ./full-page.png
|
||||
chrome-use screenshot ".header .indicator" corner.png # just one element
|
||||
chrome-use screenshot --clip 1600,0,200,40 corner.png # a pixel region
|
||||
chrome-use screenshot --scale 0.5 ./half.png # DPR-1: screenshot px == click px
|
||||
chrome-use screenshot --max-width 1400 ./shot.png # cap width for image readers
|
||||
chrome-use screenshot --annotate # Labeled screenshot + legend
|
||||
chrome-use screenshot --annotate ./page.png # Save annotated screenshot
|
||||
chrome-use screenshot --annotate --json # JSON output with annotations
|
||||
@@ -3194,7 +3295,8 @@ Core Commands:
|
||||
click <sel|x y> Click element/@ref, or a viewport coordinate
|
||||
dblclick <sel> Double-click element
|
||||
type <sel> <text> Type into element
|
||||
fill <sel> <text> Clear and fill
|
||||
fill <sel> <text> Clear and fill (handles CodeMirror/Monaco/ProseMirror/
|
||||
contenteditable; `--file <path>`/`--stdin` for large text)
|
||||
press <key> [--hold <ms>] Press key (Enter, Tab, Control+a). --hold keeps it
|
||||
down <ms> then releases — precise (in-daemon), for
|
||||
games/charge: `press d --hold 800`
|
||||
@@ -3214,11 +3316,19 @@ Core Commands:
|
||||
scroll <dir> [px] Scroll (up/down/left/right)
|
||||
scrollintoview <sel> Scroll element into view
|
||||
wait <sel|ms> Wait for element or time
|
||||
screenshot [path] Take screenshot
|
||||
screenshot [path] Take screenshot (auto-downscaled to ≤2000px long edge;
|
||||
--max-width/--max-height/--scale to override)
|
||||
pdf <path> Save as PDF
|
||||
canvas list List <canvas> elements (size, type) on the page
|
||||
canvas capture [sel] [path] Save a canvas's rendered pixels to PNG — for
|
||||
WebGL/canvas apps (Figma, games, maps, charts) that
|
||||
expose no DOM. toDataURL, with a screenshot fallback.
|
||||
snapshot Accessibility tree with refs (for AI)
|
||||
eval <js> Run JavaScript
|
||||
connect <port|url> Connect to browser via CDP
|
||||
keep Leave the active tab for the user — exempt it from
|
||||
auto-close/idle cleanup + remove it from the session
|
||||
tab group (so scratch tabs get cleaned, this one stays)
|
||||
close [--all] Close browser (--all closes every session)
|
||||
|
||||
Navigation:
|
||||
@@ -3228,6 +3338,8 @@ Navigation:
|
||||
|
||||
Get Info: chrome-use get <what> [selector]
|
||||
text, html, value, attr <name>, title, url, count, box, styles, cdp-url
|
||||
box <sel> → x,y,width,height,centerX,centerY,inViewport in CSS px (feed
|
||||
centerX/centerY into `click x y`); value reads CodeMirror/Monaco too
|
||||
text (no selector = whole page, all frames), text --main, frames (list)
|
||||
|
||||
Check State: chrome-use is <what> <selector>
|
||||
@@ -3270,6 +3382,10 @@ Tabs:
|
||||
stable targetId, no reload — preserves in-page state
|
||||
open <url> --reuse-tab Reuse an existing tab on that URL instead of spawning
|
||||
a duplicate (matches origin+path; preserves state)
|
||||
adopt <url|targetId> Read a PRE-EXISTING tab (the user's own, or another
|
||||
session's) WITHOUT opening a new one — matches by URL
|
||||
substring or stable targetId, then drives it. e.g.
|
||||
`adopt "github.com/owner/repo"`
|
||||
|
||||
Diff:
|
||||
diff snapshot Compare current vs last snapshot
|
||||
@@ -3317,6 +3433,14 @@ Batch:
|
||||
batch [--bail] ["cmd" ...] Execute multiple commands sequentially (args or stdin)
|
||||
--bail stops on first error (default: continue all)
|
||||
|
||||
Site adapters: turn a website into a structured-data CLI (runs as you, in your tab)
|
||||
site update Fetch the community adapter pack into ~/.chrome-use/sites
|
||||
site list List installed adapters (name/cmd)
|
||||
site info <name>/<cmd> Show an adapter's @meta (args, domain, capabilities)
|
||||
site <name>/<cmd> [args] Run an adapter: navigate to its site + return JSON
|
||||
e.g. site github/issues epiral/repo, site reddit/search rust
|
||||
Positional args fill declared args in order; --key value overrides
|
||||
|
||||
Auth Vault:
|
||||
auth save <name> [opts] Save auth profile (--url, --username, --password/--password-stdin)
|
||||
auth login <name> Login using saved credentials (waits for form fields)
|
||||
@@ -3331,11 +3455,21 @@ Confirmation:
|
||||
Sessions:
|
||||
session Show current session name
|
||||
session list List active sessions
|
||||
session stop [name] Stop one session daemon (default: current) — graceful,
|
||||
closes the tabs it created
|
||||
session prune Stop ALL session daemons now (closes their tabs; they
|
||||
respawn clean on next use). For clearing idle daemons.
|
||||
sessions List running session daemons (alias of daemon status)
|
||||
daemon status List running session daemons (+ relay state)
|
||||
daemon restart Kill all session daemons; keeps the extension relay
|
||||
up. Clears stale/cross-leaked state after an upgrade.
|
||||
|
||||
Lifecycle: each --session <name> spawns a background daemon that drives that
|
||||
session's tabs. A daemon auto-shuts-down after 10 min idle (no commands) —
|
||||
AGENT_BROWSER_IDLE_TIMEOUT_MS overrides, 0 disables — and on shutdown closes
|
||||
the scratch tabs IT created (its tab group). Use `keep` to leave a tab for the
|
||||
user (exempt from auto-close), `session stop/prune` to reclaim now.
|
||||
|
||||
Chat (AI):
|
||||
chat <message> Send a natural language instruction (single-shot)
|
||||
chat Start interactive chat (REPL mode when stdin is a TTY)
|
||||
@@ -3528,6 +3662,9 @@ iOS Simulator (requires Xcode and Appium):
|
||||
chrome-use -p ios device list # List simulators
|
||||
chrome-use -p ios swipe up # Swipe gesture
|
||||
chrome-use -p ios tap @e1 # Touch element
|
||||
|
||||
Hit a bug or rough edge? A 30-second issue genuinely sharpens this tool:
|
||||
https://github.com/leeguooooo/chrome-use/issues
|
||||
"#
|
||||
);
|
||||
}
|
||||
@@ -3610,6 +3747,7 @@ fn print_screenshot_diff(data: &serde_json::Map<String, serde_json::Value>) {
|
||||
|
||||
pub fn print_version() {
|
||||
println!("chrome-use {}", env!("CARGO_PKG_VERSION"));
|
||||
println!("report bugs / rough edges: https://github.com/leeguooooo/chrome-use/issues");
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
|
||||
+465
@@ -0,0 +1,465 @@
|
||||
//! Site adapters: turn any website into a structured-data CLI by running a small
|
||||
//! per-command JS adapter inside your real, logged-in browser tab (it reuses the
|
||||
//! site's cookies / same-origin fetch / its own webpack modules — the site thinks
|
||||
//! it's you, because it is).
|
||||
//!
|
||||
//! The adapter format is the community **bb-sites** convention
|
||||
//! (<https://github.com/epiral/bb-sites>): one `.js` file per command, a
|
||||
//! `/* @meta {...} */` JSON header (name, description, domain, args), then an
|
||||
//! `async function(args){ ... return {...} }`. chrome-use ships none of those
|
||||
//! adapters — `chrome-use site update` fetches the upstream repo at runtime into
|
||||
//! `~/.chrome-use/sites` (like a package manager pulling a dependency), so the
|
||||
//! adapters stay the property of their authors. Running an adapter navigates to
|
||||
//! its `@meta.domain` and `eval`s the function in the site's own logged-in page.
|
||||
|
||||
use std::path::PathBuf;
|
||||
|
||||
use serde_json::Value;
|
||||
|
||||
const SITES_ZIP_URL: &str = "https://github.com/epiral/bb-sites/archive/refs/heads/main.zip";
|
||||
|
||||
/// `~/.chrome-use/sites` — where synced adapters live.
|
||||
pub fn sites_dir() -> Option<PathBuf> {
|
||||
dirs_home().map(|h| h.join(".chrome-use").join("sites"))
|
||||
}
|
||||
|
||||
fn dirs_home() -> Option<PathBuf> {
|
||||
std::env::var_os("HOME").map(PathBuf::from)
|
||||
}
|
||||
|
||||
/// Parsed adapter: its `@meta` JSON and the raw `async function(args){...}` source.
|
||||
pub struct Adapter {
|
||||
pub meta: Value,
|
||||
pub func_src: String,
|
||||
/// The adapter's declared `args` keys in DECLARATION order. Parsed from the
|
||||
/// raw @meta text because `serde_json` sorts object keys alphabetically, which
|
||||
/// would otherwise scramble positional-arg mapping for multi-arg adapters.
|
||||
pub arg_order: Vec<String>,
|
||||
}
|
||||
|
||||
impl Adapter {
|
||||
pub fn domain(&self) -> Option<&str> {
|
||||
self.meta.get("domain").and_then(|v| v.as_str())
|
||||
}
|
||||
}
|
||||
|
||||
/// Load `<sites>/<name>/<cmd>.js`, splitting the `/* @meta {...} */` header from
|
||||
/// the function body. `spec` is `name/cmd`.
|
||||
pub fn load_adapter(spec: &str) -> Result<Adapter, String> {
|
||||
let (name, cmd) = spec
|
||||
.split_once('/')
|
||||
.ok_or_else(|| format!("site: expected <name>/<command>, got `{spec}`"))?;
|
||||
if name.is_empty()
|
||||
|| cmd.is_empty()
|
||||
|| name.contains("..")
|
||||
|| cmd.contains("..")
|
||||
|| name.contains('/')
|
||||
|| cmd.contains('/')
|
||||
{
|
||||
return Err(format!("site: invalid adapter spec `{spec}`"));
|
||||
}
|
||||
let dir = sites_dir().ok_or("site: cannot resolve home dir")?;
|
||||
let path = dir.join(name).join(format!("{cmd}.js"));
|
||||
if !path.exists() {
|
||||
return Err(format!(
|
||||
"site: adapter `{spec}` not found. Run `chrome-use site update` to sync adapters, \
|
||||
or `chrome-use site list` to see what's installed."
|
||||
));
|
||||
}
|
||||
let raw = std::fs::read_to_string(&path).map_err(|e| format!("site: read {spec}: {e}"))?;
|
||||
parse_adapter(&raw, spec)
|
||||
}
|
||||
|
||||
/// Split the `@meta` JSON block and the function source from an adapter file.
|
||||
pub fn parse_adapter(raw: &str, spec: &str) -> Result<Adapter, String> {
|
||||
let start = raw
|
||||
.find("@meta")
|
||||
.and_then(|i| raw[i..].find('{').map(|j| i + j))
|
||||
.ok_or_else(|| format!("site: {spec} missing /* @meta {{...}} */ header"))?;
|
||||
// Find the matching close brace for the @meta object (brace-count, string-aware).
|
||||
let bytes = raw.as_bytes();
|
||||
let mut depth = 0i32;
|
||||
let mut in_str = false;
|
||||
let mut esc = false;
|
||||
let mut end = None;
|
||||
for (k, &b) in bytes.iter().enumerate().skip(start) {
|
||||
if in_str {
|
||||
if esc {
|
||||
esc = false;
|
||||
} else if b == b'\\' {
|
||||
esc = true;
|
||||
} else if b == b'"' {
|
||||
in_str = false;
|
||||
}
|
||||
continue;
|
||||
}
|
||||
match b {
|
||||
b'"' => in_str = true,
|
||||
b'{' => depth += 1,
|
||||
b'}' => {
|
||||
depth -= 1;
|
||||
if depth == 0 {
|
||||
end = Some(k + 1);
|
||||
break;
|
||||
}
|
||||
}
|
||||
_ => {}
|
||||
}
|
||||
}
|
||||
let end = end.ok_or_else(|| format!("site: {spec} @meta header has no closing brace"))?;
|
||||
let meta: Value = serde_json::from_str(&raw[start..end])
|
||||
.map_err(|e| format!("site: {spec} @meta is not valid JSON: {e}"))?;
|
||||
// The function is everything after the meta comment's closing `*/`.
|
||||
let after = raw[end..].find("*/").map(|i| end + i + 2).unwrap_or(end);
|
||||
let func_src = raw[after..].trim().to_string();
|
||||
if func_src.is_empty() {
|
||||
return Err(format!("site: {spec} has no function body after @meta"));
|
||||
}
|
||||
let arg_order = arg_order_from_meta(&raw[start..end]);
|
||||
Ok(Adapter {
|
||||
meta,
|
||||
func_src,
|
||||
arg_order,
|
||||
})
|
||||
}
|
||||
|
||||
/// Extract the `args` object's keys in DECLARATION order from the raw @meta JSON
|
||||
/// text (serde sorts them, losing order). Brace/string-aware: finds the `"args"`
|
||||
/// value object and collects only its top-level keys.
|
||||
fn arg_order_from_meta(meta_json: &str) -> Vec<String> {
|
||||
let bytes = meta_json.as_bytes();
|
||||
// Locate the `"args"` key, then the `{` that opens its value object.
|
||||
let Some(args_pos) = meta_json.find("\"args\"") else {
|
||||
return Vec::new();
|
||||
};
|
||||
let Some(brace_off) = meta_json[args_pos..].find('{') else {
|
||||
return Vec::new();
|
||||
};
|
||||
let open = args_pos + brace_off;
|
||||
let mut keys = Vec::new();
|
||||
let mut depth = 0i32;
|
||||
let mut in_str = false;
|
||||
let mut esc = false;
|
||||
let mut cur = String::new();
|
||||
let mut last_str: Option<String> = None;
|
||||
for &b in bytes.iter().skip(open) {
|
||||
if in_str {
|
||||
if esc {
|
||||
esc = false;
|
||||
} else if b == b'\\' {
|
||||
esc = true;
|
||||
} else if b == b'"' {
|
||||
in_str = false;
|
||||
last_str = Some(std::mem::take(&mut cur));
|
||||
} else {
|
||||
cur.push(b as char);
|
||||
}
|
||||
continue;
|
||||
}
|
||||
match b {
|
||||
b'"' => in_str = true,
|
||||
b'{' => depth += 1,
|
||||
b'}' => {
|
||||
depth -= 1;
|
||||
if depth == 0 {
|
||||
break; // end of the args object
|
||||
}
|
||||
}
|
||||
// A `:` at depth 1 means the preceding string was a key of `args`.
|
||||
b':' if depth == 1 => {
|
||||
if let Some(k) = last_str.take() {
|
||||
keys.push(k);
|
||||
}
|
||||
}
|
||||
_ => {}
|
||||
}
|
||||
}
|
||||
keys
|
||||
}
|
||||
|
||||
/// Build the JS to eval: `(<adapter function>)(<args JSON>)`. The adapter's
|
||||
/// `async function(args)` returns a promise; chrome-use's eval awaits it.
|
||||
pub fn build_eval(adapter: &Adapter, args: &Value) -> String {
|
||||
let args_json = serde_json::to_string(args).unwrap_or_else(|_| "{}".to_string());
|
||||
format!("({})({})", adapter.func_src, args_json)
|
||||
}
|
||||
|
||||
/// List installed adapters as `name/cmd` strings (sorted).
|
||||
pub fn list_adapters() -> Result<Vec<String>, String> {
|
||||
let dir = sites_dir().ok_or("site: cannot resolve home dir")?;
|
||||
if !dir.exists() {
|
||||
return Ok(Vec::new());
|
||||
}
|
||||
let mut out = Vec::new();
|
||||
for site in std::fs::read_dir(&dir)
|
||||
.map_err(|e| e.to_string())?
|
||||
.flatten()
|
||||
{
|
||||
if !site.path().is_dir() {
|
||||
continue;
|
||||
}
|
||||
let name = site.file_name().to_string_lossy().to_string();
|
||||
for cmd in std::fs::read_dir(site.path())
|
||||
.map_err(|e| e.to_string())?
|
||||
.flatten()
|
||||
{
|
||||
let p = cmd.path();
|
||||
if p.extension().and_then(|e| e.to_str()) == Some("js") {
|
||||
if let Some(stem) = p.file_stem().and_then(|s| s.to_str()) {
|
||||
out.push(format!("{name}/{stem}"));
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
out.sort();
|
||||
Ok(out)
|
||||
}
|
||||
|
||||
/// Download the bb-sites repo zip and extract its adapters into `~/.chrome-use/sites`.
|
||||
pub async fn update() -> Result<usize, String> {
|
||||
let dir = sites_dir().ok_or("site: cannot resolve home dir")?;
|
||||
let client = reqwest::Client::builder()
|
||||
.user_agent("chrome-use")
|
||||
.build()
|
||||
.map_err(|e| e.to_string())?;
|
||||
let bytes = client
|
||||
.get(SITES_ZIP_URL)
|
||||
.send()
|
||||
.await
|
||||
.map_err(|e| format!("site update: download failed: {e}"))?
|
||||
.error_for_status()
|
||||
.map_err(|e| format!("site update: {e}"))?
|
||||
.bytes()
|
||||
.await
|
||||
.map_err(|e| format!("site update: read body: {e}"))?;
|
||||
|
||||
let cursor = std::io::Cursor::new(bytes);
|
||||
let mut zip = zip::ZipArchive::new(cursor).map_err(|e| format!("site update: bad zip: {e}"))?;
|
||||
std::fs::create_dir_all(&dir).map_err(|e| e.to_string())?;
|
||||
let mut count = 0usize;
|
||||
for i in 0..zip.len() {
|
||||
let mut f = zip.by_index(i).map_err(|e| e.to_string())?;
|
||||
let Some(enclosed) = f.enclosed_name() else {
|
||||
continue;
|
||||
};
|
||||
// Strip the top-level `bb-sites-main/` component from the archive path.
|
||||
let rel: PathBuf = enclosed.components().skip(1).collect();
|
||||
if rel.as_os_str().is_empty() {
|
||||
continue;
|
||||
}
|
||||
let out = dir.join(&rel);
|
||||
if f.is_dir() {
|
||||
let _ = std::fs::create_dir_all(&out);
|
||||
continue;
|
||||
}
|
||||
if let Some(parent) = out.parent() {
|
||||
let _ = std::fs::create_dir_all(parent);
|
||||
}
|
||||
let mut buf = Vec::new();
|
||||
std::io::copy(&mut f, &mut buf).map_err(|e| e.to_string())?;
|
||||
std::fs::write(&out, &buf).map_err(|e| e.to_string())?;
|
||||
if out.extension().and_then(|e| e.to_str()) == Some("js") {
|
||||
count += 1;
|
||||
}
|
||||
}
|
||||
// Build the domain→adapters index and stamp the sync time so navigation can
|
||||
// suggest adapters (auto-trigger) and `needs_refresh` can pace re-syncs.
|
||||
write_domain_index(&dir);
|
||||
if let Some(p) = last_update_path() {
|
||||
let _ = std::fs::write(p, now_secs().to_string());
|
||||
}
|
||||
Ok(count)
|
||||
}
|
||||
|
||||
/// `~/.chrome-use/sites/.last_update` — unix-seconds marker of the last sync.
|
||||
fn last_update_path() -> Option<PathBuf> {
|
||||
sites_dir().map(|d| d.join(".last_update"))
|
||||
}
|
||||
|
||||
/// `~/.chrome-use/sites/.index.json` — `{ "github.com": ["github/issues", …], … }`,
|
||||
/// built on `update` so navigation can look up adapters by domain without parsing
|
||||
/// all ~145 adapter files on every command.
|
||||
fn index_path() -> Option<PathBuf> {
|
||||
sites_dir().map(|d| d.join(".index.json"))
|
||||
}
|
||||
|
||||
fn now_secs() -> u64 {
|
||||
std::time::SystemTime::now()
|
||||
.duration_since(std::time::UNIX_EPOCH)
|
||||
.map(|d| d.as_secs())
|
||||
.unwrap_or(0)
|
||||
}
|
||||
|
||||
/// Parse every installed adapter and write the domain→adapters index. Within a
|
||||
/// domain, read-only adapters are listed first (then alphabetical) so the
|
||||
/// auto-suggested example leads with a safe read, not a write action.
|
||||
fn write_domain_index(dir: &std::path::Path) {
|
||||
let mut by_domain: std::collections::BTreeMap<String, Vec<(bool, String)>> = Default::default();
|
||||
for spec in list_adapters().unwrap_or_default() {
|
||||
if let Ok(a) = load_adapter(&spec) {
|
||||
if let Some(d) = a.domain() {
|
||||
let read_only = a
|
||||
.meta
|
||||
.get("readOnly")
|
||||
.and_then(|v| v.as_bool())
|
||||
.unwrap_or(false);
|
||||
by_domain
|
||||
.entry(d.to_string())
|
||||
.or_default()
|
||||
.push((read_only, spec));
|
||||
}
|
||||
}
|
||||
}
|
||||
let ordered: std::collections::BTreeMap<String, Vec<String>> = by_domain
|
||||
.into_iter()
|
||||
.map(|(domain, mut v)| {
|
||||
// read-only (true) first, then by spec name
|
||||
v.sort_by(|a, b| b.0.cmp(&a.0).then_with(|| a.1.cmp(&b.1)));
|
||||
(domain, v.into_iter().map(|(_, s)| s).collect())
|
||||
})
|
||||
.collect();
|
||||
if let Ok(json) = serde_json::to_string(&ordered) {
|
||||
let _ = std::fs::write(dir.join(".index.json"), json);
|
||||
}
|
||||
}
|
||||
|
||||
const DEFAULT_TTL_DAYS: u64 = 7;
|
||||
|
||||
/// Whether the adapter pack should be (re)synced: true on first use (nothing
|
||||
/// installed) or when the last sync is older than the TTL. Disabled by
|
||||
/// `AGENT_BROWSER_SITES_NO_AUTO_UPDATE=1`; TTL overridable via
|
||||
/// `AGENT_BROWSER_SITES_TTL_DAYS` (0 = always).
|
||||
pub fn needs_refresh() -> bool {
|
||||
if std::env::var_os("AGENT_BROWSER_SITES_NO_AUTO_UPDATE").is_some() {
|
||||
return false;
|
||||
}
|
||||
let Some(dir) = sites_dir() else {
|
||||
return false;
|
||||
};
|
||||
// First use: no adapters installed yet.
|
||||
if list_adapters().map(|l| l.is_empty()).unwrap_or(true) {
|
||||
let _ = &dir;
|
||||
return true;
|
||||
}
|
||||
let ttl_days = std::env::var("AGENT_BROWSER_SITES_TTL_DAYS")
|
||||
.ok()
|
||||
.and_then(|s| s.parse::<u64>().ok())
|
||||
.unwrap_or(DEFAULT_TTL_DAYS);
|
||||
let ttl = ttl_days.saturating_mul(86_400);
|
||||
match last_update_path().and_then(|p| std::fs::read_to_string(p).ok()) {
|
||||
Some(s) => match s.trim().parse::<u64>() {
|
||||
Ok(ts) => now_secs().saturating_sub(ts) >= ttl,
|
||||
Err(_) => true,
|
||||
},
|
||||
None => true, // no marker → treat as stale
|
||||
}
|
||||
}
|
||||
|
||||
/// Adapters whose `@meta.domain` matches `host` (exact, or `host` is a subdomain
|
||||
/// of it) — for auto-suggesting `site` commands when you land on a known site.
|
||||
/// Reads the prebuilt `.index.json`; empty if the pack isn't synced yet.
|
||||
pub fn adapters_for_domain(host: &str) -> Vec<String> {
|
||||
let host = host.trim_start_matches("www.");
|
||||
let Some(raw) = index_path().and_then(|p| std::fs::read_to_string(p).ok()) else {
|
||||
return Vec::new();
|
||||
};
|
||||
let Ok(idx) = serde_json::from_str::<std::collections::BTreeMap<String, Vec<String>>>(&raw)
|
||||
else {
|
||||
return Vec::new();
|
||||
};
|
||||
// Preserve the index's per-domain ordering (read-only adapters first); just
|
||||
// dedup if a host somehow matches multiple domain keys.
|
||||
let mut out: Vec<String> = Vec::new();
|
||||
for (domain, specs) in idx {
|
||||
let d = domain.trim_start_matches("www.");
|
||||
if host == d || host.ends_with(&format!(".{d}")) {
|
||||
for s in specs {
|
||||
if !out.contains(&s) {
|
||||
out.push(s);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
out
|
||||
}
|
||||
|
||||
/// Map CLI args to the adapter's `args` object. Positional args fill the adapter's
|
||||
/// declared `args` keys in order; `--key value` overrides by name. The adapter
|
||||
/// validates required args itself.
|
||||
pub fn map_args(adapter: &Adapter, positional: &[String], named: &[(String, String)]) -> Value {
|
||||
let mut obj = serde_json::Map::new();
|
||||
// Positional args fill the adapter's declared args in DECLARATION order
|
||||
// (`arg_order`), not serde's alphabetized key order — otherwise a 2-arg
|
||||
// adapter like `{projectId, path}` would map positionals to `{path, projectId}`.
|
||||
for (i, val) in positional.iter().enumerate() {
|
||||
if let Some(k) = adapter.arg_order.get(i) {
|
||||
obj.insert(k.clone(), Value::String(val.clone()));
|
||||
}
|
||||
}
|
||||
for (k, v) in named {
|
||||
obj.insert(k.clone(), Value::String(v.clone()));
|
||||
}
|
||||
Value::Object(obj)
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
const SAMPLE: &str = r#"/* @meta
|
||||
{
|
||||
"name": "github/issues",
|
||||
"domain": "github.com",
|
||||
"args": { "repo": {"required": true}, "state": {"required": false} }
|
||||
}
|
||||
*/
|
||||
|
||||
async function(args) { return { repo: args.repo }; }"#;
|
||||
|
||||
#[test]
|
||||
fn parses_meta_and_function() {
|
||||
let a = parse_adapter(SAMPLE, "github/issues").unwrap();
|
||||
assert_eq!(a.domain(), Some("github.com"));
|
||||
assert!(a.func_src.starts_with("async function(args)"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn build_eval_wraps_and_passes_args() {
|
||||
let a = parse_adapter(SAMPLE, "github/issues").unwrap();
|
||||
let args = map_args(
|
||||
&a,
|
||||
&["owner/repo".into()],
|
||||
&[("state".into(), "closed".into())],
|
||||
);
|
||||
let js = build_eval(&a, &args);
|
||||
assert!(js.contains("async function(args)"));
|
||||
assert!(js.contains("\"repo\":\"owner/repo\""));
|
||||
assert!(js.contains("\"state\":\"closed\""));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn rejects_bad_spec() {
|
||||
assert!(load_adapter("noslash").is_err());
|
||||
assert!(load_adapter("../etc/passwd").is_err());
|
||||
}
|
||||
|
||||
// Regression: positional args must follow DECLARATION order, not serde's
|
||||
// alphabetical key order. With `{projectId, path}` (not alphabetical),
|
||||
// `<uuid> <file>` must map projectId←uuid, path←file — not swapped.
|
||||
#[test]
|
||||
fn positional_args_follow_declaration_order_not_alphabetical() {
|
||||
let raw = r#"/* @meta
|
||||
{
|
||||
"name": "claude-design/get-file",
|
||||
"domain": "claude.ai",
|
||||
"args": { "projectId": {"required": true}, "path": {"required": true} }
|
||||
}
|
||||
*/
|
||||
async function(args){ return args; }"#;
|
||||
let a = parse_adapter(raw, "claude-design/get-file").unwrap();
|
||||
assert_eq!(a.arg_order, vec!["projectId", "path"]);
|
||||
let args = map_args(&a, &["the-uuid".into(), "misonote.dc.html".into()], &[]);
|
||||
assert_eq!(args["projectId"], "the-uuid");
|
||||
assert_eq!(args["path"], "misonote.dc.html");
|
||||
}
|
||||
}
|
||||
Binary file not shown.
Binary file not shown.
@@ -81,6 +81,35 @@ async function groupTabInto(tabId, name) {
|
||||
groupIdByName.set(name, gid)
|
||||
}
|
||||
|
||||
// Group-scoped relay isolation hints (issue #40). The relay scopes
|
||||
// Target.getTargets per agent by tab group; report two things in the synthesized
|
||||
// targetInfo so it can attribute each tab:
|
||||
// - abGroup: the tab's Chrome tab-group TITLE (= the owning session name), so
|
||||
// the relay can re-attribute existing tabs after a restart (createTarget
|
||||
// tagging won't re-run for already-open tabs).
|
||||
// - openerTargetId: the targetId of the tab that opened this one, so a pop-up
|
||||
// (window.open / target=_blank / OAuth result) inherits its opener's group
|
||||
// and the agent that opened it can follow it — without foreign tabs leaking.
|
||||
// Best-effort: any failure yields empty strings, which the relay ignores.
|
||||
async function tabScopeHints(tabId) {
|
||||
let openerTargetId = ''
|
||||
let abGroup = ''
|
||||
try {
|
||||
const t = await chrome.tabs.get(tabId)
|
||||
if (t) {
|
||||
if (typeof t.openerTabId === 'number') {
|
||||
const op = tabs.get(t.openerTabId)
|
||||
if (op) openerTargetId = op.targetId
|
||||
}
|
||||
if (t.groupId != null && t.groupId >= 0 && chrome.tabGroups) {
|
||||
const g = await chrome.tabGroups.get(t.groupId).catch(() => null)
|
||||
if (g && g.title) abGroup = g.title
|
||||
}
|
||||
}
|
||||
} catch {}
|
||||
return { openerTargetId, abGroup }
|
||||
}
|
||||
|
||||
function postToHost(msg) {
|
||||
try {
|
||||
if (port) port.postMessage(msg)
|
||||
@@ -126,7 +155,7 @@ function connectHost() {
|
||||
} catch {}
|
||||
// Tell the daemon about everything we already have attached, then attach
|
||||
// anything new.
|
||||
reannounceAttachedTabs()
|
||||
void reannounceAttachedTabs()
|
||||
void attachAllTabs()
|
||||
}
|
||||
|
||||
@@ -140,7 +169,7 @@ async function onHostMessage(msg) {
|
||||
// Daemon (re)connected — (re)attach and announce every tab so it discovers
|
||||
// the user's existing tabs rather than racing an empty target list.
|
||||
if (msg.method === 'attachAll') {
|
||||
reannounceAttachedTabs()
|
||||
void reannounceAttachedTabs()
|
||||
await attachAllTabs()
|
||||
return
|
||||
}
|
||||
@@ -259,6 +288,19 @@ async function handleForwardCdpCommand(msg) {
|
||||
const params = msg?.params?.params || undefined
|
||||
const sessionId = typeof msg?.params?.sessionId === 'string' ? msg.params.sessionId : undefined
|
||||
|
||||
// Non-CDP extension commands (ABExt.*) the daemon sends. `ungroupTab` removes a
|
||||
// tab from its per-session tab group so a `keep`-marked tab is left for the user
|
||||
// as a normal, ungrouped tab (the group can then be cleaned up). Best-effort.
|
||||
if (method === 'ABExt.ungroupTab') {
|
||||
const tabId = tabIdFromSession(sessionId) ?? tabForSession(sessionId)
|
||||
if (tabId != null && chrome.tabs.ungroup) {
|
||||
try {
|
||||
await chrome.tabs.ungroup(tabId)
|
||||
} catch {}
|
||||
}
|
||||
return { ungrouped: tabId ?? null }
|
||||
}
|
||||
|
||||
// Browser-level Target methods that map onto chrome.tabs.
|
||||
if (method === 'Target.createTarget') {
|
||||
const url = typeof params?.url === 'string' && params.url ? params.url : 'about:blank'
|
||||
@@ -387,12 +429,16 @@ async function attachTab(tabId) {
|
||||
sessionToTab.set(sessionId, tabId)
|
||||
rememberSessionTarget(sessionId, targetId)
|
||||
setBadge(tabId, port ? 'on' : 'connecting')
|
||||
const { openerTargetId, abGroup } = await tabScopeHints(tabId)
|
||||
postToHost({
|
||||
method: 'forwardCDPEvent',
|
||||
params: {
|
||||
sessionId,
|
||||
method: 'Target.attachedToTarget',
|
||||
params: { sessionId, targetInfo: { ...targetInfo, attached: true } },
|
||||
params: {
|
||||
sessionId,
|
||||
targetInfo: { ...targetInfo, attached: true, openerTargetId, abGroup },
|
||||
},
|
||||
},
|
||||
})
|
||||
return entry
|
||||
@@ -434,14 +480,32 @@ async function attachAllTabs() {
|
||||
}
|
||||
}
|
||||
|
||||
function reannounceAttachedTabs() {
|
||||
for (const [, entry] of tabs.entries()) {
|
||||
async function reannounceAttachedTabs() {
|
||||
for (const [tabId, entry] of tabs.entries()) {
|
||||
// Re-send the group hint too (issue #40) so the relay can rebuild its
|
||||
// targetId→group map after its own restart (createTarget tagging won't
|
||||
// re-run for tabs that are already open). Include the live url/title so the
|
||||
// relay's target list stays matchable by URL after a reconnect (otherwise a
|
||||
// reannounced tab shows a blank url and `adopt <url>` can't find it).
|
||||
const { openerTargetId, abGroup } = await tabScopeHints(tabId)
|
||||
let url = ''
|
||||
let title = ''
|
||||
try {
|
||||
const t = await chrome.tabs.get(tabId)
|
||||
if (t) {
|
||||
url = t.url || t.pendingUrl || ''
|
||||
title = t.title || ''
|
||||
}
|
||||
} catch {}
|
||||
postToHost({
|
||||
method: 'forwardCDPEvent',
|
||||
params: {
|
||||
sessionId: entry.sessionId,
|
||||
method: 'Target.attachedToTarget',
|
||||
params: { sessionId: entry.sessionId, targetInfo: { targetId: entry.targetId, type: 'page', attached: true } },
|
||||
params: {
|
||||
sessionId: entry.sessionId,
|
||||
targetInfo: { targetId: entry.targetId, type: 'page', url, title, attached: true, openerTargetId, abGroup },
|
||||
},
|
||||
},
|
||||
})
|
||||
}
|
||||
|
||||
@@ -1,8 +1,8 @@
|
||||
{
|
||||
"manifest_version": 3,
|
||||
"name": "chrome-use",
|
||||
"version": "0.4.9",
|
||||
"description": "Let chrome-use drive your logged-in Chrome \u2014 install once, no token, no per-use confirmation.",
|
||||
"version": "0.4.12",
|
||||
"description": "Let chrome-use drive your logged-in Chrome — install once, no token, no per-use confirmation.",
|
||||
"key": "MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA6vQIyscGIPYPZdSpPwPL0+0gxUROyRgCpmvCSDoc8XUm4qm97VbKnD9Ijc1lV22lNWZtE78gaRjt6BeSfuMgnBymnhLKjN1gU6AI5QUU0mrJyeHdWKvrKQR5FmsM2A7Xr1ykE2SiiS8zNUS3Y/6O5l+Nva7wrVy6E4a2dkBVQkOsu+DV+nEZvhIyuDY5D5SPXqNwUTWTaglwj5mjvHz36xSwCWlPmrtJ+ED0AUyrb2z4GIOmvk4kqtBVrh/UD058klLo4CkYOnIybB5aV6WYuwarfPY4bF/dLggPem+ewLNTUNBuwrxj/A4nUv0LJTuRO8rR7f8WR9qnRCY0Ic5saQIDAQAB",
|
||||
"icons": {
|
||||
"16": "icons/icon16.png",
|
||||
|
||||
+1
-1
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "chrome-use",
|
||||
"version": "1.5.10",
|
||||
"version": "1.5.27",
|
||||
"description": "chrome-use — drive your real, logged-in Chrome from any AI agent, stealth by default",
|
||||
"type": "module",
|
||||
"packageManager": "pnpm@11.1.3",
|
||||
|
||||
+141
-1
@@ -36,6 +36,41 @@ Refs (`@e1`, `@e2`, ...) are assigned fresh on every snapshot. They become
|
||||
submits, dynamic re-renders, dialog opens. Always re-snapshot before your
|
||||
next ref interaction.
|
||||
|
||||
> **Hard rule: snapshot-first, never screenshot-to-locate.** For form fields and
|
||||
> buttons, ALWAYS `snapshot -i` and act on refs/selectors. Do **not** reach for
|
||||
> `screenshot` + coordinate clicks to find or hit an element — `snapshot -i` now
|
||||
> pierces **cross-origin iframes** (embedded Google Payments / Stripe / checkout /
|
||||
> KYC forms) and lists their elements by `@ref`, including input values. Use
|
||||
> coordinates only for canvas/WebGL, or when `snapshot` genuinely returns nothing
|
||||
> for your target. Screenshots are for *visual verification you report*, never the
|
||||
> agent's own input — and a full-page `screenshot` of a real retina browser is
|
||||
> often too large for an image reader anyway. (If you ever feel you *need* a
|
||||
> screenshot to read state or locate something, that's a bug — please file it.)
|
||||
|
||||
> **Snapshot-first, always. Never default to `screenshot` + coordinate clicking
|
||||
> for form fields or buttons.** Run `snapshot -i` and act on `@refs`. Use
|
||||
> coordinates only for canvas/WebGL, or when `snapshot` genuinely returns nothing
|
||||
> for your target. This holds **even inside cross-origin embedded iframes** —
|
||||
> since v1.5.12 `snapshot -i` pierces out-of-process iframes (Google Payments,
|
||||
> Stripe, embedded checkout/KYC) and lists their elements with refs, so
|
||||
> `click @e` / `type @e` / `fill @e` work directly. A screenshot is for a genuine
|
||||
> *visual* check you report to the user — not your own input. (Full-page
|
||||
> screenshots of a real retina Chrome are often too large for the image reader
|
||||
> anyway.) Driving off pixels on the relay also risks a coordinate event drifting
|
||||
> onto the user's foreground tab — refs never do. See issue #37.
|
||||
|
||||
> **Two different intents — only one is discouraged.** The rule above is about
|
||||
> *screenshot-to-locate* (using a picture to find/hit an element) — that's the bug.
|
||||
> *screenshot-to-capture* — saving a region or element to a file as a **reusable
|
||||
> image asset** (maps, charts, og-images, visual-diff baselines, report figures) —
|
||||
> is fully supported and encouraged: `screenshot [selector] [--clip x,y,w,h] <file>`.
|
||||
> Capturing a rendered map region to a PNG for a blog post is the right tool, not a
|
||||
> smell. Screenshots are auto-downscaled to ≤2000px (longest edge) so they fit an
|
||||
> image reader and their pixels line up with `click x y`; override with
|
||||
> `--max-width`/`--max-height`/`--scale`. To click something you couldn't hit by
|
||||
> ref, `box @ref` gives the element's CSS-px box + `centerX/centerY` to feed
|
||||
> straight into `click <centerX> <centerY>` — no screenshot needed.
|
||||
|
||||
## Before you automate: pick the cheapest tool
|
||||
|
||||
Driving a browser is the heavy option. chrome-use earns its keep when you
|
||||
@@ -45,6 +80,7 @@ need a **real, logged-in browser** — not for reading text off a public page.
|
||||
|---|---|
|
||||
| Discover what exists / find sources | `WebSearch` |
|
||||
| Specific facts from a static or public page | `WebFetch` or `curl` (no browser) |
|
||||
| **Structured data from a known site** (GitHub issues, Reddit/HN search, Bilibili/Twitter feed, …) — esp. behind login | `chrome-use site <name>/<cmd>` (see below) — skip snapshot+click entirely |
|
||||
| Login state, interaction, JS-rendered or anti-bot pages | **chrome-use** (this skill) |
|
||||
| A page the user saved before / an internal system | `chrome-use find-url <keywords>` (their bookmarks), then open it |
|
||||
| The user's **own already-open, logged-in** Chrome window | the **extension connect** flow (below) |
|
||||
@@ -108,7 +144,27 @@ Each `--session` that connects gets its **own colored Chrome tab group** (named
|
||||
after the session) and drives only its own tabs — multiple agents share the one
|
||||
real browser without cross-talk, and the user's own tabs are never grouped. CDP
|
||||
drives the page without moving the user's mouse/keyboard, so it doesn't fight
|
||||
them for control. **Anti-detection ranking: this real logged-in Chrome (extension
|
||||
them for control.
|
||||
|
||||
**Strict multi-agent isolation.** A session over the relay tracks and drives
|
||||
**only the tabs it created** (its own group). It does **not** adopt the user's
|
||||
existing tabs, other agents' tabs, or pop-ups (e.g. an OAuth/login window — that's
|
||||
the user's), so several agents (and other tools opening tabs) can work in the same
|
||||
real Chrome concurrently without ever dropping or stealing each other's tabs —
|
||||
another agent's tab churn can't make your bound tab vanish or drift your commands
|
||||
onto the wrong page. Consequence: `tab list` shows only *your* session's tabs; to
|
||||
drive a specific page, navigate to it in your own tab instead of expecting a
|
||||
pre-existing or popped-up tab to appear in the list.
|
||||
|
||||
> **Need to read a tab the user already has open?** Use `chrome-use adopt
|
||||
> <url-substring|targetId>` — it finds that pre-existing tab (the user's own, or
|
||||
> another session's) across groups and drives it **without opening a new tab**.
|
||||
> e.g. `adopt "claude.ai/design"` then `snapshot`/`eval`/`get text` on it. On no
|
||||
> match it errors and lists the tabs it can see. This is the explicit, opt-in way
|
||||
> through the isolation above (it tags the adopted tab into your group). Great for
|
||||
> "read/extract from the page I'm looking at" without disturbing it.
|
||||
|
||||
**Anti-detection ranking: this real logged-in Chrome (extension
|
||||
connect) > a headed launched browser > headless (forbidden).** A genuine human
|
||||
browser has no headless/automation tells at all, so prefer it for anything
|
||||
anti-bot-sensitive.
|
||||
@@ -163,6 +219,34 @@ chrome-use eval "[...document.forms[0].elements].filter(e=>!e.validity.valid).ma
|
||||
chrome-use eval "document.querySelector('#stubborn').click()" # direct DOM click, bypasses overlays
|
||||
```
|
||||
|
||||
## Site adapters — the cheapest path for "read structured data from site X"
|
||||
|
||||
Before you `open` + `snapshot` + click your way through GitHub/Reddit/Bilibili/etc.,
|
||||
check whether a **site adapter** already exists. An adapter is a community-written JS
|
||||
function that hits the site's own JSON API *from inside your logged-in tab* and returns
|
||||
clean structured data — no clicking, no scraping, no screenshots. It's the same idea as
|
||||
`eval`, packaged per-site.
|
||||
|
||||
```bash
|
||||
chrome-use site update # one-time: fetch the adapter pack (~145 cmds)
|
||||
chrome-use site list # what's installed (github/issues, reddit/search, …)
|
||||
chrome-use site info github/issues # an adapter's args + which domain it runs on
|
||||
chrome-use site github/issues owner/repo --json # run it → JSON (navigates there for you)
|
||||
```
|
||||
|
||||
- Positional args fill the adapter's declared args **in order**; `--key value` overrides by name.
|
||||
- It navigates to the adapter's domain (reusing the current tab if you're already on it), so
|
||||
login-gated feeds (`bilibili/feed`, `twitter/...`) work because they run as *you*.
|
||||
- If no adapter fits, fall back to the normal `snapshot`/`eval` loop. Adapters come from the
|
||||
[bb-sites](https://github.com/epiral/bb-sites) community pack; chrome-use fetches & runs them.
|
||||
|
||||
> **Auto-trigger — act on it.** chrome-use keeps the pack synced automatically (first use +
|
||||
> weekly), and when you `open`/`navigate`/`snapshot` a page whose domain has adapters it tells
|
||||
> you: a `💡 site adapters for <domain>` line on stderr, and a `siteAdapters: {domain, commands}`
|
||||
> field in `--json`. **When you see that, prefer the listed `site <name>/<cmd>` over snapshot+click
|
||||
> for reading data** — it's the cheaper, more reliable path and it's already installed. You don't
|
||||
> need to run `site update` yourself; just use the command it names.
|
||||
|
||||
## Quickstart
|
||||
|
||||
```bash
|
||||
@@ -264,6 +348,10 @@ chrome-use hover @e1 # hover
|
||||
chrome-use focus @e1 # focus (useful before keyboard input)
|
||||
chrome-use fill @e2 "hello" # clear then type
|
||||
chrome-use type @e2 " world" # type without clearing
|
||||
chrome-use type @e5 "201-0001" --key-events # real keystrokes (not insertText) —
|
||||
# use for autocomplete/combobox fields that
|
||||
# only react to key events (e.g. a postal box
|
||||
# that auto-fills city/prefecture, Google Places)
|
||||
chrome-use press Enter # press a key at current focus (down+up)
|
||||
chrome-use press Control+a # key combination
|
||||
chrome-use keydown d # HOLD a key down (no auto-release)
|
||||
@@ -287,10 +375,32 @@ chrome-use upload @e5 file1.pdf # upload file(s) — works over the exten
|
||||
# a File there (chunked under native-messaging's 1 MiB cap).
|
||||
# Works on file <input>s and drop/paste composers (e.g. X).
|
||||
chrome-use scroll down 500 # scroll page (up/down/left/right)
|
||||
chrome-use scroll down 700 --at 640,400 # wheel at a pixel — scrolls a cross-origin
|
||||
# iframe (Payments/Stripe/checkout/KYC) that
|
||||
# plain page scroll can't reach
|
||||
chrome-use scroll down 700 --frame 2 # scroll frame 2 from `chrome-use frames`
|
||||
chrome-use scrollintoview @e1 # scroll element into view
|
||||
chrome-use drag @e1 @e2 # drag and drop
|
||||
```
|
||||
|
||||
**Cross-origin iframes (embedded payment / checkout / KYC widgets — Google
|
||||
Payments, Stripe, etc.) — drive them by ref, never by screenshot.** `snapshot -i`
|
||||
pierces these out-of-process iframes and lists their elements by `@ref`
|
||||
(including input values); `get text --all-frames` reads their text. Then just act
|
||||
on the refs: `click @e`, `type @e`, `hover @e`, `dblclick @e`, `drag @a @b` all
|
||||
work into the iframe. Over the extension relay these are dispatched through the
|
||||
DOM (in the element's own frame), so they hit the right element in the right tab
|
||||
— a coordinate click/scroll there can drift onto whatever tab is in the
|
||||
foreground, so prefer refs. For below-the-fold content in such a frame, scroll it
|
||||
with `scroll down N --at x,y` (a pixel over the frame) or `--frame n`. For a
|
||||
postal/autocomplete box inside the frame, `type @e "…" --key-events`.
|
||||
|
||||
> **Caveat: `find text "…"` can't reach into a cross-origin iframe** — it errors
|
||||
> "Element not found" even though `snapshot -i` lists those nodes and
|
||||
> `get text` reads them. Inside cross-origin iframes, target elements by their
|
||||
> **snapshot `@ref`**, not by `find`. (`box @ref` also works on iframe refs when
|
||||
> you need a coordinate fallback.)
|
||||
|
||||
### When refs don't work or you don't want to snapshot
|
||||
|
||||
Use semantic locators:
|
||||
@@ -344,6 +454,13 @@ tree**, so `snapshot` comes back near-empty and refs are a dead end. `snapshot`
|
||||
detects this and prints a one-line hint. Drive them the screenshot way:
|
||||
|
||||
```bash
|
||||
chrome-use canvas list # enumerate <canvas> elements (size, type)
|
||||
chrome-use canvas capture out.png # save the canvas's RENDERED pixels to PNG —
|
||||
# toDataURL (full backing-store res, e.g.
|
||||
# Figma 2522x1904), screenshot fallback for
|
||||
# WebGL w/o preserveDrawingBuffer / tainted.
|
||||
# Gets the RENDER, not hidden source data
|
||||
# (those live in the app's binary store/API).
|
||||
chrome-use screenshot /tmp/s.png # SEE the state (your only read path —
|
||||
# eval/get text return nothing useful)
|
||||
chrome-use click 640 360 # interact by viewport coordinate
|
||||
@@ -674,6 +791,29 @@ chrome-use snapshot -i
|
||||
chrome-use frame main # back to main frame
|
||||
```
|
||||
|
||||
### Viewport / window size (responsive & overflow debugging)
|
||||
|
||||
To reproduce width-dependent bugs (responsive breakpoints, horizontal-overflow
|
||||
hunts, mobile layouts) set the viewport. This is a **CDP virtual viewport**
|
||||
(`Emulation.setDeviceMetricsOverride`) — it changes the layout viewport *for the
|
||||
tab* without physically resizing the OS window, so it works headless **and** over
|
||||
the extension relay without yanking the user's real Chrome window around.
|
||||
|
||||
```bash
|
||||
chrome-use viewport 1280 800 # set width x height (alias: resize)
|
||||
chrome-use viewport 375x812 # WxH shorthand
|
||||
chrome-use viewport 375 812 --dpr 3 --mobile # retina + mobile emulation
|
||||
chrome-use viewport reset # clear the override, restore real size
|
||||
```
|
||||
|
||||
```bash
|
||||
# Find what's overflowing at a narrow width:
|
||||
chrome-use viewport 375 812
|
||||
chrome-use eval 'document.documentElement.scrollWidth + " vs " + innerWidth'
|
||||
```
|
||||
|
||||
`set viewport <w> <h> [scale]` is an equivalent alias.
|
||||
|
||||
### Dialogs
|
||||
|
||||
`alert` and `beforeunload` are auto-accepted so agents never block. For
|
||||
|
||||
Reference in New Issue
Block a user