docs(skills): teach agents the extension-connect flow + computer-use for setup

So an agent can operate the zero-confirmation real-Chrome feature itself:
- SKILL.md: tool matrix gains "the user's own already-open, logged-in window →
  extension connect", plus a short section pointing at the flow.
- commands.md: the one-time "Load unpacked" is a privileged GUI step the CLI
  can't do — call it out that the agent can perform it with a computer-use /
  GUI-automation tool (cua-driver), with the live gotchas (synthetic-keystroke
  tools like peekaboo don't reach Chrome; cua-driver does; the native file
  dialog may need the user to pick the folder).
This commit is contained in:
leeguooooo
2026-06-09 17:14:50 +09:00
parent 6c0f5cbaa1
commit 091a4ec02e
2 changed files with 29 additions and 3 deletions
+12
View File
@@ -40,11 +40,23 @@ need a **real, logged-in browser** — not for reading text off a public page.
| Specific facts from a static or public page | `WebFetch` or `curl` (no browser) |
| Login state, interaction, JS-rendered or anti-bot pages | **agent-browser** (this skill) |
| A page the user saved before / an internal system | `agent-browser find-url <keywords>` (their bookmarks), then open it |
| The user's **own already-open, logged-in** Chrome window | the **extension connect** flow (below) |
Don't hand-build deep URLs with query params — links discovered by *interacting*
with the site carry the right hidden context and dodge anti-bot checks; a
hand-constructed URL often doesn't.
### Driving the user's real, already-open Chrome (extension)
When the task needs the user's *live* logged-in window (their real session, the
window they're looking at — not a fresh browser), use the extension connect flow:
`agent-browser extension install` once, load `extensions/ab-connect` in
`chrome://extensions` once (a GUI step you can perform with a **computer-use /
GUI-automation tool** like the `cua-driver` skill — see
`references/commands.md` → "Drive your real, logged-in Chrome"), then
`agent-browser extension connect`. After that it's zero-confirmation, zero-token
CLI. Use `--launch` instead when a fresh, isolated browser is fine.
## Two ways to drive a page — and when to drop to `eval`
You have a **real Chrome with the user's DOM**. Two layers, mix them freely:
+17 -3
View File
@@ -328,11 +328,25 @@ codex/claude approach).
One-time setup:
```bash
agent-browser extension install # writes the native-messaging host manifest
# then in Chrome: chrome://extensions → Developer mode → Load unpacked →
# <repo>/extensions/ab-connect (load once)
```
Then load the extension **once** — this is a GUI step (Chrome's `chrome://extensions`
is privileged; the CLI can't load an unpacked extension):
Then, any time:
> chrome://extensions → enable **Developer mode** (top-right) → **Load unpacked** →
> select `<repo>/extensions/ab-connect`
**You can do this load step yourself with a computer-use / GUI-automation tool**
(e.g. the `cua-driver` skill) — drive `chrome://extensions`, toggle Developer
mode, click *Load unpacked*, pick the folder in the Open dialog. If the extension
is already loaded, clicking its **Reload** (↻) button after a code change is
enough. Notes from doing this live: tools that send *synthetic keystrokes* (e.g.
peekaboo) often don't reach Chrome — **`cua-driver` works** because it reads
Chrome's accessibility tree and clicks real elements. The native "Open" file
dialog is the fiddly part; if keystroke entry there fails, ask the user to pick
the folder (one click). After it loads, Chrome assigns the extension a fixed id
(pinned in its manifest) and auto-connects the host.
Then, any time (pure CLI, zero confirmation):
```bash
agent-browser extension connect # auto-attaches to the live, logged-in tabs
agent-browser tab # list the real tabs it now controls