docs(skills): teach agents the extension-connect flow + computer-use for setup
So an agent can operate the zero-confirmation real-Chrome feature itself: - SKILL.md: tool matrix gains "the user's own already-open, logged-in window → extension connect", plus a short section pointing at the flow. - commands.md: the one-time "Load unpacked" is a privileged GUI step the CLI can't do — call it out that the agent can perform it with a computer-use / GUI-automation tool (cua-driver), with the live gotchas (synthetic-keystroke tools like peekaboo don't reach Chrome; cua-driver does; the native file dialog may need the user to pick the folder).
This commit is contained in:
@@ -40,11 +40,23 @@ need a **real, logged-in browser** — not for reading text off a public page.
|
||||
| Specific facts from a static or public page | `WebFetch` or `curl` (no browser) |
|
||||
| Login state, interaction, JS-rendered or anti-bot pages | **agent-browser** (this skill) |
|
||||
| A page the user saved before / an internal system | `agent-browser find-url <keywords>` (their bookmarks), then open it |
|
||||
| The user's **own already-open, logged-in** Chrome window | the **extension connect** flow (below) |
|
||||
|
||||
Don't hand-build deep URLs with query params — links discovered by *interacting*
|
||||
with the site carry the right hidden context and dodge anti-bot checks; a
|
||||
hand-constructed URL often doesn't.
|
||||
|
||||
### Driving the user's real, already-open Chrome (extension)
|
||||
|
||||
When the task needs the user's *live* logged-in window (their real session, the
|
||||
window they're looking at — not a fresh browser), use the extension connect flow:
|
||||
`agent-browser extension install` once, load `extensions/ab-connect` in
|
||||
`chrome://extensions` once (a GUI step you can perform with a **computer-use /
|
||||
GUI-automation tool** like the `cua-driver` skill — see
|
||||
`references/commands.md` → "Drive your real, logged-in Chrome"), then
|
||||
`agent-browser extension connect`. After that it's zero-confirmation, zero-token
|
||||
CLI. Use `--launch` instead when a fresh, isolated browser is fine.
|
||||
|
||||
## Two ways to drive a page — and when to drop to `eval`
|
||||
|
||||
You have a **real Chrome with the user's DOM**. Two layers, mix them freely:
|
||||
|
||||
@@ -328,11 +328,25 @@ codex/claude approach).
|
||||
One-time setup:
|
||||
```bash
|
||||
agent-browser extension install # writes the native-messaging host manifest
|
||||
# then in Chrome: chrome://extensions → Developer mode → Load unpacked →
|
||||
# <repo>/extensions/ab-connect (load once)
|
||||
```
|
||||
Then load the extension **once** — this is a GUI step (Chrome's `chrome://extensions`
|
||||
is privileged; the CLI can't load an unpacked extension):
|
||||
|
||||
Then, any time:
|
||||
> chrome://extensions → enable **Developer mode** (top-right) → **Load unpacked** →
|
||||
> select `<repo>/extensions/ab-connect`
|
||||
|
||||
**You can do this load step yourself with a computer-use / GUI-automation tool**
|
||||
(e.g. the `cua-driver` skill) — drive `chrome://extensions`, toggle Developer
|
||||
mode, click *Load unpacked*, pick the folder in the Open dialog. If the extension
|
||||
is already loaded, clicking its **Reload** (↻) button after a code change is
|
||||
enough. Notes from doing this live: tools that send *synthetic keystrokes* (e.g.
|
||||
peekaboo) often don't reach Chrome — **`cua-driver` works** because it reads
|
||||
Chrome's accessibility tree and clicks real elements. The native "Open" file
|
||||
dialog is the fiddly part; if keystroke entry there fails, ask the user to pick
|
||||
the folder (one click). After it loads, Chrome assigns the extension a fixed id
|
||||
(pinned in its manifest) and auto-connects the host.
|
||||
|
||||
Then, any time (pure CLI, zero confirmation):
|
||||
```bash
|
||||
agent-browser extension connect # auto-attaches to the live, logged-in tabs
|
||||
agent-browser tab # list the real tabs it now controls
|
||||
|
||||
Reference in New Issue
Block a user