From 091a4ec02e32abaf517f7b6dfcbd017f9eff2a64 Mon Sep 17 00:00:00 2001 From: leeguooooo Date: Tue, 9 Jun 2026 17:14:50 +0900 Subject: [PATCH] docs(skills): teach agents the extension-connect flow + computer-use for setup MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit So an agent can operate the zero-confirmation real-Chrome feature itself: - SKILL.md: tool matrix gains "the user's own already-open, logged-in window → extension connect", plus a short section pointing at the flow. - commands.md: the one-time "Load unpacked" is a privileged GUI step the CLI can't do — call it out that the agent can perform it with a computer-use / GUI-automation tool (cua-driver), with the live gotchas (synthetic-keystroke tools like peekaboo don't reach Chrome; cua-driver does; the native file dialog may need the user to pick the folder). --- skill-data/core/SKILL.md | 12 ++++++++++++ skill-data/core/references/commands.md | 20 +++++++++++++++++--- 2 files changed, 29 insertions(+), 3 deletions(-) diff --git a/skill-data/core/SKILL.md b/skill-data/core/SKILL.md index 225fd6c..9d5fb80 100644 --- a/skill-data/core/SKILL.md +++ b/skill-data/core/SKILL.md @@ -40,11 +40,23 @@ need a **real, logged-in browser** — not for reading text off a public page. | Specific facts from a static or public page | `WebFetch` or `curl` (no browser) | | Login state, interaction, JS-rendered or anti-bot pages | **agent-browser** (this skill) | | A page the user saved before / an internal system | `agent-browser find-url ` (their bookmarks), then open it | +| The user's **own already-open, logged-in** Chrome window | the **extension connect** flow (below) | Don't hand-build deep URLs with query params — links discovered by *interacting* with the site carry the right hidden context and dodge anti-bot checks; a hand-constructed URL often doesn't. +### Driving the user's real, already-open Chrome (extension) + +When the task needs the user's *live* logged-in window (their real session, the +window they're looking at — not a fresh browser), use the extension connect flow: +`agent-browser extension install` once, load `extensions/ab-connect` in +`chrome://extensions` once (a GUI step you can perform with a **computer-use / +GUI-automation tool** like the `cua-driver` skill — see +`references/commands.md` → "Drive your real, logged-in Chrome"), then +`agent-browser extension connect`. After that it's zero-confirmation, zero-token +CLI. Use `--launch` instead when a fresh, isolated browser is fine. + ## Two ways to drive a page — and when to drop to `eval` You have a **real Chrome with the user's DOM**. Two layers, mix them freely: diff --git a/skill-data/core/references/commands.md b/skill-data/core/references/commands.md index 1680645..e9da008 100644 --- a/skill-data/core/references/commands.md +++ b/skill-data/core/references/commands.md @@ -328,11 +328,25 @@ codex/claude approach). One-time setup: ```bash agent-browser extension install # writes the native-messaging host manifest -# then in Chrome: chrome://extensions → Developer mode → Load unpacked → -# /extensions/ab-connect (load once) ``` +Then load the extension **once** — this is a GUI step (Chrome's `chrome://extensions` +is privileged; the CLI can't load an unpacked extension): -Then, any time: +> chrome://extensions → enable **Developer mode** (top-right) → **Load unpacked** → +> select `/extensions/ab-connect` + +**You can do this load step yourself with a computer-use / GUI-automation tool** +(e.g. the `cua-driver` skill) — drive `chrome://extensions`, toggle Developer +mode, click *Load unpacked*, pick the folder in the Open dialog. If the extension +is already loaded, clicking its **Reload** (↻) button after a code change is +enough. Notes from doing this live: tools that send *synthetic keystrokes* (e.g. +peekaboo) often don't reach Chrome — **`cua-driver` works** because it reads +Chrome's accessibility tree and clicks real elements. The native "Open" file +dialog is the fiddly part; if keystroke entry there fails, ask the user to pick +the folder (one click). After it loads, Chrome assigns the extension a fixed id +(pinned in its manifest) and auto-connects the host. + +Then, any time (pure CLI, zero confirmation): ```bash agent-browser extension connect # auto-attaches to the live, logged-in tabs agent-browser tab # list the real tabs it now controls