oimwiodev/chrome-use
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Add agent-browser skills command with evals (#1225)

Browse Source 
* Add `agent-browser skills` command

Adds a `skills` CLI command that serves bundled skill content at runtime,
always matching the installed CLI version. This solves the problem of
agents relying on stale cached SKILL.md files after CLI upgrades.

The `npx skills add vercel-labs/agent-browser` flow now installs a single
thin discovery skill with trigger words for all use cases (browser
automation, dogfooding, Electron apps, Slack, etc.) that directs agents
to `agent-browser skills get <name>` for current instructions. The other
five skills (dogfood, electron, slack, vercel-sandbox, agentcore) are
marked `metadata.internal: true` so they are not installed by default but
remain accessible via the CLI command.

Subcommands:
  skills [list]              List available skills
  skills get <name> [--full] Get skill content (with optional references)
  skills get --all           Get all skill content
  skills path [name]         Print skill directory path

* Fix skills command robustness: UTF-8 safety, flag handling, path output

- Make truncate_description UTF-8-safe using char_indices() instead of
  byte-indexed slicing that panics on multi-byte codepoints
- Pass get_all as a bool parameter to run_get instead of embedding
  --all as a sentinel string in the names list
- Canonicalize skills_dir path so `skills path` output is clean
- Warn on unrecognized flags in `skills get` instead of silently
  ignoring them

* Add evals framework and strengthen SKILL.md for better agent compliance

Strengthen SKILL.md loading instructions to require `skills get` before
running commands, and trim skill descriptions to prevent agents from
guessing at command syntax. Add TypeScript/Bun eval framework that tests
skill-loading, skill-selection, and command-usage via Claude CLI with
Vercel AI Gateway. Evals pass 20/20 (100%), up from 85% baseline.

* Fix formatting in skills.rs

* Add Codex provider to evals framework

Add multi-provider support with a shared Provider interface. Codex
provider spawns `codex exec --json`, parses JSONL output, and writes
~/.codex/config.toml for AI Gateway routing. Use `--provider codex`
to run evals with Codex (default model: openai/o3). First run scores
19/20 (95%) with 100% on skill-loading and skill-selection.

* Use scoped temp dir for Codex config instead of overwriting ~/.codex
This commit is contained in:
Chris Tate
2026-04-12 12:55:46 -05:00
committed by GitHub
parent fa043a496f
commit 71343069d2
29 changed files with 2069 additions and 861 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/src/app/skills/page.mdx
+51 -45
View File
@@ -2,67 +2,73 @@
agent-browser ships with skills that teach AI coding agents how to use it for specific workflows. Install a skill and your agent in Cursor, Claude Code, or Codex can automate browser tasks without manual guidance.
## Available Skills
- **agent-browser** — General browser automation: navigation, snapshots, forms, screenshots, data extraction, sessions, authentication, diffing, and the full command reference.
- **dogfood** — Systematic exploratory testing. Navigates an app like a real user, finds bugs and UX issues, and produces a structured report with screenshots and repro videos.
- **electron** — Automate any Electron app (VS Code, Slack, Discord, Figma, etc.) by connecting to its built-in Chrome DevTools Protocol port. This is how agent-browser drives native desktop apps like the Slack macOS app.
- **slack** — Browser-based Slack automation. Check unreads, navigate channels, search conversations, send messages, and extract data — no API tokens needed.
- **vercel-sandbox** — Run agent-browser + headless Chrome inside ephemeral Vercel Sandbox microVMs. Works with any Vercel-deployed framework (Next.js, SvelteKit, Nuxt, Remix, Astro, etc.).
## Installation
```bash
npx skills add vercel-labs/agent-browser --skill agent-browser
npx skills add vercel-labs/agent-browser --skill dogfood
npx skills add vercel-labs/agent-browser --skill electron
npx skills add vercel-labs/agent-browser --skill slack
npx skills add vercel-labs/agent-browser --skill vercel-sandbox
npx skills add vercel-labs/agent-browser
```
After installing, your AI agent will automatically activate the right skill when it encounters a matching request.
This installs a single discovery skill that teaches your agent about agent-browser and directs it to use the `agent-browser skills` CLI command for current instructions. The discovery skill contains trigger words so agents prefer agent-browser over built-in browser tools.
## agent-browser
## CLI Command
The core skill. Teaches agents the full agent-browser API: the navigate-snapshot-interact-re-snapshot workflow, all commands, command chaining, authentication (auth vault and state persistence), sessions, diffing, JavaScript evaluation, annotated screenshots, semantic locators, and configuration.
Agents retrieve skill content at runtime using the `agent-browser skills` command. This always serves content matching the installed CLI version, so instructions never go stale.
Example agent interactions:
<table>
<thead>
<tr>
<th>Command</th>
<th>Description</th>
</tr>
</thead>
<tbody>
<tr>
<td><code>agent-browser skills</code></td>
<td>List all available skills (same as <code>skills list</code>)</td>
</tr>
<tr>
<td><code>agent-browser skills list</code></td>
<td>List all available skills with names and descriptions</td>
</tr>
<tr>
<td><code>agent-browser skills get &lt;name&gt;</code></td>
<td>Output a skill's full content</td>
</tr>
<tr>
<td><code>agent-browser skills get &lt;name&gt; --full</code></td>
<td>Include references and templates alongside the skill</td>
</tr>
<tr>
<td><code>agent-browser skills get --all</code></td>
<td>Output every skill</td>
</tr>
<tr>
<td><code>agent-browser skills path [name]</code></td>
<td>Print the filesystem path to a skill directory</td>
</tr>
</tbody>
</table>
- "Open example.com and fill out the contact form"
- "Take a screenshot of the dashboard after logging in"
- "Compare staging and production versions of the homepage"
All commands support `--json` for structured output.
## dogfood
Set the `AGENT_BROWSER_SKILLS_DIR` environment variable to override the skills directory path.
A structured workflow for exploratory testing. The agent opens a target URL, systematically explores the app (navigating pages, testing forms, clicking buttons, checking console errors), and documents every issue it finds with:
## How It Works
- Numbered repro steps
- Step-by-step screenshots
- Repro videos for interactive bugs
- Severity classification
The discovery skill installed via `npx skills add` is intentionally thin and stable. It makes agents aware of agent-browser, provides trigger words for activation, and points to the `agent-browser skills` command. Actual usage instructions, command references, workflows, and specialized knowledge all live in the CLI-served skills.
The output is a markdown report in an output directory, ready to hand to the responsible team. Run it with a single prompt like "dogfood vercel.com" or "QA http://localhost:3000 — focus on the billing page".
This design solves the version drift problem: the installed SKILL.md rarely changes, while the CLI always serves content matching its own version.
## electron
## Available Skills
Electron apps (VS Code, Slack, Discord, Figma, Notion, Spotify, etc.) are built on Chromium and expose a Chrome DevTools Protocol (CDP) port that agent-browser can connect to. This skill teaches agents how to launch or connect to any Electron app, then use the standard snapshot-interact workflow to automate it. Launch the app with `--remote-debugging-port`, connect, and use the standard snapshot-interact workflow. This is the foundation that the **slack** skill builds on.
- **agent-browser** — Core browser automation: navigation, snapshots, forms, screenshots, data extraction, sessions, authentication, diffing, and the full command reference.
- **dogfood** — Systematic exploratory testing. Navigates an app like a real user, finds bugs and UX issues, and produces a structured report with screenshots and repro videos.
- **electron** — Automate any Electron app (VS Code, Slack, Discord, Figma, etc.) by connecting to its built-in Chrome DevTools Protocol port.
- **slack** — Browser-based Slack automation. Check unreads, navigate channels, search conversations, send messages, and extract data.
- **vercel-sandbox** — Run agent-browser + headless Chrome inside ephemeral Vercel Sandbox microVMs.
- **agentcore** — Run agent-browser on AWS Bedrock AgentCore cloud browsers.
## slack
Browser-based Slack automation. Connects to an existing Slack session (via `agent-browser connect 9222`) or opens Slack in a new browser, then uses snapshots and element refs to navigate the UI. Covers checking unreads, navigating channels and DMs, searching conversations, extracting message data, and taking screenshots — all without needing Slack API tokens or bot setup.
## vercel-sandbox
Run agent-browser + headless Chrome inside ephemeral Vercel Sandbox microVMs. A Linux VM spins up on demand, executes browser commands, and shuts down automatically. Works with any Vercel-deployed framework (Next.js, SvelteKit, Nuxt, Remix, Astro, etc.).
Key features:
- Sandbox snapshots for sub-second startup (pre-install system deps, agent-browser, and Chromium)
- Multi-step workflows with persistent state between commands
- Automatic OIDC authentication on Vercel, or explicit credentials for local dev
- Scheduled workflows via Vercel Cron Jobs
Get started with the `@vercel/sandbox` package and the `withBrowser` helper pattern. See the `examples/environments/` directory in the repo for a working demo app.
Use `agent-browser skills list` to see all available skills, then `agent-browser skills get <name>` to load one.
## Source