Add agent-browser skills command with evals (#1225)

* Add `agent-browser skills` command

Adds a `skills` CLI command that serves bundled skill content at runtime,
always matching the installed CLI version. This solves the problem of
agents relying on stale cached SKILL.md files after CLI upgrades.

The `npx skills add vercel-labs/agent-browser` flow now installs a single
thin discovery skill with trigger words for all use cases (browser
automation, dogfooding, Electron apps, Slack, etc.) that directs agents
to `agent-browser skills get <name>` for current instructions. The other
five skills (dogfood, electron, slack, vercel-sandbox, agentcore) are
marked `metadata.internal: true` so they are not installed by default but
remain accessible via the CLI command.

Subcommands:
  skills [list]              List available skills
  skills get <name> [--full] Get skill content (with optional references)
  skills get --all           Get all skill content
  skills path [name]         Print skill directory path

* Fix skills command robustness: UTF-8 safety, flag handling, path output

- Make truncate_description UTF-8-safe using char_indices() instead of
  byte-indexed slicing that panics on multi-byte codepoints
- Pass get_all as a bool parameter to run_get instead of embedding
  --all as a sentinel string in the names list
- Canonicalize skills_dir path so `skills path` output is clean
- Warn on unrecognized flags in `skills get` instead of silently
  ignoring them

* Add evals framework and strengthen SKILL.md for better agent compliance

Strengthen SKILL.md loading instructions to require `skills get` before
running commands, and trim skill descriptions to prevent agents from
guessing at command syntax. Add TypeScript/Bun eval framework that tests
skill-loading, skill-selection, and command-usage via Claude CLI with
Vercel AI Gateway. Evals pass 20/20 (100%), up from 85% baseline.

* Fix formatting in skills.rs

* Add Codex provider to evals framework

Add multi-provider support with a shared Provider interface. Codex
provider spawns `codex exec --json`, parses JSONL output, and writes
~/.codex/config.toml for AI Gateway routing. Use `--provider codex`
to run evals with Codex (default model: openai/o3). First run scores
19/20 (95%) with 100% on skill-loading and skill-selection.

* Use scoped temp dir for Codex config instead of overwriting ~/.codex
This commit is contained in:
Chris Tate
2026-04-12 12:55:46 -05:00
committed by GitHub
parent fa043a496f
commit 71343069d2
29 changed files with 2069 additions and 861 deletions
+40
View File
@@ -2712,6 +2712,40 @@ Examples:
"##
}
"skills" => {
r##"
agent-browser skills - List and retrieve bundled skill content
Usage: agent-browser skills [subcommand] [options]
Subcommands:
list List all available skills (default)
get <name> [name...] Output a skill's full content
get <name> --full Include references and templates
get --all Output every skill
path [name] Print filesystem path to skill directory
Options:
--json Output as JSON
The skills command serves bundled skill content that always matches the
installed CLI version. Agents should use this to get current instructions
rather than relying on cached copies.
Examples:
agent-browser skills
agent-browser skills list
agent-browser skills get agent-browser
agent-browser skills get electron --full
agent-browser skills get --all
agent-browser skills path agent-browser
agent-browser skills list --json
Environment:
AGENT_BROWSER_SKILLS_DIR Override the skills directory path
"##
}
_ => return false,
};
println!("{}", help.trim());
@@ -2844,6 +2878,12 @@ Setup:
dashboard start Start the observability dashboard
profiles List available Chrome profiles
Skills:
skills [list] List available skills
skills get <name> [--full] Get skill content (--full includes references)
skills get --all Get all skill content
skills path [name] Print skill directory path
Snapshot Options:
-i, --interactive Only interactive elements
-c, --compact Remove empty structural elements