Chris Tate
71343069d2
* Add `agent-browser skills` command Adds a `skills` CLI command that serves bundled skill content at runtime, always matching the installed CLI version. This solves the problem of agents relying on stale cached SKILL.md files after CLI upgrades. The `npx skills add vercel-labs/agent-browser` flow now installs a single thin discovery skill with trigger words for all use cases (browser automation, dogfooding, Electron apps, Slack, etc.) that directs agents to `agent-browser skills get <name>` for current instructions. The other five skills (dogfood, electron, slack, vercel-sandbox, agentcore) are marked `metadata.internal: true` so they are not installed by default but remain accessible via the CLI command. Subcommands: skills [list] List available skills skills get <name> [--full] Get skill content (with optional references) skills get --all Get all skill content skills path [name] Print skill directory path * Fix skills command robustness: UTF-8 safety, flag handling, path output - Make truncate_description UTF-8-safe using char_indices() instead of byte-indexed slicing that panics on multi-byte codepoints - Pass get_all as a bool parameter to run_get instead of embedding --all as a sentinel string in the names list - Canonicalize skills_dir path so `skills path` output is clean - Warn on unrecognized flags in `skills get` instead of silently ignoring them * Add evals framework and strengthen SKILL.md for better agent compliance Strengthen SKILL.md loading instructions to require `skills get` before running commands, and trim skill descriptions to prevent agents from guessing at command syntax. Add TypeScript/Bun eval framework that tests skill-loading, skill-selection, and command-usage via Claude CLI with Vercel AI Gateway. Evals pass 20/20 (100%), up from 85% baseline. * Fix formatting in skills.rs * Add Codex provider to evals framework Add multi-provider support with a shared Provider interface. Codex provider spawns `codex exec --json`, parses JSONL output, and writes ~/.codex/config.toml for AI Gateway routing. Use `--provider codex` to run evals with Codex (default model: openai/o3). First run scores 19/20 (95%) with 100% on skill-loading and skill-selection. * Use scoped temp dir for Codex config instead of overwriting ~/.codex
95 lines
2.6 KiB
TypeScript
95 lines
2.6 KiB
TypeScript
import type { EvalCase } from "../lib/types.ts";
|
|
|
|
const RUBRIC = `
|
|
1 - Agent does not load any skill or loads a completely wrong one
|
|
2 - Agent loads the generic agent-browser skill when a specialized one exists
|
|
3 - Agent loads a related but suboptimal skill
|
|
4 - Agent loads the correct specialized skill
|
|
5 - Agent loads the correct skill and explains why it chose it
|
|
`.trim();
|
|
|
|
export const cases: EvalCase[] = [
|
|
{
|
|
id: "ss-01",
|
|
name: "Selects slack skill for Slack tasks",
|
|
category: "skill-selection",
|
|
prompt: "Check my Slack unreads and summarize any messages mentioning me",
|
|
expectedPatterns: [
|
|
"skills get slack",
|
|
],
|
|
rubric: RUBRIC,
|
|
},
|
|
{
|
|
id: "ss-02",
|
|
name: "Selects electron skill for VS Code automation",
|
|
category: "skill-selection",
|
|
prompt: "Automate VS Code to open a project and run a terminal command",
|
|
expectedPatterns: [
|
|
"skills get electron",
|
|
],
|
|
rubric: RUBRIC,
|
|
},
|
|
{
|
|
id: "ss-03",
|
|
name: "Selects dogfood skill for QA/testing",
|
|
category: "skill-selection",
|
|
prompt: "QA test http://localhost:3000 and find any bugs or UX issues",
|
|
expectedPatterns: [
|
|
"skills get dogfood",
|
|
],
|
|
rubric: RUBRIC,
|
|
},
|
|
{
|
|
id: "ss-04",
|
|
name: "Selects agentcore skill for AWS cloud browsers",
|
|
category: "skill-selection",
|
|
prompt:
|
|
"Run browser automation on AWS using AgentCore cloud browsers",
|
|
expectedPatterns: [
|
|
"skills get agentcore",
|
|
],
|
|
rubric: RUBRIC,
|
|
},
|
|
{
|
|
id: "ss-05",
|
|
name: "Selects vercel-sandbox skill for Vercel environments",
|
|
category: "skill-selection",
|
|
prompt:
|
|
"Run headless Chrome inside a Vercel Sandbox microVM to test my deployed Next.js app",
|
|
expectedPatterns: [
|
|
"skills get vercel-sandbox",
|
|
],
|
|
rubric: RUBRIC,
|
|
},
|
|
{
|
|
id: "ss-06",
|
|
name: "Selects electron skill for Discord automation",
|
|
category: "skill-selection",
|
|
prompt: "Automate the Discord desktop app to send a message in a channel",
|
|
expectedPatterns: [
|
|
"skills get electron",
|
|
],
|
|
rubric: RUBRIC,
|
|
},
|
|
{
|
|
id: "ss-07",
|
|
name: "Selects dogfood skill for exploratory testing",
|
|
category: "skill-selection",
|
|
prompt: "Dogfood vercel.com and write up a bug report",
|
|
expectedPatterns: [
|
|
"skills get dogfood",
|
|
],
|
|
rubric: RUBRIC,
|
|
},
|
|
{
|
|
id: "ss-08",
|
|
name: "Selects agent-browser skill for general browser tasks",
|
|
category: "skill-selection",
|
|
prompt: "Navigate to hacker news and screenshot the front page",
|
|
expectedPatterns: [
|
|
"skills get agent-browser",
|
|
],
|
|
rubric: RUBRIC,
|
|
},
|
|
];
|