Add agent-browser skills command with evals (#1225)
* Add `agent-browser skills` command Adds a `skills` CLI command that serves bundled skill content at runtime, always matching the installed CLI version. This solves the problem of agents relying on stale cached SKILL.md files after CLI upgrades. The `npx skills add vercel-labs/agent-browser` flow now installs a single thin discovery skill with trigger words for all use cases (browser automation, dogfooding, Electron apps, Slack, etc.) that directs agents to `agent-browser skills get <name>` for current instructions. The other five skills (dogfood, electron, slack, vercel-sandbox, agentcore) are marked `metadata.internal: true` so they are not installed by default but remain accessible via the CLI command. Subcommands: skills [list] List available skills skills get <name> [--full] Get skill content (with optional references) skills get --all Get all skill content skills path [name] Print skill directory path * Fix skills command robustness: UTF-8 safety, flag handling, path output - Make truncate_description UTF-8-safe using char_indices() instead of byte-indexed slicing that panics on multi-byte codepoints - Pass get_all as a bool parameter to run_get instead of embedding --all as a sentinel string in the names list - Canonicalize skills_dir path so `skills path` output is clean - Warn on unrecognized flags in `skills get` instead of silently ignoring them * Add evals framework and strengthen SKILL.md for better agent compliance Strengthen SKILL.md loading instructions to require `skills get` before running commands, and trim skill descriptions to prevent agents from guessing at command syntax. Add TypeScript/Bun eval framework that tests skill-loading, skill-selection, and command-usage via Claude CLI with Vercel AI Gateway. Evals pass 20/20 (100%), up from 85% baseline. * Fix formatting in skills.rs * Add Codex provider to evals framework Add multi-provider support with a shared Provider interface. Codex provider spawns `codex exec --json`, parses JSONL output, and writes ~/.codex/config.toml for AI Gateway routing. Use `--provider codex` to run evals with Codex (default model: openai/o3). First run scores 19/20 (95%) with 100% on skill-loading and skill-selection. * Use scoped temp dir for Codex config instead of overwriting ~/.codex
This commit is contained in:
Chris Tate
@@ -0,0 +1,2 @@
|
||||
# Vercel AI Gateway key (required)
|
||||
AI_GATEWAY_API_KEY=
|
||||
@@ -0,0 +1,3 @@
|
||||
node_modules/
|
||||
dist/
|
||||
bun.lockb
|
||||
+127
@@ -0,0 +1,127 @@
|
||||
# Skills Evals
|
||||
|
||||
Tests whether the thin SKILL.md + CLI-served skills approach works: do agents load the right skill via `agent-browser skills get`, then produce correct agent-browser commands?
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- [Bun](https://bun.sh) installed
|
||||
- `AI_GATEWAY_API_KEY` set (Vercel AI Gateway key)
|
||||
- One or both CLIs installed:
|
||||
- `claude` CLI (`npm i -g @anthropic-ai/claude-code`) for the Claude provider
|
||||
- `codex` CLI (`npm i -g @openai/codex`) for the Codex provider
|
||||
|
||||
The evals route all calls through the Vercel AI Gateway (`https://ai-gateway.vercel.sh`). Set your key before running:
|
||||
|
||||
```bash
|
||||
export AI_GATEWAY_API_KEY=gw_your_key_here
|
||||
```
|
||||
|
||||
Or copy `.env.example` to `.env` and source it.
|
||||
|
||||
## Usage
|
||||
|
||||
```bash
|
||||
cd evals
|
||||
|
||||
# Run all evals (default: Claude provider)
|
||||
bun run run.ts
|
||||
|
||||
# Use Codex provider
|
||||
bun run run.ts --provider codex
|
||||
|
||||
# Filter by category
|
||||
bun run run.ts --category skill-loading
|
||||
bun run run.ts --category skill-selection
|
||||
bun run run.ts --category command-usage
|
||||
|
||||
# Use a specific model (overrides provider default)
|
||||
bun run run.ts --model anthropic/claude-opus-4.6
|
||||
bun run run.ts --provider codex --model openai/gpt-4.1
|
||||
|
||||
# Enable LLM judge for quality scoring (1-5)
|
||||
bun run run.ts --judge
|
||||
|
||||
# JSON output (for CI or further analysis)
|
||||
bun run run.ts --json
|
||||
|
||||
# Combine options
|
||||
bun run run.ts --provider codex --category skill-selection --judge
|
||||
```
|
||||
|
||||
Or via package scripts:
|
||||
|
||||
```bash
|
||||
bun run eval # run all (Claude)
|
||||
bun run eval:claude # run all (Claude, explicit)
|
||||
bun run eval:codex # run all (Codex)
|
||||
bun run eval:judge # run all with LLM judge
|
||||
bun run eval:json # JSON output
|
||||
```
|
||||
|
||||
## Providers
|
||||
|
||||
<table>
|
||||
<tr><th>Provider</th><th>CLI</th><th>Default Model</th><th>Notes</th></tr>
|
||||
<tr><td>claude</td><td><code>claude -p</code></td><td>anthropic/claude-sonnet-4.6</td><td>Uses ANTHROPIC_API_KEY + ANTHROPIC_BASE_URL env vars</td></tr>
|
||||
<tr><td>codex</td><td><code>codex exec --json</code></td><td>openai/o3</td><td>Writes ~/.codex/config.toml with AI Gateway config</td></tr>
|
||||
</table>
|
||||
|
||||
The LLM judge always uses Claude (anthropic/claude-opus-4.6), regardless of the eval provider.
|
||||
|
||||
## Eval Categories
|
||||
|
||||
### skill-loading
|
||||
|
||||
Tests that the agent runs `agent-browser skills get` before issuing browser commands. The thin SKILL.md instructs agents to load skills first; these evals verify compliance.
|
||||
|
||||
### skill-selection
|
||||
|
||||
Tests that the agent picks the correct specialized skill for the task. For example, a Slack task should load the `slack` skill, not the generic `agent-browser` skill.
|
||||
|
||||
### command-usage
|
||||
|
||||
Tests that the agent produces correct agent-browser commands for common workflows: navigation + screenshot, form filling with snapshot-interact pattern, diffing, authentication, data extraction.
|
||||
|
||||
## How It Works
|
||||
|
||||
1. Each eval case provides a user task prompt
|
||||
2. The thin `skills/agent-browser/SKILL.md` is injected as context (simulating a skill installation)
|
||||
3. The chosen provider CLI is called to get a single response
|
||||
4. Pattern matching checks for expected/forbidden command patterns (pass/fail)
|
||||
5. Optionally, a second Claude call judges response quality on a 1-5 scale
|
||||
|
||||
## Adding Cases
|
||||
|
||||
Create or edit files in `cases/`. Each file exports a `cases` array of `EvalCase` objects:
|
||||
|
||||
```typescript
|
||||
import type { EvalCase } from "../lib/types.ts";
|
||||
|
||||
export const cases: EvalCase[] = [
|
||||
{
|
||||
id: "xx-01",
|
||||
name: "Description of what this tests",
|
||||
category: "skill-loading",
|
||||
prompt: "The user task to send to the model",
|
||||
expectedPatterns: ["regex.*that.*must.*match"],
|
||||
forbiddenPatterns: ["regex.*that.*must.*not.*match"],
|
||||
rubric: "1 - worst ... 5 - best",
|
||||
},
|
||||
];
|
||||
```
|
||||
|
||||
Then import and add the cases to `ALL_CASES` in `run.ts`.
|
||||
|
||||
## Output
|
||||
|
||||
Console mode shows pass/fail per case with failed pattern details:
|
||||
|
||||
```
|
||||
skill-loading
|
||||
----------------------------------------------------------------------
|
||||
✓ Loads skill before opening a page PASS 3200ms
|
||||
✗ Loads skill before form interaction FAIL 2800ms
|
||||
✗ Expected pattern not found: agent-browser skills get
|
||||
```
|
||||
|
||||
JSON mode (`--json`) outputs structured results for programmatic consumption.
|
||||
@@ -0,0 +1,19 @@
|
||||
{
|
||||
"lockfileVersion": 1,
|
||||
"configVersion": 1,
|
||||
"workspaces": {
|
||||
"": {
|
||||
"name": "agent-browser-evals",
|
||||
"dependencies": {
|
||||
"bun-types": "^1.3.12",
|
||||
},
|
||||
},
|
||||
},
|
||||
"packages": {
|
||||
"@types/node": ["@types/node@25.6.0", "", { "dependencies": { "undici-types": "~7.19.0" } }, "sha512-+qIYRKdNYJwY3vRCZMdJbPLJAtGjQBudzZzdzwQYkEPQd+PJGixUL5QfvCLDaULoLv+RhT3LDkwEfKaAkgSmNQ=="],
|
||||
|
||||
"bun-types": ["bun-types@1.3.12", "", { "dependencies": { "@types/node": "*" } }, "sha512-HqOLj5PoFajAQciOMRiIZGNoKxDJSr6qigAttOX40vJuSp6DN/CxWp9s3C1Xwm4oH7ybueITwiaOcWXoYVoRkA=="],
|
||||
|
||||
"undici-types": ["undici-types@7.19.2", "", {}, "sha512-qYVnV5OEm2AW8cJMCpdV20CDyaN3g0AjDlOGf1OW4iaDEx8MwdtChUp4zu4H0VP3nDRF/8RKWH+IPp9uW0YGZg=="],
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,120 @@
|
||||
import type { EvalCase } from "../lib/types.ts";
|
||||
|
||||
const RUBRIC = `
|
||||
1 - Agent does not produce valid agent-browser commands
|
||||
2 - Agent uses agent-browser but with wrong commands or missing steps
|
||||
3 - Agent uses correct commands but skips the snapshot-interact workflow
|
||||
4 - Agent follows the correct workflow with appropriate commands
|
||||
5 - Agent follows the optimal workflow: navigate, snapshot, interact with refs, re-snapshot as needed
|
||||
`.trim();
|
||||
|
||||
const COMMAND_CONTEXT = `You already ran \`agent-browser skills get agent-browser\` and loaded these commands:
|
||||
- agent-browser open <url> (navigate to a page)
|
||||
- agent-browser snapshot -i (get interactive elements with refs like @e1, @e2)
|
||||
- agent-browser click @ref (click element)
|
||||
- agent-browser fill @ref "text" (clear and type)
|
||||
- agent-browser type @ref "text" (type without clearing)
|
||||
- agent-browser select @ref "option" (select dropdown)
|
||||
- agent-browser screenshot (screenshot to temp dir)
|
||||
- agent-browser screenshot --full (full page screenshot)
|
||||
- agent-browser diff url <url1> <url2> (compare two pages)
|
||||
- agent-browser diff snapshot (compare current vs last snapshot)
|
||||
- agent-browser state save ./file.json (save auth state)
|
||||
- agent-browser state load ./file.json (restore auth state)
|
||||
- agent-browser get text @ref (get element text)
|
||||
- agent-browser wait <selector|ms> (wait for element or time)
|
||||
- agent-browser --session-name <name> open <url> (named session with auto-save)
|
||||
|
||||
Workflow: open -> snapshot -i -> interact with refs -> re-snapshot after changes.`;
|
||||
|
||||
export const cases: EvalCase[] = [
|
||||
{
|
||||
id: "cu-01",
|
||||
name: "Navigate and screenshot workflow",
|
||||
category: "command-usage",
|
||||
prompt: "Open example.com and take a screenshot",
|
||||
context: COMMAND_CONTEXT,
|
||||
expectedPatterns: [
|
||||
"agent-browser\\s+(open|goto|navigate)",
|
||||
"agent-browser\\s+screenshot",
|
||||
],
|
||||
rubric: RUBRIC,
|
||||
},
|
||||
{
|
||||
id: "cu-02",
|
||||
name: "Form filling workflow",
|
||||
category: "command-usage",
|
||||
prompt:
|
||||
"Go to example.com/signup, fill in name as 'Jane Doe' and email as 'jane@test.com', then submit",
|
||||
context: COMMAND_CONTEXT,
|
||||
expectedPatterns: [
|
||||
"agent-browser\\s+(open|goto|navigate)",
|
||||
"agent-browser\\s+snapshot",
|
||||
"agent-browser\\s+(fill|type)",
|
||||
"agent-browser\\s+(click|press|key)",
|
||||
],
|
||||
rubric: RUBRIC,
|
||||
},
|
||||
{
|
||||
id: "cu-03",
|
||||
name: "Snapshot with element refs",
|
||||
category: "command-usage",
|
||||
prompt: "Get all interactive elements on example.com",
|
||||
context: COMMAND_CONTEXT,
|
||||
expectedPatterns: [
|
||||
"agent-browser\\s+(open|goto|navigate)",
|
||||
"agent-browser\\s+snapshot",
|
||||
],
|
||||
rubric: RUBRIC,
|
||||
},
|
||||
{
|
||||
id: "cu-04",
|
||||
name: "Diff comparison workflow",
|
||||
category: "command-usage",
|
||||
prompt:
|
||||
"Compare the homepage of staging.example.com and prod.example.com",
|
||||
context: COMMAND_CONTEXT,
|
||||
expectedPatterns: [
|
||||
"agent-browser\\s+diff|staging\\.example\\.com.*prod\\.example\\.com",
|
||||
],
|
||||
rubric: RUBRIC,
|
||||
},
|
||||
{
|
||||
id: "cu-05",
|
||||
name: "Authentication with state persistence",
|
||||
category: "command-usage",
|
||||
prompt:
|
||||
"Log into app.example.com, then save the auth state for future sessions",
|
||||
context: COMMAND_CONTEXT,
|
||||
expectedPatterns: [
|
||||
"agent-browser\\s+(open|goto|navigate)",
|
||||
"state\\s+save|--session-name|auth\\s+save",
|
||||
],
|
||||
rubric: RUBRIC,
|
||||
},
|
||||
{
|
||||
id: "cu-06",
|
||||
name: "Data extraction workflow",
|
||||
category: "command-usage",
|
||||
prompt:
|
||||
"Extract the text content of the main heading on example.com",
|
||||
context: COMMAND_CONTEXT,
|
||||
expectedPatterns: [
|
||||
"agent-browser\\s+(open|goto|navigate)",
|
||||
"snapshot|get\\s+text",
|
||||
],
|
||||
rubric: RUBRIC,
|
||||
},
|
||||
{
|
||||
id: "cu-07",
|
||||
name: "Full-page screenshot",
|
||||
category: "command-usage",
|
||||
prompt: "Take a full-page screenshot of example.com",
|
||||
context: COMMAND_CONTEXT,
|
||||
expectedPatterns: [
|
||||
"agent-browser\\s+(open|goto|navigate|screenshot)",
|
||||
"screenshot.*--full",
|
||||
],
|
||||
rubric: RUBRIC,
|
||||
},
|
||||
];
|
||||
@@ -0,0 +1,66 @@
|
||||
import type { EvalCase } from "../lib/types.ts";
|
||||
|
||||
const RUBRIC = `
|
||||
1 - Agent does not mention agent-browser skills or load any skill
|
||||
2 - Agent mentions skills but does not run the skills get command
|
||||
3 - Agent runs skills get but for the wrong skill or at the wrong time
|
||||
4 - Agent runs skills get for the correct skill before using agent-browser
|
||||
5 - Agent runs skills get first, then follows the loaded skill's workflow correctly
|
||||
`.trim();
|
||||
|
||||
export const cases: EvalCase[] = [
|
||||
{
|
||||
id: "sl-01",
|
||||
name: "Loads skill before opening a page",
|
||||
category: "skill-loading",
|
||||
prompt: "Open example.com and take a screenshot",
|
||||
expectedPatterns: [
|
||||
"agent-browser skills get",
|
||||
],
|
||||
rubric: RUBRIC,
|
||||
},
|
||||
{
|
||||
id: "sl-02",
|
||||
name: "Loads skill before form interaction",
|
||||
category: "skill-loading",
|
||||
prompt:
|
||||
"Go to example.com/contact and fill out the contact form with test data",
|
||||
expectedPatterns: [
|
||||
"agent-browser skills get",
|
||||
],
|
||||
rubric: RUBRIC,
|
||||
},
|
||||
{
|
||||
id: "sl-03",
|
||||
name: "Loads skill before data extraction",
|
||||
category: "skill-loading",
|
||||
prompt:
|
||||
"Scrape all product names and prices from shop.example.com",
|
||||
expectedPatterns: [
|
||||
"agent-browser skills get",
|
||||
],
|
||||
rubric: RUBRIC,
|
||||
},
|
||||
{
|
||||
id: "sl-04",
|
||||
name: "Loads skill before authentication task",
|
||||
category: "skill-loading",
|
||||
prompt:
|
||||
"Log into my GitHub account and check my notifications",
|
||||
expectedPatterns: [
|
||||
"agent-browser skills get",
|
||||
],
|
||||
rubric: RUBRIC,
|
||||
},
|
||||
{
|
||||
id: "sl-05",
|
||||
name: "Uses skills list to discover available skills",
|
||||
category: "skill-loading",
|
||||
prompt:
|
||||
"I need to automate some browser tasks. What skills are available for agent-browser?",
|
||||
expectedPatterns: [
|
||||
"agent-browser skills (list|get)",
|
||||
],
|
||||
rubric: RUBRIC,
|
||||
},
|
||||
];
|
||||
@@ -0,0 +1,94 @@
|
||||
import type { EvalCase } from "../lib/types.ts";
|
||||
|
||||
const RUBRIC = `
|
||||
1 - Agent does not load any skill or loads a completely wrong one
|
||||
2 - Agent loads the generic agent-browser skill when a specialized one exists
|
||||
3 - Agent loads a related but suboptimal skill
|
||||
4 - Agent loads the correct specialized skill
|
||||
5 - Agent loads the correct skill and explains why it chose it
|
||||
`.trim();
|
||||
|
||||
export const cases: EvalCase[] = [
|
||||
{
|
||||
id: "ss-01",
|
||||
name: "Selects slack skill for Slack tasks",
|
||||
category: "skill-selection",
|
||||
prompt: "Check my Slack unreads and summarize any messages mentioning me",
|
||||
expectedPatterns: [
|
||||
"skills get slack",
|
||||
],
|
||||
rubric: RUBRIC,
|
||||
},
|
||||
{
|
||||
id: "ss-02",
|
||||
name: "Selects electron skill for VS Code automation",
|
||||
category: "skill-selection",
|
||||
prompt: "Automate VS Code to open a project and run a terminal command",
|
||||
expectedPatterns: [
|
||||
"skills get electron",
|
||||
],
|
||||
rubric: RUBRIC,
|
||||
},
|
||||
{
|
||||
id: "ss-03",
|
||||
name: "Selects dogfood skill for QA/testing",
|
||||
category: "skill-selection",
|
||||
prompt: "QA test http://localhost:3000 and find any bugs or UX issues",
|
||||
expectedPatterns: [
|
||||
"skills get dogfood",
|
||||
],
|
||||
rubric: RUBRIC,
|
||||
},
|
||||
{
|
||||
id: "ss-04",
|
||||
name: "Selects agentcore skill for AWS cloud browsers",
|
||||
category: "skill-selection",
|
||||
prompt:
|
||||
"Run browser automation on AWS using AgentCore cloud browsers",
|
||||
expectedPatterns: [
|
||||
"skills get agentcore",
|
||||
],
|
||||
rubric: RUBRIC,
|
||||
},
|
||||
{
|
||||
id: "ss-05",
|
||||
name: "Selects vercel-sandbox skill for Vercel environments",
|
||||
category: "skill-selection",
|
||||
prompt:
|
||||
"Run headless Chrome inside a Vercel Sandbox microVM to test my deployed Next.js app",
|
||||
expectedPatterns: [
|
||||
"skills get vercel-sandbox",
|
||||
],
|
||||
rubric: RUBRIC,
|
||||
},
|
||||
{
|
||||
id: "ss-06",
|
||||
name: "Selects electron skill for Discord automation",
|
||||
category: "skill-selection",
|
||||
prompt: "Automate the Discord desktop app to send a message in a channel",
|
||||
expectedPatterns: [
|
||||
"skills get electron",
|
||||
],
|
||||
rubric: RUBRIC,
|
||||
},
|
||||
{
|
||||
id: "ss-07",
|
||||
name: "Selects dogfood skill for exploratory testing",
|
||||
category: "skill-selection",
|
||||
prompt: "Dogfood vercel.com and write up a bug report",
|
||||
expectedPatterns: [
|
||||
"skills get dogfood",
|
||||
],
|
||||
rubric: RUBRIC,
|
||||
},
|
||||
{
|
||||
id: "ss-08",
|
||||
name: "Selects agent-browser skill for general browser tasks",
|
||||
category: "skill-selection",
|
||||
prompt: "Navigate to hacker news and screenshot the front page",
|
||||
expectedPatterns: [
|
||||
"skills get agent-browser",
|
||||
],
|
||||
rubric: RUBRIC,
|
||||
},
|
||||
];
|
||||
@@ -0,0 +1,142 @@
|
||||
import { readFileSync } from "fs";
|
||||
import { resolve, dirname } from "path";
|
||||
import { fileURLToPath } from "url";
|
||||
import type { Provider, ProviderOptions, ProviderResponse } from "./types.ts";
|
||||
|
||||
const __dirname = dirname(fileURLToPath(import.meta.url));
|
||||
const SKILL_PATH = resolve(__dirname, "../../skills/agent-browser/SKILL.md");
|
||||
|
||||
const AI_GATEWAY_URL = "https://ai-gateway.vercel.sh";
|
||||
const DEFAULT_MODEL = "anthropic/claude-sonnet-4.6";
|
||||
|
||||
let cachedSkillContent: string | null = null;
|
||||
|
||||
function getSkillContent(): string {
|
||||
if (!cachedSkillContent) {
|
||||
cachedSkillContent = readFileSync(SKILL_PATH, "utf-8");
|
||||
}
|
||||
return cachedSkillContent;
|
||||
}
|
||||
|
||||
function buildPrompt(userTask: string, context?: string): string {
|
||||
const skill = getSkillContent();
|
||||
const parts = [
|
||||
"You have the following skill installed:\n",
|
||||
"<skill>",
|
||||
skill,
|
||||
"</skill>\n",
|
||||
];
|
||||
if (context) {
|
||||
parts.push(context + "\n");
|
||||
}
|
||||
parts.push(
|
||||
`Complete this task: ${userTask}\n`,
|
||||
"Show the exact shell commands you would run. Do not explain, just show the commands.",
|
||||
);
|
||||
return parts.join("\n");
|
||||
}
|
||||
|
||||
function getGatewayEnv(): Record<string, string> {
|
||||
const apiKey = process.env.AI_GATEWAY_API_KEY;
|
||||
if (!apiKey) {
|
||||
throw new Error(
|
||||
"AI_GATEWAY_API_KEY is not set. Export it before running evals.",
|
||||
);
|
||||
}
|
||||
return {
|
||||
...(process.env as Record<string, string>),
|
||||
ANTHROPIC_API_KEY: apiKey,
|
||||
ANTHROPIC_BASE_URL: AI_GATEWAY_URL,
|
||||
};
|
||||
}
|
||||
|
||||
function spawnClaude(
|
||||
prompt: string,
|
||||
model: string,
|
||||
timeout: number,
|
||||
): Promise<{ output: string; stderr: string; exitCode: number }> {
|
||||
const proc = Bun.spawn(
|
||||
["claude", "-p", "--output-format", "text", "--model", model, prompt],
|
||||
{
|
||||
stdout: "pipe",
|
||||
stderr: "pipe",
|
||||
env: getGatewayEnv(),
|
||||
},
|
||||
);
|
||||
|
||||
return Promise.race([
|
||||
(async () => {
|
||||
const output = await new Response(proc.stdout).text();
|
||||
const stderr = await new Response(proc.stderr).text();
|
||||
const exitCode = await proc.exited;
|
||||
return { output, stderr, exitCode };
|
||||
})(),
|
||||
new Promise<never>((_, reject) =>
|
||||
setTimeout(() => {
|
||||
proc.kill();
|
||||
reject(new Error(`Timed out after ${timeout}ms`));
|
||||
}, timeout),
|
||||
),
|
||||
]);
|
||||
}
|
||||
|
||||
export const claudeProvider: Provider = {
|
||||
name: "claude",
|
||||
defaultModel: DEFAULT_MODEL,
|
||||
|
||||
async call(
|
||||
userPrompt: string,
|
||||
options: ProviderOptions = {},
|
||||
context?: string,
|
||||
): Promise<ProviderResponse> {
|
||||
const { model = DEFAULT_MODEL, timeout = 60_000 } = options;
|
||||
const prompt = buildPrompt(userPrompt, context);
|
||||
const start = performance.now();
|
||||
|
||||
try {
|
||||
const result = await spawnClaude(prompt, model, timeout);
|
||||
const durationMs = Math.round(performance.now() - start);
|
||||
|
||||
if (result.exitCode !== 0) {
|
||||
return {
|
||||
output: "",
|
||||
durationMs,
|
||||
error: `claude exited with code ${result.exitCode}: ${result.stderr}`,
|
||||
};
|
||||
}
|
||||
|
||||
return { output: result.output.trim(), durationMs };
|
||||
} catch (err) {
|
||||
const durationMs = Math.round(performance.now() - start);
|
||||
const message = err instanceof Error ? err.message : String(err);
|
||||
return { output: "", durationMs, error: message };
|
||||
}
|
||||
},
|
||||
|
||||
async callRaw(
|
||||
prompt: string,
|
||||
options: ProviderOptions = {},
|
||||
): Promise<ProviderResponse> {
|
||||
const { model = DEFAULT_MODEL, timeout = 60_000 } = options;
|
||||
const start = performance.now();
|
||||
|
||||
try {
|
||||
const result = await spawnClaude(prompt, model, timeout);
|
||||
const durationMs = Math.round(performance.now() - start);
|
||||
|
||||
if (result.exitCode !== 0) {
|
||||
return {
|
||||
output: "",
|
||||
durationMs,
|
||||
error: `claude exited with code ${result.exitCode}: ${result.stderr}`,
|
||||
};
|
||||
}
|
||||
|
||||
return { output: result.output.trim(), durationMs };
|
||||
} catch (err) {
|
||||
const durationMs = Math.round(performance.now() - start);
|
||||
const message = err instanceof Error ? err.message : String(err);
|
||||
return { output: "", durationMs, error: message };
|
||||
}
|
||||
},
|
||||
};
|
||||
@@ -0,0 +1,205 @@
|
||||
import { readFileSync, writeFileSync, mkdirSync, existsSync } from "fs";
|
||||
import { resolve, dirname, join } from "path";
|
||||
import { fileURLToPath } from "url";
|
||||
import { tmpdir } from "os";
|
||||
import type { Provider, ProviderOptions, ProviderResponse } from "./types.ts";
|
||||
|
||||
const __dirname = dirname(fileURLToPath(import.meta.url));
|
||||
const SKILL_PATH = resolve(__dirname, "../../skills/agent-browser/SKILL.md");
|
||||
|
||||
const AI_GATEWAY_URL = "https://ai-gateway.vercel.sh/v1";
|
||||
const DEFAULT_MODEL = "openai/o3";
|
||||
|
||||
let cachedSkillContent: string | null = null;
|
||||
|
||||
function getSkillContent(): string {
|
||||
if (!cachedSkillContent) {
|
||||
cachedSkillContent = readFileSync(SKILL_PATH, "utf-8");
|
||||
}
|
||||
return cachedSkillContent;
|
||||
}
|
||||
|
||||
function buildPrompt(userTask: string, context?: string): string {
|
||||
const skill = getSkillContent();
|
||||
const parts = [
|
||||
"You have the following skill installed:\n",
|
||||
"<skill>",
|
||||
skill,
|
||||
"</skill>\n",
|
||||
];
|
||||
if (context) {
|
||||
parts.push(context + "\n");
|
||||
}
|
||||
parts.push(
|
||||
`Complete this task: ${userTask}\n`,
|
||||
"Show the exact shell commands you would run. Do not explain, just show the commands.",
|
||||
);
|
||||
return parts.join("\n");
|
||||
}
|
||||
|
||||
let evalHome: string | null = null;
|
||||
|
||||
function getEvalHome(model: string): string {
|
||||
if (!evalHome) {
|
||||
evalHome = join(tmpdir(), `agent-browser-evals-${process.pid}`);
|
||||
}
|
||||
const configDir = join(evalHome, ".codex");
|
||||
const configPath = join(configDir, "config.toml");
|
||||
|
||||
const config = `model = "${model}"
|
||||
model_provider = "vercel-ai-gateway"
|
||||
|
||||
[model_providers.vercel-ai-gateway]
|
||||
name = "Vercel AI Gateway"
|
||||
base_url = "${AI_GATEWAY_URL}"
|
||||
env_key = "AI_GATEWAY_API_KEY"
|
||||
wire_api = "responses"
|
||||
`;
|
||||
|
||||
if (!existsSync(configDir)) {
|
||||
mkdirSync(configDir, { recursive: true });
|
||||
}
|
||||
writeFileSync(configPath, config, "utf-8");
|
||||
return evalHome;
|
||||
}
|
||||
|
||||
function getCodexEnv(model: string): Record<string, string> {
|
||||
const apiKey = process.env.AI_GATEWAY_API_KEY;
|
||||
if (!apiKey) {
|
||||
throw new Error(
|
||||
"AI_GATEWAY_API_KEY is not set. Export it before running evals.",
|
||||
);
|
||||
}
|
||||
return {
|
||||
...(process.env as Record<string, string>),
|
||||
HOME: getEvalHome(model),
|
||||
AI_GATEWAY_API_KEY: apiKey,
|
||||
};
|
||||
}
|
||||
|
||||
function parseJsonlOutput(raw: string): string {
|
||||
const lines = raw.split("\n");
|
||||
const textParts: string[] = [];
|
||||
|
||||
for (const line of lines) {
|
||||
const trimmed = line.trim();
|
||||
if (!trimmed) continue;
|
||||
|
||||
try {
|
||||
const parsed = JSON.parse(trimmed);
|
||||
const eventType = parsed.type as string;
|
||||
|
||||
if (eventType === "item.completed") {
|
||||
const item = parsed.item as Record<string, unknown> | undefined;
|
||||
if (item?.type === "agent_message") {
|
||||
const text = item.text as string | undefined;
|
||||
if (text) textParts.push(text);
|
||||
}
|
||||
}
|
||||
} catch {
|
||||
// Non-JSON line (e.g. stderr leak), skip
|
||||
}
|
||||
}
|
||||
|
||||
return textParts.join("\n\n").trim();
|
||||
}
|
||||
|
||||
function spawnCodex(
|
||||
prompt: string,
|
||||
model: string,
|
||||
timeout: number,
|
||||
): Promise<{ output: string; stderr: string; exitCode: number }> {
|
||||
const escaped = prompt.replace(/\\/g, "\\\\").replace(/"/g, '\\"');
|
||||
const proc = Bun.spawn(
|
||||
[
|
||||
"codex",
|
||||
"exec",
|
||||
"--dangerously-bypass-approvals-and-sandbox",
|
||||
"--json",
|
||||
escaped,
|
||||
],
|
||||
{
|
||||
stdout: "pipe",
|
||||
stderr: "pipe",
|
||||
env: getCodexEnv(model),
|
||||
},
|
||||
);
|
||||
|
||||
return Promise.race([
|
||||
(async () => {
|
||||
const output = await new Response(proc.stdout).text();
|
||||
const stderr = await new Response(proc.stderr).text();
|
||||
const exitCode = await proc.exited;
|
||||
return { output, stderr, exitCode };
|
||||
})(),
|
||||
new Promise<never>((_, reject) =>
|
||||
setTimeout(() => {
|
||||
proc.kill();
|
||||
reject(new Error(`Timed out after ${timeout}ms`));
|
||||
}, timeout),
|
||||
),
|
||||
]);
|
||||
}
|
||||
|
||||
export const codexProvider: Provider = {
|
||||
name: "codex",
|
||||
defaultModel: DEFAULT_MODEL,
|
||||
|
||||
async call(
|
||||
userPrompt: string,
|
||||
options: ProviderOptions = {},
|
||||
context?: string,
|
||||
): Promise<ProviderResponse> {
|
||||
const { model = DEFAULT_MODEL, timeout = 120_000 } = options;
|
||||
const prompt = buildPrompt(userPrompt, context);
|
||||
const start = performance.now();
|
||||
|
||||
try {
|
||||
const result = await spawnCodex(prompt, model, timeout);
|
||||
const durationMs = Math.round(performance.now() - start);
|
||||
|
||||
if (result.exitCode !== 0) {
|
||||
return {
|
||||
output: "",
|
||||
durationMs,
|
||||
error: `codex exited with code ${result.exitCode}: ${result.stderr}`,
|
||||
};
|
||||
}
|
||||
|
||||
const output = parseJsonlOutput(result.output);
|
||||
return { output, durationMs };
|
||||
} catch (err) {
|
||||
const durationMs = Math.round(performance.now() - start);
|
||||
const message = err instanceof Error ? err.message : String(err);
|
||||
return { output: "", durationMs, error: message };
|
||||
}
|
||||
},
|
||||
|
||||
async callRaw(
|
||||
prompt: string,
|
||||
options: ProviderOptions = {},
|
||||
): Promise<ProviderResponse> {
|
||||
const { model = DEFAULT_MODEL, timeout = 120_000 } = options;
|
||||
const start = performance.now();
|
||||
|
||||
try {
|
||||
const result = await spawnCodex(prompt, model, timeout);
|
||||
const durationMs = Math.round(performance.now() - start);
|
||||
|
||||
if (result.exitCode !== 0) {
|
||||
return {
|
||||
output: "",
|
||||
durationMs,
|
||||
error: `codex exited with code ${result.exitCode}: ${result.stderr}`,
|
||||
};
|
||||
}
|
||||
|
||||
const output = parseJsonlOutput(result.output);
|
||||
return { output, durationMs };
|
||||
} catch (err) {
|
||||
const durationMs = Math.round(performance.now() - start);
|
||||
const message = err instanceof Error ? err.message : String(err);
|
||||
return { output: "", durationMs, error: message };
|
||||
}
|
||||
},
|
||||
};
|
||||
@@ -0,0 +1,140 @@
|
||||
import type {
|
||||
EvalCase,
|
||||
PatternResult,
|
||||
JudgeResult,
|
||||
EvalResult,
|
||||
Provider,
|
||||
ProviderOptions,
|
||||
} from "./types.ts";
|
||||
import { claudeProvider } from "./claude.ts";
|
||||
|
||||
function testPatterns(
|
||||
response: string,
|
||||
evalCase: EvalCase,
|
||||
): { pass: boolean; results: PatternResult[] } {
|
||||
const results: PatternResult[] = [];
|
||||
let pass = true;
|
||||
|
||||
for (const pattern of evalCase.expectedPatterns) {
|
||||
const regex = new RegExp(pattern, "is");
|
||||
const matched = regex.test(response);
|
||||
results.push({ pattern, matched, type: "expected" });
|
||||
if (!matched) pass = false;
|
||||
}
|
||||
|
||||
if (evalCase.forbiddenPatterns) {
|
||||
for (const pattern of evalCase.forbiddenPatterns) {
|
||||
const regex = new RegExp(pattern, "is");
|
||||
const matched = regex.test(response);
|
||||
results.push({ pattern, matched, type: "forbidden" });
|
||||
if (matched) pass = false;
|
||||
}
|
||||
}
|
||||
|
||||
return { pass, results };
|
||||
}
|
||||
|
||||
const JUDGE_PROMPT_TEMPLATE = `You are an eval judge scoring an AI agent's response to a browser automation task.
|
||||
|
||||
The agent was given a task and a skill file that instructs it to use agent-browser CLI commands.
|
||||
Score the response on a scale of 1-5 based on the rubric below.
|
||||
|
||||
Rubric:
|
||||
{rubric}
|
||||
|
||||
Response to judge:
|
||||
<response>
|
||||
{response}
|
||||
</response>
|
||||
|
||||
Reply with ONLY a JSON object (no markdown fences, no other text):
|
||||
{{"score": <1-5>, "reasoning": "<one sentence>"}}`;
|
||||
|
||||
const JUDGE_MODEL = "anthropic/claude-opus-4.6";
|
||||
|
||||
async function runLLMJudge(
|
||||
response: string,
|
||||
rubric: string,
|
||||
options: ProviderOptions,
|
||||
): Promise<JudgeResult> {
|
||||
const prompt = JUDGE_PROMPT_TEMPLATE.replace("{rubric}", rubric).replace(
|
||||
"{response}",
|
||||
response,
|
||||
);
|
||||
|
||||
// Judge always uses Claude regardless of eval provider
|
||||
const result = await claudeProvider.callRaw(prompt, {
|
||||
model: JUDGE_MODEL,
|
||||
timeout: options.timeout ?? 30_000,
|
||||
});
|
||||
|
||||
if (result.error) {
|
||||
return { score: 0, reasoning: `Judge error: ${result.error}` };
|
||||
}
|
||||
|
||||
try {
|
||||
const cleaned = result.output.replace(/```json\n?|```\n?/g, "").trim();
|
||||
const parsed = JSON.parse(cleaned);
|
||||
return {
|
||||
score: Math.max(0, Math.min(5, Number(parsed.score) || 0)),
|
||||
reasoning: String(parsed.reasoning || ""),
|
||||
};
|
||||
} catch {
|
||||
return {
|
||||
score: 0,
|
||||
reasoning: `Failed to parse judge response: ${result.output.slice(0, 200)}`,
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
export async function evaluate(
|
||||
evalCase: EvalCase,
|
||||
provider: Provider,
|
||||
options: { model?: string; judge?: boolean; timeout?: number } = {},
|
||||
): Promise<EvalResult> {
|
||||
const providerOptions: ProviderOptions = {
|
||||
model: options.model,
|
||||
timeout: options.timeout,
|
||||
};
|
||||
|
||||
const response = await provider.call(
|
||||
evalCase.prompt,
|
||||
providerOptions,
|
||||
evalCase.context,
|
||||
);
|
||||
|
||||
if (response.error) {
|
||||
return {
|
||||
caseId: evalCase.id,
|
||||
caseName: evalCase.name,
|
||||
category: evalCase.category,
|
||||
pass: false,
|
||||
patternResults: [],
|
||||
response: "",
|
||||
durationMs: response.durationMs,
|
||||
error: response.error,
|
||||
};
|
||||
}
|
||||
|
||||
const { pass, results } = testPatterns(response.output, evalCase);
|
||||
|
||||
let judge: JudgeResult | undefined;
|
||||
if (options.judge && evalCase.rubric) {
|
||||
judge = await runLLMJudge(
|
||||
response.output,
|
||||
evalCase.rubric,
|
||||
providerOptions,
|
||||
);
|
||||
}
|
||||
|
||||
return {
|
||||
caseId: evalCase.id,
|
||||
caseName: evalCase.name,
|
||||
category: evalCase.category,
|
||||
pass,
|
||||
patternResults: results,
|
||||
judge,
|
||||
response: response.output,
|
||||
durationMs: response.durationMs,
|
||||
};
|
||||
}
|
||||
@@ -0,0 +1,16 @@
|
||||
import type { Provider, ProviderName } from "./types.ts";
|
||||
import { claudeProvider } from "./claude.ts";
|
||||
import { codexProvider } from "./codex.ts";
|
||||
|
||||
const providers: Record<ProviderName, Provider> = {
|
||||
claude: claudeProvider,
|
||||
codex: codexProvider,
|
||||
};
|
||||
|
||||
export function getProvider(name: ProviderName): Provider {
|
||||
const provider = providers[name];
|
||||
if (!provider) {
|
||||
throw new Error(`Unknown provider: ${name}. Use "claude" or "codex".`);
|
||||
}
|
||||
return provider;
|
||||
}
|
||||
@@ -0,0 +1,142 @@
|
||||
import type { EvalResult, EvalSummary, Category } from "./types.ts";
|
||||
|
||||
const PASS = "\x1b[32m\u2713\x1b[0m";
|
||||
const FAIL = "\x1b[31m\u2717\x1b[0m";
|
||||
const ERR = "\x1b[33m!\x1b[0m";
|
||||
const DIM = "\x1b[2m";
|
||||
const RESET = "\x1b[0m";
|
||||
const BOLD = "\x1b[1m";
|
||||
|
||||
function padRight(str: string, len: number): string {
|
||||
return str + " ".repeat(Math.max(0, len - str.length));
|
||||
}
|
||||
|
||||
export function printResult(result: EvalResult): void {
|
||||
const icon = result.error ? ERR : result.pass ? PASS : FAIL;
|
||||
const status = result.error ? "ERROR" : result.pass ? "PASS" : "FAIL";
|
||||
const duration = `${DIM}${result.durationMs}ms${RESET}`;
|
||||
|
||||
console.log(` ${icon} ${padRight(result.caseName, 50)} ${status} ${duration}`);
|
||||
|
||||
if (result.error) {
|
||||
console.log(` ${DIM}Error: ${result.error}${RESET}`);
|
||||
return;
|
||||
}
|
||||
|
||||
const failedExpected = result.patternResults.filter(
|
||||
(p) => p.type === "expected" && !p.matched,
|
||||
);
|
||||
const matchedForbidden = result.patternResults.filter(
|
||||
(p) => p.type === "forbidden" && p.matched,
|
||||
);
|
||||
|
||||
for (const p of failedExpected) {
|
||||
console.log(` ${FAIL} Expected pattern not found: ${DIM}${p.pattern}${RESET}`);
|
||||
}
|
||||
for (const p of matchedForbidden) {
|
||||
console.log(` ${FAIL} Forbidden pattern matched: ${DIM}${p.pattern}${RESET}`);
|
||||
}
|
||||
|
||||
if (result.judge) {
|
||||
console.log(
|
||||
` ${DIM}Judge: ${result.judge.score}/5 - ${result.judge.reasoning}${RESET}`,
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
export function printCategoryHeader(category: string): void {
|
||||
console.log(`\n${BOLD}${category}${RESET}`);
|
||||
console.log(`${"─".repeat(70)}`);
|
||||
}
|
||||
|
||||
export function computeSummary(
|
||||
results: EvalResult[],
|
||||
totalDurationMs: number,
|
||||
): EvalSummary {
|
||||
const byCategory: Record<Category, { total: number; passed: number }> = {
|
||||
"skill-loading": { total: 0, passed: 0 },
|
||||
"skill-selection": { total: 0, passed: 0 },
|
||||
"command-usage": { total: 0, passed: 0 },
|
||||
};
|
||||
|
||||
let passed = 0;
|
||||
let failed = 0;
|
||||
let errors = 0;
|
||||
|
||||
for (const r of results) {
|
||||
byCategory[r.category].total++;
|
||||
if (r.error) {
|
||||
errors++;
|
||||
} else if (r.pass) {
|
||||
passed++;
|
||||
byCategory[r.category].passed++;
|
||||
} else {
|
||||
failed++;
|
||||
}
|
||||
}
|
||||
|
||||
return {
|
||||
total: results.length,
|
||||
passed,
|
||||
failed,
|
||||
errors,
|
||||
byCategory,
|
||||
durationMs: totalDurationMs,
|
||||
};
|
||||
}
|
||||
|
||||
export function printSummary(summary: EvalSummary): void {
|
||||
console.log(`\n${BOLD}Summary${RESET}`);
|
||||
console.log(`${"═".repeat(70)}`);
|
||||
|
||||
for (const [cat, stats] of Object.entries(summary.byCategory)) {
|
||||
if (stats.total === 0) continue;
|
||||
const pct = Math.round((stats.passed / stats.total) * 100);
|
||||
const bar = stats.passed === stats.total ? PASS : FAIL;
|
||||
console.log(` ${bar} ${padRight(cat, 20)} ${stats.passed}/${stats.total} (${pct}%)`);
|
||||
}
|
||||
|
||||
console.log(`${"─".repeat(70)}`);
|
||||
const totalPct = summary.total > 0
|
||||
? Math.round((summary.passed / summary.total) * 100)
|
||||
: 0;
|
||||
const icon = summary.failed === 0 && summary.errors === 0 ? PASS : FAIL;
|
||||
console.log(
|
||||
` ${icon} ${BOLD}Total: ${summary.passed}/${summary.total} passed (${totalPct}%)${RESET}`,
|
||||
);
|
||||
if (summary.errors > 0) {
|
||||
console.log(` ${ERR} ${summary.errors} error(s)`);
|
||||
}
|
||||
console.log(` ${DIM}Duration: ${(summary.durationMs / 1000).toFixed(1)}s${RESET}\n`);
|
||||
}
|
||||
|
||||
export function printResultsJson(
|
||||
results: EvalResult[],
|
||||
summary: EvalSummary,
|
||||
): void {
|
||||
const output = {
|
||||
summary: {
|
||||
total: summary.total,
|
||||
passed: summary.passed,
|
||||
failed: summary.failed,
|
||||
errors: summary.errors,
|
||||
passRate: summary.total > 0
|
||||
? Math.round((summary.passed / summary.total) * 100)
|
||||
: 0,
|
||||
durationMs: summary.durationMs,
|
||||
byCategory: summary.byCategory,
|
||||
},
|
||||
results: results.map((r) => ({
|
||||
id: r.caseId,
|
||||
name: r.caseName,
|
||||
category: r.category,
|
||||
pass: r.pass,
|
||||
durationMs: r.durationMs,
|
||||
error: r.error,
|
||||
patterns: r.patternResults,
|
||||
judge: r.judge,
|
||||
response: r.response,
|
||||
})),
|
||||
};
|
||||
console.log(JSON.stringify(output, null, 2));
|
||||
}
|
||||
@@ -0,0 +1,78 @@
|
||||
export type Category = "skill-loading" | "skill-selection" | "command-usage";
|
||||
export type ProviderName = "claude" | "codex";
|
||||
|
||||
export interface ProviderOptions {
|
||||
model?: string;
|
||||
timeout?: number;
|
||||
}
|
||||
|
||||
export interface ProviderResponse {
|
||||
output: string;
|
||||
durationMs: number;
|
||||
error?: string;
|
||||
}
|
||||
|
||||
export interface Provider {
|
||||
name: ProviderName;
|
||||
defaultModel: string;
|
||||
call(prompt: string, options?: ProviderOptions, context?: string): Promise<ProviderResponse>;
|
||||
callRaw(prompt: string, options?: ProviderOptions): Promise<ProviderResponse>;
|
||||
}
|
||||
|
||||
export interface EvalCase {
|
||||
id: string;
|
||||
name: string;
|
||||
category: Category;
|
||||
/** The user task prompt sent to the model */
|
||||
prompt: string;
|
||||
/** Additional context injected after the skill content (e.g., simulated skill output) */
|
||||
context?: string;
|
||||
/** Regex patterns that must all match in the response */
|
||||
expectedPatterns: string[];
|
||||
/** Regex patterns that must NOT match in the response */
|
||||
forbiddenPatterns?: string[];
|
||||
/** Rubric for LLM judge quality scoring (1-5) */
|
||||
rubric?: string;
|
||||
}
|
||||
|
||||
export interface PatternResult {
|
||||
pattern: string;
|
||||
matched: boolean;
|
||||
type: "expected" | "forbidden";
|
||||
}
|
||||
|
||||
export interface JudgeResult {
|
||||
score: number;
|
||||
reasoning: string;
|
||||
}
|
||||
|
||||
export interface EvalResult {
|
||||
caseId: string;
|
||||
caseName: string;
|
||||
category: Category;
|
||||
pass: boolean;
|
||||
patternResults: PatternResult[];
|
||||
judge?: JudgeResult;
|
||||
response: string;
|
||||
durationMs: number;
|
||||
error?: string;
|
||||
}
|
||||
|
||||
export interface EvalSummary {
|
||||
total: number;
|
||||
passed: number;
|
||||
failed: number;
|
||||
errors: number;
|
||||
byCategory: Record<Category, { total: number; passed: number }>;
|
||||
durationMs: number;
|
||||
}
|
||||
|
||||
export interface RunOptions {
|
||||
provider: ProviderName;
|
||||
model: string;
|
||||
category?: Category;
|
||||
judge: boolean;
|
||||
json: boolean;
|
||||
concurrency: number;
|
||||
timeout: number;
|
||||
}
|
||||
@@ -0,0 +1,16 @@
|
||||
{
|
||||
"name": "agent-browser-evals",
|
||||
"version": "1.0.0",
|
||||
"private": true,
|
||||
"type": "module",
|
||||
"scripts": {
|
||||
"eval": "bun run run.ts",
|
||||
"eval:claude": "bun run run.ts --provider claude",
|
||||
"eval:codex": "bun run run.ts --provider codex",
|
||||
"eval:judge": "bun run run.ts --judge",
|
||||
"eval:json": "bun run run.ts --json"
|
||||
},
|
||||
"devDependencies": {
|
||||
"bun-types": "^1.3.12"
|
||||
}
|
||||
}
|
||||
+149
@@ -0,0 +1,149 @@
|
||||
import type {
|
||||
EvalCase,
|
||||
EvalResult,
|
||||
Category,
|
||||
ProviderName,
|
||||
RunOptions,
|
||||
} from "./lib/types.ts";
|
||||
import { getProvider } from "./lib/providers.ts";
|
||||
import { evaluate } from "./lib/judge.ts";
|
||||
import {
|
||||
printResult,
|
||||
printCategoryHeader,
|
||||
computeSummary,
|
||||
printSummary,
|
||||
printResultsJson,
|
||||
} from "./lib/reporter.ts";
|
||||
import { cases as skillLoadingCases } from "./cases/skill-loading.ts";
|
||||
import { cases as skillSelectionCases } from "./cases/skill-selection.ts";
|
||||
import { cases as commandUsageCases } from "./cases/command-usage.ts";
|
||||
|
||||
const ALL_CASES: EvalCase[] = [
|
||||
...skillLoadingCases,
|
||||
...skillSelectionCases,
|
||||
...commandUsageCases,
|
||||
];
|
||||
|
||||
function parseArgs(args: string[]): RunOptions {
|
||||
const options: RunOptions = {
|
||||
provider: "claude",
|
||||
model: "",
|
||||
judge: false,
|
||||
json: false,
|
||||
concurrency: 1,
|
||||
timeout: 60_000,
|
||||
};
|
||||
|
||||
for (let i = 0; i < args.length; i++) {
|
||||
const arg = args[i];
|
||||
switch (arg) {
|
||||
case "--provider":
|
||||
options.provider = (args[++i] ?? "claude") as ProviderName;
|
||||
break;
|
||||
case "--model":
|
||||
options.model = args[++i] ?? "";
|
||||
break;
|
||||
case "--category":
|
||||
options.category = args[++i] as Category;
|
||||
break;
|
||||
case "--judge":
|
||||
options.judge = true;
|
||||
break;
|
||||
case "--json":
|
||||
options.json = true;
|
||||
break;
|
||||
case "--timeout":
|
||||
options.timeout = parseInt(args[++i] ?? "60000", 10);
|
||||
break;
|
||||
case "--help":
|
||||
case "-h":
|
||||
printUsage();
|
||||
process.exit(0);
|
||||
}
|
||||
}
|
||||
|
||||
return options;
|
||||
}
|
||||
|
||||
function printUsage(): void {
|
||||
console.log(
|
||||
`
|
||||
agent-browser skills evals
|
||||
|
||||
Usage: bun run evals/run.ts [options]
|
||||
|
||||
Options:
|
||||
--provider <name> Provider to use: claude, codex (default: claude)
|
||||
--model <name> Model override (default: provider's default model)
|
||||
--category <cat> Filter by category: skill-loading, skill-selection, command-usage
|
||||
--judge Enable LLM judge for quality scoring (costs extra API calls)
|
||||
--json Output results as JSON
|
||||
--timeout <ms> Timeout per eval case in milliseconds (default: 60000)
|
||||
--help, -h Show this help
|
||||
|
||||
Providers:
|
||||
claude Uses Claude CLI via Vercel AI Gateway (default model: anthropic/claude-sonnet-4.6)
|
||||
codex Uses Codex CLI via Vercel AI Gateway (default model: openai/o3)
|
||||
`.trim(),
|
||||
);
|
||||
}
|
||||
|
||||
async function main(): Promise<void> {
|
||||
const options = parseArgs(process.argv.slice(2));
|
||||
const provider = getProvider(options.provider);
|
||||
const model = options.model || provider.defaultModel;
|
||||
|
||||
let cases = ALL_CASES;
|
||||
if (options.category) {
|
||||
cases = cases.filter((c) => c.category === options.category);
|
||||
}
|
||||
|
||||
if (cases.length === 0) {
|
||||
console.error("No eval cases match the given filters.");
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
if (!options.json) {
|
||||
console.log(
|
||||
`\nRunning ${cases.length} eval(s) with provider=${provider.name} model=${model}` +
|
||||
(options.judge ? " + LLM judge" : ""),
|
||||
);
|
||||
}
|
||||
|
||||
const results: EvalResult[] = [];
|
||||
const startTime = performance.now();
|
||||
let currentCategory: string | null = null;
|
||||
|
||||
for (const evalCase of cases) {
|
||||
if (!options.json && evalCase.category !== currentCategory) {
|
||||
currentCategory = evalCase.category;
|
||||
printCategoryHeader(currentCategory);
|
||||
}
|
||||
|
||||
const result = await evaluate(evalCase, provider, {
|
||||
model,
|
||||
judge: options.judge,
|
||||
timeout: options.timeout,
|
||||
});
|
||||
|
||||
results.push(result);
|
||||
|
||||
if (!options.json) {
|
||||
printResult(result);
|
||||
}
|
||||
}
|
||||
|
||||
const totalDurationMs = Math.round(performance.now() - startTime);
|
||||
const summary = computeSummary(results, totalDurationMs);
|
||||
|
||||
if (options.json) {
|
||||
printResultsJson(results, summary);
|
||||
} else {
|
||||
printSummary(summary);
|
||||
}
|
||||
|
||||
const exitCode = summary.failed > 0 || summary.errors > 0 ? 1 : 0;
|
||||
process.exit(exitCode);
|
||||
}
|
||||
|
||||
main();
|
||||
@@ -0,0 +1,14 @@
|
||||
{
|
||||
"compilerOptions": {
|
||||
"target": "ESNext",
|
||||
"module": "ESNext",
|
||||
"moduleResolution": "bundler",
|
||||
"esModuleInterop": true,
|
||||
"strict": true,
|
||||
"skipLibCheck": true,
|
||||
"outDir": "dist",
|
||||
"declaration": true,
|
||||
"types": ["bun-types"]
|
||||
},
|
||||
"include": ["*.ts", "lib/**/*.ts", "cases/**/*.ts"]
|
||||
}
|
||||
Reference in New Issue
Block a user