585d93a02b9e88602fd2c5fb7bb6462e505cd96a
13
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Chris Tate
|
585d93a02b |
feat(tabs): t<N> prefix for tab ids; --label for named tabs; drop --tab peek flag (#1250)
* fix(tabs): preserve refs across --tab peek and cover outer-tab-closed path Follow-up to #1249 so `--tab <id>` is actually useful for agents: - Save and restore the outer tab's `ref_map`, `iframe_sessions`, and `active_frame_id` across a scoped command instead of clearing them. `snapshot` → `--tab N <cmd>` → `click @e1` now keeps the outer tab's refs intact. Scoped commands still see a clean slate so outer refs can't resolve against the scoped tab's DOM. - Close the coverage gap the Vercel review bot flagged on #1249: the previous `e2e_tab_scoped_command_handles_outer_tab_closed` test used `tab_close`, which is in the scoped-dispatch exclusion list, so it never exercised the restore-skip branch it claimed to test. Renamed to `e2e_tab_close_with_tab_id_closes_active_tab` with an honest docstring, and added `e2e_tab_scoped_command_outer_tab_closed_mid_dispatch` that actually hits the branch via `window.opener.close()` on a script-opened intermediate tab. - Add `e2e_tab_scoped_command_isolates_refs_from_outer_tab` pinning that outer refs don't bleed into the scoped tab's DOM resolution. - Rewrite `e2e_tab_scoped_command_clears_state_on_switch` as `e2e_tab_scoped_command_preserves_outer_tab_state`, verifying the restored @e1 still clicks end-to-end. - Update the 52 `--help` entries for `--tab <id>` to describe peek / restore semantics instead of a vague "Target specific tab ID". - Update README, docs site, config schema, and the agent-facing skills reference with working examples (refs survive the peek) and a "when to use \`--tab <id>\` vs \`tab <id>\`" guide so agents pick the right flag for their workflow. * fix(tabs): use t<N> prefix for tab ids, add --label for named tabs Follow-on to the tab work in #1249 and the prior commit, redesigning the tab handle surface before release since nothing ships these features yet. ## Why Incrementing integer tab ids (`1`, `2`, `3`) look indistinguishable from positional indices in command output, LLM-generated scripts, and docs. In the common single-agent case where position and id coincide, readers have no visual cue for which mental model they're using. Positional indices silently shift when unrelated tabs open/close, so misreading a handle as an index is a correctness hazard. ## Changes **Tab ids are now `t1`, `t2`, `t3` (strings).** Bare integer `tabId` values are rejected with a teaching message rather than silently accepted. The `t` prefix matches the `@e1` element-ref convention and makes ids unmistakably non-positional at a glance. **Labels.** Tabs can be created with a user-assigned label (e.g. `docs`, `app`) via `tab new --label <name> [url]`. Labels are interchangeable with `t<N>` ids everywhere a tab ref is accepted. They're never auto-generated, never rewritten on navigation, and must be unique within a session. **Dashboard fix.** `packages/dashboard/src/types.ts` declared `TabInfo.index: number` but the daemon has been sending `tabId` (not `index`) since #892, making `tab.index` `undefined` and breaking the dashboard's close/switch buttons silently. Updated the TS types and usages to consume `tabId` (string) and optional `label`, restoring the dashboard's tab interactions. ## Surface - `cli/src/native/browser.rs`: `TabRef::parse` / `format_tab_id` / `is_valid_label` / `PageInfo.label` / `BrowserManager::resolve_tab_ref` / `BrowserManager::has_label`. `tab_new` gains an optional label argument with duplicate rejection. All JSON responses use the string form and include the label. - `cli/src/native/actions.rs`: scoped-command pre-dispatch and `handle_tab_{switch,close,new}` parse string refs and resolve to stable ids. - `cli/src/{flags,commands,main,output}.rs`: `--tab` / config `tab` are `String`; `tab` subcommand accepts `t<N>` or a label and supports `tab new --label <name> [url]`. All 52 `--help` entries updated. - `agent-browser.schema.json`: `tab` property type is now `string` with a pattern matching `t<N>` or label form. - `packages/dashboard`: `TabInfo.tabId: string` / `label?: string | null`; `closeTabAtom`/`switchTabAtom` take `tabRef: string`; component props updated. - Docs: README, docs site (`commands/` and `configuration/`), and the agent-facing skills reference rewritten with the new examples. ## Tests - Added `TabRef::parse` / `format_tab_id` / `is_valid_label` unit tests pinning the bare-integer rejection, the teaching error, label rules, and round-tripping. - Added `test_tab_switch_by_id` / `_by_label` / `test_tab_new_with_label` / `_with_label_and_url` / `_with_url_then_label` in `commands.rs`; rewrote `test_tab_unknown_subcommand_errors` since labels make `tab select` a legitimate ref. - Added `e2e_tab_new_with_label_can_be_switched_and_peeked`, `e2e_tab_new_with_duplicate_label_errors`, `e2e_tab_scoped_command_rejects_bare_integer`. - Migrated every existing tab e2e test (and one unit test) from integer `tabId` to the string form. `cargo fmt`, `cargo clippy -- -D warnings`, all 30 non-ignored tab unit tests, all 13 tab e2e tests, and `tsc --noEmit` on the dashboard all pass. * refactor(tabs): drop --tab scoped peek flag; keep t<N> ids and labels After fleshing out `--tab <id|label>` in the previous commits (scoped pre/post-dispatch save/restore, ref preservation, outer-tab-closed edge case, full e2e coverage), the machinery-to-value ratio makes the feature hard to justify. Nixing it now while nothing has shipped. ## Why - Every new daemon feature touching per-tab state has to reason about scoped-dispatch interleaving. `ScopedRestore`, pre/post-dispatch hooks, and the exclusion list add ongoing maintenance tax. - Three separate PRs (#892, #1249, and this one pre-nix) were needed to reach "works correctly." That's a smell. - `tab <id|label>` switch + labels already cover the legible multi-tab workflow case. - `--tab` vs `tab <id>` have opposite lifecycle semantics but look identical, teaching every agent two things where one would do. - "Non-disruptive peek" isn't actually race-free: the daemon does swap active tab during execution, so a concurrent client between pre- and post-dispatch sees the scoped tab as active. - Ref-based interaction with scoped tabs never worked ergonomically — refs are per-tab, so `--tab N click @e1` requires `@e1` to already be on tab N, which means a prior switch, which negates the peek. - Adding a feature back is easy; removing shipped API is hard. If per-tab caching (`HashMap<tab_id, RefMap>`) lands later, `--tab` can be reintroduced essentially for free. That's the right time. ## Removed - `--tab <id|label>` global flag (`cli/src/flags.rs`, `cli/src/main.rs`, all 52 `--help` entries in `cli/src/output.rs`). - `tab` property in `agent-browser.schema.json` and the config-options row in `docs/src/app/configuration/page.mdx`. - `ScopedRestore` struct, pre/post-dispatch save/restore in `execute_command` (`cli/src/native/actions.rs`). - `impl Default for RefMap` in `cli/src/native/element.rs` (only added for `mem::take` in the scoped machinery). - `e2e_tab_global_targeting`, `_snapshot`, `_snapshot_non_contiguous`, `e2e_tab_scoped_command_preserves_outer_tab_state`, `_isolates_refs_from_outer_tab`, `_restores_active_tab`, `_outer_tab_closed_mid_dispatch`. 590 lines. - The "When to use `--tab` vs `tab <id|label>`" sections in README, docs site, and skills reference. ## Kept - Stable tab ids (`t1`, `t2`, `t3`) with bare-integer rejection. - User-assigned labels (`tab new --label docs [url]`), with duplicate rejection and interchangeable use everywhere a tab ref is accepted. - `BrowserManager::{active_tab_id, has_tab_id, resolve_tab_ref, has_label}` accessors (still used by the remaining tab handlers). - `TabRef::parse`, `format_tab_id`, `is_valid_label` and their unit tests. - Dashboard TS fix (`TabInfo.tabId` + `label`). - `e2e_tab_close_with_tab_id_closes_active_tab` (renamed docstring to drop the gone exclusion-list reference). - `e2e_tab_new_with_label_can_be_switched_and_closed` (rewrite of the previous `_and_peeked` test — now exercises only switch and close). - `e2e_tab_switch_rejects_bare_integer` (rewrite targeting the `tab_switch` daemon handler rather than the removed scoped path). net: -900 lines across 12 files. `cargo fmt`, `cargo clippy -D warnings`, all 25 non-ignored tab unit tests, all 6 tab e2e tests, and `tsc --noEmit` on the dashboard all pass. |
||
|
Chris Tate
|
c201623710 |
fix(tabs): correct --tab scoped commands and un-break provider direct-page path (#1249)
* fix(tabs): initialize tab_id on missing PageInfo sites PR #892 added a required `tab_id: u32` field to `PageInfo` but missed two initializer sites, which broke the build on the PR branch. CI never caught this because the external-contributor workflow status was `action_required` and never ran. - `cli/src/native/browser.rs:395` — the `direct_page` branch of `connect_cdp_inner` used by the cloud providers (Browserbase, Browserless, Browser Use, Kernel, AgentCore). Use `assign_tab_id()` to get a fresh id. - `cli/src/native/browser.rs:1580` — a unit test initializer. Use `tab_id: 1` since the test doesn't exercise id assignment. * feat(tabs): restore active tab and clear per-tab state for scoped --tab Follow-up on PR #892's `--tab <id>` flag. The original implementation called `tab_switch_by_id` directly from the pre-dispatch block in `execute_command` but didn't touch the daemon's per-tab state, and never restored the previously-active tab. Two concrete issues this fixes: 1. `state.ref_map`, `state.iframe_sessions`, and `state.active_frame_id` were left intact across the pre-dispatch switch, so `--tab N click @e1` would try to resolve `@e1` against the scoped tab's DOM using a backend-node id from the outer tab. In practice the click handler's role+name fallback hid this as "element not found" errors, but on pages where both tabs have similarly-labelled elements it could click the wrong one. 2. The PR description promised scoped routing would "restore the previous active tab", but the implementation permanently switched. `--tab 3 snapshot` would leave tab 3 as the active tab even after the command returned, surprising subsequent non-scoped commands. This change: - Saves the current tab's stable `tab_id` (not its array index, which would shift if the scoped command closed other tabs) before switching. - Clears per-tab daemon state before the switch so refs/iframes/frame context can't leak between tabs. - After the action runs, restores the original active tab (also via stable id) unless that tab was closed during the scoped command, in which case we leave the scoped tab active. - Adds `BrowserManager::active_tab_id()` and `has_tab_id()` accessors to support the above without exposing the internal `pages` vector. * test(tabs): regression tests for scoped --tab state clearing and restoration Three new `#[ignore]` e2e tests pinning the fixed behavior: - `e2e_tab_scoped_command_clears_state_on_switch` — populates `ref_map` on tab 1, runs a `tabId: 2`-scoped command, asserts `ref_map`, `iframe_sessions`, and `active_frame_id` are all cleared. - `e2e_tab_scoped_command_restores_active_tab` — sets up two tabs, runs a scoped command against the non-active one, asserts a subsequent unscoped command reflects the originally-active tab. - `e2e_tab_scoped_command_handles_outer_tab_closed` — runs a scoped `tab_close` that kills the outer tab itself, asserts no error and the scoped tab becomes active. Also updates two misleading comments in the PR's existing `e2e_tab_global_targeting*` tests to reflect restoration semantics; the assertions themselves were already consistent with restoration. * docs(tabs): document stable tab IDs and --tab scoped-command flag Per AGENTS.md, changes that users or agents would need to know about must land in every doc surface. Fills the gaps PR #892 left: - `README.md` — new `--tab <id>` row in the Options table, rewrite the tab command examples to use `<id>` instead of `<n>`, add a paragraph explaining stable tab IDs and `--tab` peek semantics. - `docs/src/app/commands/page.mdx` — same command-example rewrite plus a new "Stable tab IDs and `--tab`" subsection. - `docs/src/app/configuration/page.mdx` — add `tab` row to the config options table so JSON config users can discover it. - `agent-browser.schema.json` — add `tab` property with description, matching the config schema. - `skills/agent-browser/references/commands.md` — same command-example rewrite plus a short paragraph for agents on when to use `--tab`. |
||
|
Chris Tateandctate
|
6dd53449e8 |
Add auto-dismissal for alert and beforeunload dialogs (#1075)
* Add auto-dismissal for alert and beforeunload dialogs This PR adds automatic handling of JavaScript dialogs to prevent the agent from blocking indefinitely when `alert()` or `beforeunload` dialogs appear on web pages. ## Summary Previously, when a website displayed native browser confirmation dialogs (like alerts or "Are you sure you want to leave?" prompts), agent-browser would hang waiting for manual intervention. This is a common issue since many websites use these dialogs for notifications or navigation warnings. ## Changes Made - **Auto-dismiss functionality**: Added a background task that automatically accepts `alert` and `beforeunload` dialogs while leaving `confirm` and `prompt` dialogs for explicit handling - **New flag**: Added `--no-auto-dialog` flag to disable automatic handling when needed - **Environment variable**: Added `AGENT_BROWSER_NO_AUTO_DIALOG` for configuration - **Documentation**: Updated README and docs with usage examples and configuration details - **Tests**: Added comprehensive test coverage for flag parsing and dialog handling logic ## Implementation Details - Only `alert` (notification-only) and `beforeunload` (navigation warning) dialogs are auto-handled for safety - `confirm` and `prompt` dialogs still require explicit `dialog accept/dismiss` commands to ensure agents make deliberate choices for destructive actions - The feature is enabled by default since these dialog types rarely require user decision-making - Uses Chrome DevTools Protocol's `Page.handleJavaScriptDialog` for reliable dialog dismissal Fixes #1070 * Log dialog type and message before auto-dismissal Without this, auto-dismissed alert/beforeunload dialogs are silently swallowed and the agent has no way to see what the dialog said. Adding an eprintln before the CDP call makes the dismissal visible in stderr for debugging. * Log dialog dismissal errors instead of silently discarding them - Remove premature "accepted" from log message since it fires before the CDP command executes - Replace `let _ =` with `if let Err(e)` to log failures when Page.handleJavaScriptDialog fails - Apply rustfmt to auto-dialog tests --------- Co-authored-by: ctate <366502+ctate@users.noreply.github.com> |
||
|
Chris Tatectategithub-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>Stefan SmiljkoviczhanbaxuyongliangxuyongliangThomas Kosiewski |
f9174513c2 |
dashboard (#1034)
* dashboard * fix: re-apply download behavior on recording context (#1019) * fix: re-apply download behavior on recording context record start creates a new browser context via Target.createBrowserContext. Browser.setDownloadBehavior called at launch only applies to the default context, so downloads in the recording context are silently dropped. Fix: 1. Store download_path on BrowserManager (from LaunchOptions) 2. After creating the recording context, call Browser.setDownloadBehavior with the new browserContextId This ensures downloads work during recording. Fixes #1018 * fix: add download_path to third BrowserManager constructor (auto_connect_cdp) * fix: reap zombie Chrome process and fast-detect crash for auto-restart (#1023) When Chrome crashes (e.g. SIGTRAP from CHECK() assertion), the daemon now: 1. Reaps the zombie immediately via a SIGCHLD handler in the event loop that calls waitpid(-1, WNOHANG) 2. Detects the crash instantly on the next command via a non-blocking try_wait() check (has_process_exited), avoiding the 3-second CDP timeout that is_connection_alive() would incur 3. Auto-relaunches Chrome transparently for the caller Fixes #1017 Co-authored-by: ctate <366502+ctate@users.noreply.github.com> * fix: route keyboard type through text input (#1014) * fix: handle --clear flag in console command (#1015) The console and errors commands parsed --clear from CLI args but the action handlers silently ignored the flag. The handlers did not accept the cmd parameter so they had no way to read the clear field. Changes: - Add clear_console() method to EventTracker in network.rs - Update handle_console to accept cmd, read the clear field, and clear the buffer when --clear is passed (returns {cleared: true}) - Update call site in execute_command to pass cmd Co-authored-by: xuyongliang <yongliang.xyl@alibaba-inc.com> * chore: patch release - ### Bug Fixes - **Re-apply download behavior on r... (#1025) * Add runtime stream enable/disable/status commands (#951) * Add runtime stream management commands * Run rustfmt and satisfy clippy * Fix stream disable cleanup semantics * Format stream disable regression tests * fix: retain radio/checkbox elements in compact snapshot tree (#1008) compact_tree() checked for "[ref=" to identify lines worth keeping, but radio and checkbox elements render as e.g. [checked=false, ref=e1] where the "[" opens before "checked=", not "ref=". Dropping the leading bracket so the check is just "ref=" fixes the match for all elements with refs. Fixes #1006 Co-authored-by: ctate <366502+ctate@users.noreply.github.com> * chore: version packages (#1027) Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> * fixes * dashboard * fixes * remove observe * fmt * fixes * fixes * jotai * fmt * upload dashboard --------- Co-authored-by: Stefan Smiljkovic <stefan@vanila.io> Co-authored-by: ctate <366502+ctate@users.noreply.github.com> Co-authored-by: zhanba <c5e1856@gmail.com> Co-authored-by: xuyongliang <478439790@qq.com> Co-authored-by: xuyongliang <yongliang.xyl@alibaba-inc.com> Co-authored-by: Thomas Kosiewski <thoma471@googlemail.com> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> |
||
|
Chris Tateandctate
|
60f3afcf61 |
Add iframe support for CLI interactions and snapshots (#869)
* Add iframe support for CLI interactions and snapshots This PR adds comprehensive iframe support to the agent browser CLI, allowing users to interact with elements inside iframes seamlessly. ## Problem Users couldn't interact with elements inside iframes via the command line. The existing `frame` command was non-functional as it set `active_frame_id` but no other code read this value. ## Changes Made ### Enhanced Frame Context Tracking - Added `frame_id` field to `RefEntry` to track which frame each element reference belongs to - Updated `RefMap::add` and related methods to accept and store frame context - Modified element resolution functions to use frame context from ref entries ### Improved Frame Command - Fixed the existing `frame` command to actually work by threading `active_frame_id` through snapshot operations - Added support for iframe element references (e.g., `frame @e2`) in addition to CSS selectors - Enhanced frame detection to work with both named frames and iframe elements ### Updated Snapshot Behavior - Modified `take_snapshot` to accept optional frame context parameter - Updated all snapshot call sites to pass appropriate frame context - Maintained backward compatibility while enabling frame-scoped operations ### Element Resolution Updates - Updated `resolve_element_center` and `resolve_element_object_id` to use frame context from ref entries - Modified `find_node_id_by_role_name` to support frame-specific element lookup - Ensured all interaction functions work correctly within iframe contexts ## Implementation Details - Frame context is now properly propagated through the entire element interaction pipeline - The `frame` command can accept both CSS selectors and element references - All existing functionality remains intact while adding iframe capabilities - Added `Iframe` to interactive roles for better element discovery Fixes #863 * docs: add iframe support documentation Document the new iframe capabilities across all documentation surfaces: - Auto-inlining of iframe content in snapshots - Direct interaction with iframe element refs - frame command support for element refs (@e3) - Scoped snapshots via frame switching * fix: pass active frame context to diff snapshots and fix nameless iframe lookup - handle_diff_snapshot now respects active_frame_id instead of always passing None, so diff snapshots work correctly inside iframes - Nameless/id-less iframes now fall back to src URL (or null) instead of the literal string 'frame' which never matched any frame in the tree * fix: resolve iframe frame ID via DOM.describeNode and reduce code duplication - handle_frame: Use DOM.describeNode + contentDocument.frameId to resolve iframe frame IDs directly, fixing failures for nameless iframes that lack name/id/src attributes - element.rs: Deduplicate add() by delegating to add_with_frame() - snapshot.rs: Guard against out-of-bounds insert_str when iframe marker is on the last line without a trailing newline --------- Co-authored-by: ctate <366502+ctate@users.noreply.github.com> |
||
|
Chris Tate
|
315d191606 |
inspect (#736)
* inspect * fixes * improvements * fixes * fixes * improvements * fix null cdp url * fix rust reader loop * improvements * improvements * fixes |
||
|
Chris Tate
|
f2d4089284 |
auth docs (#730)
* add docs * note * format |
||
|
Chris Tate
|
644a4f5b63 |
add scale factor to set viewport for retina screenshots (#691)
* device scale * fix node.js daemon * fix cargo fmt formatting for scale factor code * fixes |
||
|
Andrew ImmandChris Tate
|
59fa36b6e2 |
feat: Enable capture of profiling data (#290)
* feat: Enable capture of profiling data Adding a new set of commands: ``` agent-browser profiler start agent-browser profiler stop trace.json ``` With this, agents can start a profiling trace, perform a set of actions, and then extract the profiling data for analysis. **Note:** I was originally going to call it `agent-browser profile` but I realized that might cause confusion with the `--profile` flag CDP supports a couple commands for starting/stopping a trace. When a trace is running, it emits events that need to be picked up. We store these locally in the daemon until the trace is completed. When the final event is received, we dump all of them into an output file. That file can be loaded directly into chrome devtools or another analysis tool to visualize what happened during the agentic run. Added some basic rust tests for parsing the commands (since they have some optional / required args) TS daemon adds ~6 tests to make sure the profiling lifecycle (including saving the output file) works as intended * add docs * fixes * fixes --------- Co-authored-by: Chris Tate <chris@ctate.dev> |
||
|
Chris Tate
|
76df589aea | update docs (#493) | ||
|
Chris Tate
|
0dc36f2cff |
Add --stdin flag for eval command (#348)
Adds --stdin flag to read JavaScript from stdin, enabling heredoc usage for multiline scripts without shell escaping issues. |
||
|
Chris Tate
|
e52aa49706 |
Add skill-creator and improve agent-browser skill (#341)
* add skills-creator * update skill * better docs * minor fixes |
||
|
Yonatan
|
f74924cd0c |
feat(skills): Add hierarchical structure with references and templates (#157)
* feat(skills): Add hierarchical structure with references and templates Adds modular documentation and executable templates to the agent-browser skill for better AI agent consumption and progressive disclosure. ## Added ### References (deep-dive documentation) - `references/snapshot-refs.md` - Ref lifecycle, invalidation, troubleshooting - `references/session-management.md` - Parallel sessions, state persistence - `references/authentication.md` - Login flows, OAuth, 2FA patterns - `references/video-recording.md` - Recording for debugging/docs - `references/proxy-support.md` - Proxy configuration, geo-testing ### Templates (ready-to-use workflows) - `templates/form-automation.sh` - Form filling with validation - `templates/authenticated-session.sh` - Login once, reuse state - `templates/capture-workflow.sh` - Content extraction with screenshots ## Modified - `SKILL.md` - Added reference tables linking to new documentation ## Benefits - Progressive disclosure: Load overview first, deep dives on demand - Reduced context: Smaller chunks for better LLM token efficiency - Ready workflows: Copy-paste templates for common patterns * fix(templates): Make authenticated-session.sh runnable out-of-box Addresses review feedback: login actions were commented but verification wasn't, causing script to fail when run as-is. New approach: - DISCOVERY MODE runs first (shows form structure) - LOGIN FLOW section is fully commented as a unit - User runs once to see refs, then customizes ┌─────────────────────────────────────────────────────────────┐ │ LOGIN FORM STRUCTURE │ ├─────────────────────────────────────────────────────────────┤ │ @e1 [input type="email"] │ │ @e2 [input type="password"] │ │ @e3 [button] "Sign In" │ └─────────────────────────────────────────────────────────────┘ |