Follow-up to #36 — make the whole interaction surface reach cross-origin OOPIFs
and stop coordinate events drifting onto the user's foreground tab over the relay.
- hover/dblclick/drag now DOM-dispatch over the relay or into an iframe (like
click already did): a coordinate Input event isn't confined to the target tab
on a busy real Chrome and can't map an OOPIF element's box to a top-viewport
point. drag does an HTML5 DnD in the element's frame; cross-frame drag errors
loudly instead of drifting.
- snapshot recurses iframes to MAX_IFRAME_DEPTH (3) instead of one level, so refs
inside nested payment/checkout widgets get a frame_id and resolve into the
right frame.
- relay tab adoption merges several Target.getTargets snapshots — a single flaky
relay snapshot was dropping live tabs (a driven tab vanished after restart).
- `type --key-events` (alias --keys) sends real per-character keyDown/keyUp
instead of Input.insertText, so autocomplete/combobox widgets that ignore the
insertText input event fire (Google address postal lookup; commits Angular
reactive forms so Save enables).
- SKILL: hard "snapshot-first, never default to screenshot+coordinates" rule;
snapshot -i pierces cross-origin iframes since v1.5.12; cross-origin iframe
driving guidance (#37).
Release binaries / Build macOS ARM64 (push) Has been cancelled
Release binaries / Build macOS x64 (push) Has been cancelled
Release binaries / Build Linux ARM64 (push) Has been cancelled
Release binaries / Build Linux musl ARM64 (push) Has been cancelled
Release binaries / Build Linux musl x64 (push) Has been cancelled
Release binaries / Build Linux x64 (push) Has been cancelled
Release binaries / Build Windows x64 (push) Has been cancelled
Release binaries / Attach binaries to GitHub Release (push) Has been cancelled
Standalone product rename across the whole repo (issue: project identity):
- Binary/package/repo/skill/docs: agent-browser[-stealth] → chrome-use
(single binary name `chrome-use`; old aliases agent-browser/abs dropped).
- Version: 0.27.0-fork.51 → 1.0.0 (drop the upstream-fork counter).
- Native-messaging host: com.agent_browser.connect → com.leeguoo.chrome_use
(CLI + ab-connect extension in lockstep — this is a breaking handshake change,
extension bumped 0.4.2 → 0.5.0, needs a Web Store republish).
- Config dir: ~/.agent-browser → ~/.chrome-use.
- README/zh: reframed from "stealth fork of agent-browser" to a standalone
product with a small `originally based on vercel-labs/agent-browser` credit.
- Kept AGENT_BROWSER_* env vars working (63 vars across the codebase; renaming
them would break every existing script/skill for no user-facing gain).
Build green, 802 unit tests pass, fmt + clippy clean. Upstream attribution to
vercel-labs/agent-browser preserved.
- snapshot -c (compact) now always keeps lines with an interactive ARIA role
(button/link/textbox/combobox/option/…), not only `ref=`/`": "` lines — so a
clickable control can't vanish from compact output and leave the agent clicking
an empty ref (issue #2 P1). Additive: only ever keeps more. compact tests green.
- stale-ref error now leads with "take a fresh snapshot" and points to the `eval`
fallback for ref-churning SPAs, and demotes AGENT_BROWSER_VERIFY_REF=0 to a
flagged last resort instead of presenting it as the fix (issue #3 P1).
The fork's CI had never been green. Pre-existing failures:
- version-sync: check-version-sync.js read packages/dashboard/package.json,
which doesn't exist in this fork (workspace is just "."). Drop the dashboard
comparison; check package.json vs cli/Cargo.toml only.
- Dashboard job: `pnpm install --filter dashboard` for a non-existent package.
Remove the job.
- Format check: repo was never `cargo fmt`-clean. Ran cargo fmt (mechanical).
- Clippy -D warnings (newly enforced on Rust 1.94 stable): manual_contains in
commands.rs (.iter().any()->.contains()), question_mark in element.rs
(if-let-Err -> ?), result_large_err on the tungstenite handshake callback in
connect.rs (allow — the Result type is fixed by the accept_hdr_async contract).
- rust-cross: lightpanda::waits_for_ready_without_logs spawns a real process +
binds a socket with timing assumptions; flaky in CI. Marked #[ignore].
Also: skill docs note fork.30's relay-preferred auto-connect (plain
`agent-browser open` is dialog-free once the ab-connect extension is loaded) and
the extension's new "agent-browser-stealth" display name.
- snapshot: make collect_fingerprints private (TreeNode is private, so a
pub(super) fn leaked a more-private type)
- adaptive: if-let instead of single-arm match in attr_score
- stealth: move timezone test module to end of file (items-after-test-module)
No behavior change. Pre-release cleanup.
Borrow Scrapling's adaptive element finding, adapted to this project's
in-session AX-ref model. When a saved @ref's node is gone (or its identity
no longer matches) and the role/name/nth re-query also fails, score the
current page's candidate elements against an AX fingerprint captured at
snapshot time and relocate to the best match.
- New `adaptive` module: pure, browser-free scoring (role, accessible name
via Levenshtein, AX properties, ancestor-role LCS, parent/sibling) plus
pick_best with a high absolute threshold (0.70) AND a clear margin (0.15)
over the runner-up — so ambiguous twins are refused rather than mis-clicked,
matching the existing "fail loudly over wrong click" posture.
- Fingerprint captured during the existing AX-tree snapshot walk — no extra
CDP round-trips. TreeNode is AX-only (no DOM tag/attrs), so we use AX role
as the type and a few discriminating AX properties (value/url/level/checked);
DOM id/class would have cost an N×describeNode storm per snapshot.
- Wired into both resolve_element_center and resolve_element_object_id: on a
verify-identity mismatch or a stale-node fallback miss, relocation is tried
before erroring. A confident match overrides the identity guard; otherwise
the original error is surfaced. Opt out with AGENT_BROWSER_ADAPTIVE_REF=0.
README documents the new tuning knobs. Adds 9 unit tests; full suite 760 passed.
* fix: promote hidden radio/checkbox inputs in snapshot refs (#1024)
When a <label> wraps a display:none <input type="radio">, Chrome
excludes the input from the accessibility tree entirely. The label
appears as role="LabelText" with an empty name, making it impossible
for AI agents to identify radio buttons via data.refs.
Detect hidden radio/checkbox inputs during cursor-interactive scanning
and promote their parent LabelText/generic nodes to the correct role
with proper name and checked state.
- Add HiddenInputKind enum to validate input types at parse boundary
- Extend cursor-interactive JS to detect hidden inputs inside elements
- Extract promote_hidden_inputs() for testable role promotion logic
- Add unit tests for promotion, name preservation, and skip conditions
* style: apply cargo fmt
---------
Co-authored-by: hyunjinee <leehj0110@kakao.com>
compact_tree() checked for "[ref=" to identify lines worth keeping, but
radio and checkbox elements render as e.g. [checked=false, ref=e1] where
the "[" opens before "checked=", not "ref=". Dropping the leading bracket
so the check is just "ref=" fixes the match for all elements with refs.
Fixes#1006
Co-authored-by: ctate <366502+ctate@users.noreply.github.com>
* fix: add ref for cursor-interactive content roles
* fix: format
* feat: always include cursor-interactive elements in snapshot, -C is deprecated
* feat: process StaticText aggregation and deduplication
* update test
* clean up
* fix: escape text of elements in snapshot
* fix: redundant slicing
* fix: cargo fmt
* feat: deduplicate redundant StaticText
---------
Co-authored-by: 羲洋 <lipengyang.lpy@alibaba-inc.com>
* Add iframe support for CLI interactions and snapshots
This PR adds comprehensive iframe support to the agent browser CLI, allowing users to interact with elements inside iframes seamlessly.
## Problem
Users couldn't interact with elements inside iframes via the command line. The existing `frame` command was non-functional as it set `active_frame_id` but no other code read this value.
## Changes Made
### Enhanced Frame Context Tracking
- Added `frame_id` field to `RefEntry` to track which frame each element reference belongs to
- Updated `RefMap::add` and related methods to accept and store frame context
- Modified element resolution functions to use frame context from ref entries
### Improved Frame Command
- Fixed the existing `frame` command to actually work by threading `active_frame_id` through snapshot operations
- Added support for iframe element references (e.g., `frame @e2`) in addition to CSS selectors
- Enhanced frame detection to work with both named frames and iframe elements
### Updated Snapshot Behavior
- Modified `take_snapshot` to accept optional frame context parameter
- Updated all snapshot call sites to pass appropriate frame context
- Maintained backward compatibility while enabling frame-scoped operations
### Element Resolution Updates
- Updated `resolve_element_center` and `resolve_element_object_id` to use frame context from ref entries
- Modified `find_node_id_by_role_name` to support frame-specific element lookup
- Ensured all interaction functions work correctly within iframe contexts
## Implementation Details
- Frame context is now properly propagated through the entire element interaction pipeline
- The `frame` command can accept both CSS selectors and element references
- All existing functionality remains intact while adding iframe capabilities
- Added `Iframe` to interactive roles for better element discovery
Fixes#863
* docs: add iframe support documentation
Document the new iframe capabilities across all documentation surfaces:
- Auto-inlining of iframe content in snapshots
- Direct interaction with iframe element refs
- frame command support for element refs (@e3)
- Scoped snapshots via frame switching
* fix: pass active frame context to diff snapshots and fix nameless iframe lookup
- handle_diff_snapshot now respects active_frame_id instead of always
passing None, so diff snapshots work correctly inside iframes
- Nameless/id-less iframes now fall back to src URL (or null) instead of
the literal string 'frame' which never matched any frame in the tree
* fix: resolve iframe frame ID via DOM.describeNode and reduce code duplication
- handle_frame: Use DOM.describeNode + contentDocument.frameId to resolve
iframe frame IDs directly, fixing failures for nameless iframes that
lack name/id/src attributes
- element.rs: Deduplicate add() by delegating to add_with_frame()
- snapshot.rs: Guard against out-of-bounds insert_str when iframe marker
is on the last line without a trailing newline
---------
Co-authored-by: ctate <366502+ctate@users.noreply.github.com>
Generic elements (e.g. <div>) with cursor:pointer/onclick have empty ARIA
names because their text lives in StaticText children, which get filtered
in interactive mode. Fall back to the JS-collected textContent so the text
appears on the rendered tree line.
Fixes e2e_snapshot_cursor_many_elements CI failure from #855.
* feat: embed cursor-interactive elements into snapshot tree
* optimize format
* fix: address review feedback for e2e_snapshot_cursor_interactive unitest
---------
Co-authored-by: 羲洋 <lipengyang.lpy@alibaba-inc.com>
* fix: resolve snapshot -C and screenshot --annotate hang over WSS (#841)
Root cause: sequential CDP round-trips per element in
find_cursor_interactive_elements() and collect_annotations() caused
timeouts over high-latency WSS connections (~200ms × 200+ elements
exceeds the 30s CDP timeout).
Fix:
- snapshot -C: Replace per-element CDP calls with a single JS eval
that detects cursor:pointer/onclick/tabindex elements in-browser,
then batch-resolve via DOM.querySelectorAll + concurrent
DOM.describeNode calls using join_all
- screenshot --annotate: Replace sequential DOM.resolveNode +
getRect calls with concurrent join_all, matching v0.19.0's
Promise.all() pattern
Behavioral parity with v0.19.0 (Node.js/Playwright):
- cursor:pointer detection via getComputedStyle
- Inherited cursor:pointer dedup (skip children of pointer parents)
- interactiveTags and interactive ARIA roles exclusion
- Role differentiation: clickable vs focusable
- Text dedup against ARIA tree ref names and quoted strings
- Edge case: -i -C shows cursor elements even when ARIA tree is empty
Tests:
- 5 unit tests for build_dedup_set() helper
- 3 e2e regression tests: cursor-interactive detection, annotation
scaling to 50 elements, cursor scaling to 100 elements
* fix: add hidden/aria-hidden filtering, contentEditable support, and cleanup robustness
- Restore hidden/aria-hidden element filtering in cursor-interactive JS
(was present in old code, dropped during rewrite)
- Add contentEditable detection with 'editable' role and hint
- Replace fire-and-forget cleanup with warning on failure
- Simplify build_dedup_set to use ref_map only (eliminates fragile
tree-text quote parsing; ref_map already has all ref-bearing names)
---------
Co-authored-by: ctate <366502+ctate@users.noreply.github.com>
* fix: snapshot --selector scopes to the matched element subtree
The native Rust daemon accepted the --selector flag but never used it —
the full accessibility tree was always returned regardless of the
selector. This restores the 0.19.0 behaviour where snapshot --selector
returns only the subtree rooted at the matched CSS selector.
The implementation resolves the selector via Runtime.evaluate, fetches
the full DOM subtree with DOM.describeNode(depth: -1) to collect all
descendant backendNodeIds, then filters the AX tree to render only the
nodes whose backendDOMNodeId falls within that set. This correctly
handles elements like <body> that don't map to a direct AX node.
Also fixes handle_snapshot reading "depth" instead of "maxDepth" from
the command JSON, which caused --depth to be silently ignored.
Fixes#822
* style: run cargo fmt on snapshot.rs
---------
Co-authored-by: ctate <366502+ctate@users.noreply.github.com>
* Fix clippy warnings across CLI codebase
Fixes#653
* Fix remaining items_after_test_module clippy warnings
Move functions defined after `mod tests` blocks to before the test
modules in recording.rs and webdriver/client.rs.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: ctate <366502+ctate@users.noreply.github.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
leeguooooo