* fix: rewrite getByRole to use CDP accessibility tree instead of CSS selectors
The old `handle_getbyrole` generated `querySelectorAll('[role="link"], link')`
which matched `<link>` stylesheet elements instead of `<a>` anchor tags.
This happened because ARIA role names were used directly as CSS tag selectors,
and several roles differ from their HTML element names (e.g. link → a,
heading → h1-h6, textbox → input/textarea).
The fix replaces the JS-based DOM query with the CDP `Accessibility.getFullAXTree`
API, where the browser engine correctly computes implicit ARIA roles per the
WAI-ARIA / HTML-AAM spec. This is the same approach already used by `snapshot.rs`
and `element.rs` in this codebase.
Changes:
- Rewrite `handle_getbyrole` to query the browser's accessibility tree via CDP
- Add `find_ax_node_by_role` helper for AX tree traversal with role/name/exact matching
- Use `DOM.resolveNode` + `Runtime.callFunctionOn` to bridge AX node → DOM marker
- Add iframe support via `resolve_ax_session` (missing in old implementation)
- Fix cleanup to use correct CDP session (old code used default session, breaking iframe cleanup)
- Export `extract_ax_string` as `pub(super)` for reuse
- Add 4 regression tests for `find_ax_node_by_role`
Fixes#1123
* style: apply cargo fmt
* chore: remove redundant comments
* refactor: replace marker attribute with temporary ref for element resolution
Eliminates 3 CDP round-trips (DOM.resolveNode, Runtime.callFunctionOn,
Runtime.evaluate cleanup) by registering a temporary ref in the ref_map.
execute_subaction resolves the element via backendNodeId directly.
No more DOM pollution with marker attributes.
* fix: ref counter collision, ref_map leak, and stale fallback name
- Increment next_ref_num after inserting temp ref to prevent id collision
- Remove temp ref after execute_subaction to prevent unbounded ref_map growth
- Return actual AX name from find_ax_node_by_role for accurate fallback resolution
- Add RefMap::remove method
---------
Co-authored-by: hyunjinee <leehj0110@kakao.com>
* fix: support xpath= selector prefix in element resolution
Resolves#907. When a selector starts with "xpath=", use
document.evaluate() instead of document.querySelector() so that
XPath expressions like "xpath=//button" work correctly.
* test: replace overlapping test with edge case tests
Replace test_build_selector_js_xpath_strips_prefix (which overlapped
with the xpath test) with two edge case tests: empty xpath and
selector starting with "xpath" without "=" delimiter.
* fix: support xpath= selector in resolve_element_object_id
Apply the same xpath= handling to resolve_element_object_id, which is
used by type, fill, focus, hover, check, select, screenshot, drag,
and all other selector-based commands beyond basic click.
* refactor: extract build_find_element_js to deduplicate xpath/css logic
The xpath= vs querySelector branching was duplicated in both
build_selector_js and resolve_element_object_id. Extract the shared
logic into build_find_element_js and reuse it in both places.
* fix: support xpath= selector in get_element_count
Use ORDERED_NODE_SNAPSHOT_TYPE with snapshotLength for XPath counting,
matching the querySelectorAll().length behavior for CSS selectors.
* refactor: rename find to find_expr for clarity
* refactor: extract build_count_elements_js and add regression tests
Extract element counting JS generation into build_count_elements_js
helper (matching the pattern of build_find_element_js) and add tests
for both CSS and XPath counting paths.
---------
Co-authored-by: hyunjinee <leehj0110@kakao.com>
* Add iframe support for CLI interactions and snapshots
This PR adds comprehensive iframe support to the agent browser CLI, allowing users to interact with elements inside iframes seamlessly.
## Problem
Users couldn't interact with elements inside iframes via the command line. The existing `frame` command was non-functional as it set `active_frame_id` but no other code read this value.
## Changes Made
### Enhanced Frame Context Tracking
- Added `frame_id` field to `RefEntry` to track which frame each element reference belongs to
- Updated `RefMap::add` and related methods to accept and store frame context
- Modified element resolution functions to use frame context from ref entries
### Improved Frame Command
- Fixed the existing `frame` command to actually work by threading `active_frame_id` through snapshot operations
- Added support for iframe element references (e.g., `frame @e2`) in addition to CSS selectors
- Enhanced frame detection to work with both named frames and iframe elements
### Updated Snapshot Behavior
- Modified `take_snapshot` to accept optional frame context parameter
- Updated all snapshot call sites to pass appropriate frame context
- Maintained backward compatibility while enabling frame-scoped operations
### Element Resolution Updates
- Updated `resolve_element_center` and `resolve_element_object_id` to use frame context from ref entries
- Modified `find_node_id_by_role_name` to support frame-specific element lookup
- Ensured all interaction functions work correctly within iframe contexts
## Implementation Details
- Frame context is now properly propagated through the entire element interaction pipeline
- The `frame` command can accept both CSS selectors and element references
- All existing functionality remains intact while adding iframe capabilities
- Added `Iframe` to interactive roles for better element discovery
Fixes#863
* docs: add iframe support documentation
Document the new iframe capabilities across all documentation surfaces:
- Auto-inlining of iframe content in snapshots
- Direct interaction with iframe element refs
- frame command support for element refs (@e3)
- Scoped snapshots via frame switching
* fix: pass active frame context to diff snapshots and fix nameless iframe lookup
- handle_diff_snapshot now respects active_frame_id instead of always
passing None, so diff snapshots work correctly inside iframes
- Nameless/id-less iframes now fall back to src URL (or null) instead of
the literal string 'frame' which never matched any frame in the tree
* fix: resolve iframe frame ID via DOM.describeNode and reduce code duplication
- handle_frame: Use DOM.describeNode + contentDocument.frameId to resolve
iframe frame IDs directly, fixing failures for nameless iframes that
lack name/id/src attributes
- element.rs: Deduplicate add() by delegating to add_with_frame()
- snapshot.rs: Guard against out-of-bounds insert_str when iframe marker
is on the last line without a trailing newline
---------
Co-authored-by: ctate <366502+ctate@users.noreply.github.com>
The v0.20.0 migration from Playwright to the native Rust daemon introduced
two regressions in checkbox/radio handling:
1. `is_element_checked` only read `this.checked`, which is undefined on
non-input elements. Material Design and ARIA controls use wrapper divs
with `role="checkbox"` and `aria-checked`, or hide the native input
off-screen inside a label. The function now mirrors Playwright's
`getChecked()` with follow-label retargeting: native `.checked`,
`aria-checked` for ARIA roles, `label.control` traversal, and nested
input lookup.
2. `check`/`uncheck` accepted the coordinate-based CDP click result
without verifying the state actually changed. When the AX tree's
`backendDOMNodeId` points to a hidden off-screen input (common in
Material Design), `Input.dispatchMouseEvent` hits nothing. The actions
now re-check state after clicking and fall back to a JS `.click()` on
the resolved input — matching Playwright's `_setChecked` verify step.
Adds e2e regression test covering Material Design (hidden input + ripple
overlay), ARIA-only, and native checkbox patterns.
Co-authored-by: ctate <366502+ctate@users.noreply.github.com>
* fix: gracefully fall back to role/name lookup when backend_node_id is stale
When the DOM changes between snapshot and click (common with SPAs and
dynamic UIs), the stored backend_node_id becomes invalid. Previously,
DOM.getBoxModel and DOM.resolveNode failures propagated as hard errors,
bypassing the role/name fallback path entirely. Now these failures are
caught and the code falls through to a JS-based element lookup.
Also adds resolve_object_id_by_role_name so that resolve_element_object_id
has a fallback for ref-based lookups (previously it had none), and
improves the role matching JS to correctly map implicit ARIA roles
(e.g. <input type="submit"> → "button", <a href> → "link").
Closes#805
* test: add e2e regression test for stale ref click fallback (#805)
Verifies that clicking a ref whose backend_node_id has become stale
(because the DOM was replaced by JavaScript) falls back to role/name
lookup instead of failing with "Could not compute box model".
* fix: use accessibility tree for stale ref fallback instead of JS heuristic
Replace the hand-rolled JS role/name matching (getImplicitRole,
getAccessibleName) with a re-query of Accessibility.getFullAXTree —
the same data source that built the ref map during snapshot. This
guarantees role/name matching is identical to what was stored,
preventing silent wrong-element clicks from name computation
divergence (e.g. aria-labelledby, <label for>, alt text).
Matches v0.19.0 (Playwright) behavior where getByRole always
re-queried the live accessibility tree.
---------
Co-authored-by: ctate <366502+ctate@users.noreply.github.com>
