The `press` command was not parsing modifier+key chords (e.g.
`Control+a`, `Shift+Enter`). It sent the raw string as a single key
name, so CDP never applied the modifier — and the `text` field caused
the literal character to be inserted instead of triggering the shortcut.
Two changes:
1. `actions.rs` — add `parse_key_chord()` to split inputs like
`Control+Shift+a` into the base key (`a`) and a CDP modifier
bitmask (Alt=1, Ctrl=2, Meta=4, Shift=8), then call
`press_key_with_modifiers` instead of `press_key`.
2. `interaction.rs` — suppress the `text` field in `keyDown`/`keyUp`
events when Control or Meta modifiers are active, so the browser
treats them as command chords rather than text input.
Includes unit tests for the chord parser covering plain keys, single
modifiers, multi-modifier combos, modifier aliases, and edge cases.
Co-authored-by: ctate <366502+ctate@users.noreply.github.com>
This PR fixes an issue where pressing the Enter key via the `press Enter` command had no effect on web pages, even though the command executed successfully without errors.
## Problem
The Chrome DevTools Protocol `Input.dispatchKeyEvent` requires a `text` field in `keyDown` events for certain keys to trigger their default browser actions. Without this field, keys like Enter don't actually submit forms or perform their expected actions.
## Changes
- Added new `key_text()` function that returns the appropriate text value for special keys:
- Enter returns `"\r"` (carriage return)
- Tab returns `"\t"` (tab character)
- Space returns `" "` (space character)
- Single printable characters return themselves
- Non-printable keys (Escape, Arrow keys, etc.) return `None`
- Modified `press_key_with_modifiers()` to populate both `text` and `unmodified_text` fields in the CDP keyDown event
- Added comprehensive unit tests for the new functionality
## Implementation Details
The fix ensures that when pressing Enter on a filled search box, the form actually submits as expected, matching the behavior of manually pressing the Enter key.
Fixes#966
* Add iframe support for CLI interactions and snapshots
This PR adds comprehensive iframe support to the agent browser CLI, allowing users to interact with elements inside iframes seamlessly.
## Problem
Users couldn't interact with elements inside iframes via the command line. The existing `frame` command was non-functional as it set `active_frame_id` but no other code read this value.
## Changes Made
### Enhanced Frame Context Tracking
- Added `frame_id` field to `RefEntry` to track which frame each element reference belongs to
- Updated `RefMap::add` and related methods to accept and store frame context
- Modified element resolution functions to use frame context from ref entries
### Improved Frame Command
- Fixed the existing `frame` command to actually work by threading `active_frame_id` through snapshot operations
- Added support for iframe element references (e.g., `frame @e2`) in addition to CSS selectors
- Enhanced frame detection to work with both named frames and iframe elements
### Updated Snapshot Behavior
- Modified `take_snapshot` to accept optional frame context parameter
- Updated all snapshot call sites to pass appropriate frame context
- Maintained backward compatibility while enabling frame-scoped operations
### Element Resolution Updates
- Updated `resolve_element_center` and `resolve_element_object_id` to use frame context from ref entries
- Modified `find_node_id_by_role_name` to support frame-specific element lookup
- Ensured all interaction functions work correctly within iframe contexts
## Implementation Details
- Frame context is now properly propagated through the entire element interaction pipeline
- The `frame` command can accept both CSS selectors and element references
- All existing functionality remains intact while adding iframe capabilities
- Added `Iframe` to interactive roles for better element discovery
Fixes#863
* docs: add iframe support documentation
Document the new iframe capabilities across all documentation surfaces:
- Auto-inlining of iframe content in snapshots
- Direct interaction with iframe element refs
- frame command support for element refs (@e3)
- Scoped snapshots via frame switching
* fix: pass active frame context to diff snapshots and fix nameless iframe lookup
- handle_diff_snapshot now respects active_frame_id instead of always
passing None, so diff snapshots work correctly inside iframes
- Nameless/id-less iframes now fall back to src URL (or null) instead of
the literal string 'frame' which never matched any frame in the tree
* fix: resolve iframe frame ID via DOM.describeNode and reduce code duplication
- handle_frame: Use DOM.describeNode + contentDocument.frameId to resolve
iframe frame IDs directly, fixing failures for nameless iframes that
lack name/id/src attributes
- element.rs: Deduplicate add() by delegating to add_with_frame()
- snapshot.rs: Guard against out-of-bounds insert_str when iframe marker
is on the last line without a trailing newline
---------
Co-authored-by: ctate <366502+ctate@users.noreply.github.com>
* fix: use correct Windows virtual-key codes for punctuation in type command
The `type` command was dropping punctuation characters like `.`, `'`, and
`#` because `char_to_key_info()` used raw ASCII codes as the
`windowsVirtualKeyCode` in CDP `Input.dispatchKeyEvent` calls. For
punctuation the ASCII value collides with unrelated VK codes — most
critically '.' (ASCII 46) equals VK_DELETE (0x2E), causing Chrome to
interpret periods as Delete key presses.
Changes:
- Add `punctuation_key_info()` with correct VK_OEM_* codes matching
Playwright's USKeyboardLayout (e.g. Period=190, Slash=191, Semicolon=186)
- Fall back to `Input.insertText` for characters without a US keyboard
mapping (emoji, CJK, etc.), matching Playwright's `keyboard.type()`
- Update e2e test to use `type` instead of `fill` workaround for email
- Add unit tests verifying VK code parity with Playwright's layout
Fixes#833
* style: fix rustfmt formatting for InsertTextParams
---------
Co-authored-by: ctate <366502+ctate@users.noreply.github.com>
The v0.20.0 migration from Playwright to the native Rust daemon introduced
two regressions in checkbox/radio handling:
1. `is_element_checked` only read `this.checked`, which is undefined on
non-input elements. Material Design and ARIA controls use wrapper divs
with `role="checkbox"` and `aria-checked`, or hide the native input
off-screen inside a label. The function now mirrors Playwright's
`getChecked()` with follow-label retargeting: native `.checked`,
`aria-checked` for ARIA roles, `label.control` traversal, and nested
input lookup.
2. `check`/`uncheck` accepted the coordinate-based CDP click result
without verifying the state actually changed. When the AX tree's
`backendDOMNodeId` points to a hidden off-screen input (common in
Material Design), `Input.dispatchMouseEvent` hits nothing. The actions
now re-check state after clicking and fall back to a JS `.click()` on
the resolved input — matching Playwright's `_setChecked` verify step.
Adds e2e regression test covering Material Design (hidden input + ripple
overlay), ARIA-only, and native checkbox patterns.
Co-authored-by: ctate <366502+ctate@users.noreply.github.com>
* feat: add screenshot output config, clipboard CLI commands, and fix wait --text native path
## Summary
- Add `--screenshot-dir`, `--screenshot-quality`, and `--screenshot-format` CLI flags (with corresponding `AGENT_BROWSER_SCREENSHOT_DIR`, `AGENT_BROWSER_SCREENSHOT_QUALITY`, `AGENT_BROWSER_SCREENSHOT_FORMAT` env vars) so users can configure where and how screenshots are saved without specifying a full path every time
- Add `clipboard read`, `clipboard write <text>`, `clipboard copy`, and `clipboard paste` CLI commands, exposing the existing protocol-level clipboard handlers that were previously only accessible via JSON-RPC
- Fix `wait --text` in native mode: the CLI was emitting `selector: "text=..."` (a Playwright-style locator) which native's `querySelector` can't handle. Now emits a `text` field that correctly hits the native `wait_for_text` polling path
- Add native clipboard `copy` and `paste` support via CDP `Input.dispatchKeyEvent`, and a `write` operation to the Node.js handler
* fix: resolve CI failures in Rust formatting and TypeScript typecheck
Use string-based page.evaluate for clipboard writeText to avoid
referencing `navigator` in Node.js compilation context. Run cargo fmt
to fix formatting in commands.rs and screenshot.rs.
* fix: clipboard write captures full multi-word text
Use rest[1..].join(" ") instead of rest.get(1) so unquoted multi-word
input like `clipboard write hello world` sends the full string rather
than silently dropping everything after the first word.
* improvements
* fixes
* improvements
* improvements
zhanba