fix(relay): DOM-dispatch hover/dblclick/drag; deeper iframe snapshot; key-events typing (#37)

Follow-up to #36 — make the whole interaction surface reach cross-origin OOPIFs
and stop coordinate events drifting onto the user's foreground tab over the relay.

- hover/dblclick/drag now DOM-dispatch over the relay or into an iframe (like
  click already did): a coordinate Input event isn't confined to the target tab
  on a busy real Chrome and can't map an OOPIF element's box to a top-viewport
  point. drag does an HTML5 DnD in the element's frame; cross-frame drag errors
  loudly instead of drifting.
- snapshot recurses iframes to MAX_IFRAME_DEPTH (3) instead of one level, so refs
  inside nested payment/checkout widgets get a frame_id and resolve into the
  right frame.
- relay tab adoption merges several Target.getTargets snapshots — a single flaky
  relay snapshot was dropping live tabs (a driven tab vanished after restart).
- `type --key-events` (alias --keys) sends real per-character keyDown/keyUp
  instead of Input.insertText, so autocomplete/combobox widgets that ignore the
  insertText input event fire (Google address postal lookup; commits Angular
  reactive forms so Save enables).
- SKILL: hard "snapshot-first, never default to screenshot+coordinates" rule;
  snapshot -i pierces cross-origin iframes since v1.5.12; cross-origin iframe
  driving guidance (#37).
This commit is contained in:
leeguooooo
2026-06-17 00:58:12 +09:00
parent 70ab38d35f
commit 9f24e66033
10 changed files with 420 additions and 42 deletions
+7
View File
@@ -1464,6 +1464,12 @@ Usage: chrome-use type <selector> <text>
Types text into the specified element character by character.
Unlike fill, this does not clear existing content first.
Options:
--key-events Send real per-character keyDown/keyUp instead of
(alias --keys) Input.insertText. Use for autocomplete / combobox fields
that only react to key events — e.g. a postal-code box
that auto-fills city/prefecture, or Google Places.
Global Options:
--json Output as JSON
--session <name> Use specific session
@@ -1471,6 +1477,7 @@ Global Options:
Examples:
chrome-use type "#search" "hello"
chrome-use type @e2 "additional text"
chrome-use type @e5 "201-0001" --key-events # trigger the address autocomplete
See Also:
For typing into contenteditable editors (Lexical, ProseMirror, etc.)