fix(relay): don't hijack a user tab on open; surface keydown/keyup + canvas hint
Dogfooding a canvas game over the extension relay surfaced three issues: 1. (serious) A fresh relay session's first `open` navigated one of the USER's existing tabs instead of opening its own — in testing it replaced a half-filled form with the target site. On connect the daemon passively attaches to the user's tabs and pinned one as active; navigate() then drove it. Now: on the relay (agent_group set), if the active tab isn't one this session created, navigate() opens its own tab in the session's group first. Off the relay (a browser we launched) reusing the active tab stays correct. Pure helper active_index_is_owned() + regression tests. 2. (discoverability) `keydown <key>` / `keyup <key>` (hold-to-move, essential for games/shortcuts) already existed as commands+daemon handlers but were absent from --help and the skill, so they were undiscoverable. Documented in --help, the core skill, and the canvas-app hint. 3. (UX) Canvas/WebGL pages expose almost no a11y tree, so `snapshot` is empty and agents get stuck hunting refs. snapshot now detects a viewport-dominating canvas with a sparse tree and prints a hint pointing at the screenshot + coordinate-click + keydown/keyup path. Verified live over the relay: `open` now lands the game in its own new tab with the user's tabs (incl. the Rakuten recovery form) untouched; the canvas hint fires on the game page; `close` cleans up only the session's own tab.
This commit is contained in:
@@ -226,8 +226,11 @@ chrome-use hover @e1 # hover
|
||||
chrome-use focus @e1 # focus (useful before keyboard input)
|
||||
chrome-use fill @e2 "hello" # clear then type
|
||||
chrome-use type @e2 " world" # type without clearing
|
||||
chrome-use press Enter # press a key at current focus
|
||||
chrome-use press Enter # press a key at current focus (down+up)
|
||||
chrome-use press Control+a # key combination
|
||||
chrome-use keydown d # HOLD a key down (no auto-release)
|
||||
chrome-use keyup d # release it — pair them to hold-to-move
|
||||
# in a game: `keydown d; sleep; keyup d`
|
||||
chrome-use check @e3 # check checkbox
|
||||
chrome-use uncheck @e3 # uncheck
|
||||
chrome-use select @e4 "option-value" # native <select> only
|
||||
@@ -296,6 +299,23 @@ chrome-use click --coords 449,320 # same, explicit flag
|
||||
|
||||
A bare-number argument is always a coordinate, never a selector.
|
||||
|
||||
### Canvas / WebGL apps (games, map & 3D viewers, drawing tools)
|
||||
|
||||
These paint everything to a `<canvas>` and expose **almost no accessibility
|
||||
tree**, so `snapshot` comes back near-empty and refs are a dead end. `snapshot`
|
||||
detects this and prints a one-line hint. Drive them the screenshot way:
|
||||
|
||||
```bash
|
||||
chrome-use screenshot /tmp/s.png # SEE the state (your only read path —
|
||||
# eval/get text return nothing useful)
|
||||
chrome-use click 640 360 # interact by viewport coordinate
|
||||
chrome-use keydown d; sleep 0.6; chrome-use keyup d # hold-to-move
|
||||
chrome-use press Space # discrete actions (jump/attack/confirm)
|
||||
```
|
||||
|
||||
Each command is a ~250ms round-trip, so this is fine for turn-based / canvas
|
||||
*apps* but too slow to play a real-time 60fps action game frame-by-frame.
|
||||
|
||||
## Waiting (read this)
|
||||
|
||||
Agents fail more often from bad waits than from bad selectors. Pick the
|
||||
|
||||
Reference in New Issue
Block a user