fix(relay): don't hijack a user tab on open; surface keydown/keyup + canvas hint
Dogfooding a canvas game over the extension relay surfaced three issues: 1. (serious) A fresh relay session's first `open` navigated one of the USER's existing tabs instead of opening its own — in testing it replaced a half-filled form with the target site. On connect the daemon passively attaches to the user's tabs and pinned one as active; navigate() then drove it. Now: on the relay (agent_group set), if the active tab isn't one this session created, navigate() opens its own tab in the session's group first. Off the relay (a browser we launched) reusing the active tab stays correct. Pure helper active_index_is_owned() + regression tests. 2. (discoverability) `keydown <key>` / `keyup <key>` (hold-to-move, essential for games/shortcuts) already existed as commands+daemon handlers but were absent from --help and the skill, so they were undiscoverable. Documented in --help, the core skill, and the canvas-app hint. 3. (UX) Canvas/WebGL pages expose almost no a11y tree, so `snapshot` is empty and agents get stuck hunting refs. snapshot now detects a viewport-dominating canvas with a sparse tree and prints a hint pointing at the screenshot + coordinate-click + keydown/keyup path. Verified live over the relay: `open` now lands the game in its own new tab with the user's tabs (incl. the Rakuten recovery form) untouched; the canvas hint fires on the game page; `close` cleans up only the session's own tab.
This commit is contained in:
@@ -297,6 +297,11 @@ pub fn print_response_with_opts(resp: &Response, action: Option<&str>, opts: &Ou
|
||||
// Snapshot
|
||||
if let Some(snapshot) = data.get("snapshot").and_then(|v| v.as_str()) {
|
||||
print_with_boundaries(snapshot, origin, opts);
|
||||
// Canvas-app hint: the tree was near-empty but the page paints to a
|
||||
// <canvas>, so refs are a dead end — point at the screenshot path.
|
||||
if let Some(note) = data.get("note").and_then(|v| v.as_str()) {
|
||||
eprintln!("{}", color::dim(note));
|
||||
}
|
||||
return;
|
||||
}
|
||||
// Title
|
||||
@@ -3066,6 +3071,9 @@ Core Commands:
|
||||
type <sel> <text> Type into element
|
||||
fill <sel> <text> Clear and fill
|
||||
press <key> Press key (Enter, Tab, Control+a)
|
||||
keydown <key> Hold a key down (no auto-release) — for games/shortcuts
|
||||
keyup <key> Release a held key. Pair with keydown to hold-to-move:
|
||||
`keydown d` … `keyup d`
|
||||
keyboard type <text> Type text with real keystrokes (no selector)
|
||||
keyboard inserttext <text> Insert text without key events
|
||||
hover <sel> Hover element
|
||||
|
||||
Reference in New Issue
Block a user