fix(relay): don't hijack a user tab on open; surface keydown/keyup + canvas hint
Dogfooding a canvas game over the extension relay surfaced three issues: 1. (serious) A fresh relay session's first `open` navigated one of the USER's existing tabs instead of opening its own — in testing it replaced a half-filled form with the target site. On connect the daemon passively attaches to the user's tabs and pinned one as active; navigate() then drove it. Now: on the relay (agent_group set), if the active tab isn't one this session created, navigate() opens its own tab in the session's group first. Off the relay (a browser we launched) reusing the active tab stays correct. Pure helper active_index_is_owned() + regression tests. 2. (discoverability) `keydown <key>` / `keyup <key>` (hold-to-move, essential for games/shortcuts) already existed as commands+daemon handlers but were absent from --help and the skill, so they were undiscoverable. Documented in --help, the core skill, and the canvas-app hint. 3. (UX) Canvas/WebGL pages expose almost no a11y tree, so `snapshot` is empty and agents get stuck hunting refs. snapshot now detects a viewport-dominating canvas with a sparse tree and prints a hint pointing at the screenshot + coordinate-click + keydown/keyup path. Verified live over the relay: `open` now lands the game in its own new tab with the user's tabs (incl. the Rakuten recovery form) untouched; the canvas hint fires on the game page; `close` cleans up only the session's own tab.
This commit is contained in:
@@ -2888,7 +2888,32 @@ async fn handle_snapshot(cmd: &Value, state: &mut DaemonState) -> Result<Value,
|
||||
})
|
||||
.collect();
|
||||
|
||||
Ok(json!({ "snapshot": tree, "origin": url, "refs": refs }))
|
||||
let ref_count = refs.len();
|
||||
let mut out = json!({ "snapshot": tree, "origin": url, "refs": refs });
|
||||
|
||||
// Canvas/WebGL apps (games, map/3D viewers, drawing tools) paint to a
|
||||
// <canvas> and expose almost no accessibility tree, so `snapshot` comes back
|
||||
// near-empty and agents get stuck looking for refs that will never exist
|
||||
// (dogfood: the Dead Cell game). When the tree is sparse but a canvas
|
||||
// dominates the viewport, tell them to switch to the screenshot-driven path.
|
||||
if ref_count < 3 {
|
||||
let canvas_js =
|
||||
"(() => { const c = document.querySelector('canvas'); if (!c) return false; \
|
||||
const r = c.getBoundingClientRect(); \
|
||||
return r.width * r.height > innerWidth * innerHeight * 0.5; })()";
|
||||
if let Ok(v) = mgr.evaluate(canvas_js, None).await {
|
||||
if v.as_bool() == Some(true) {
|
||||
out["note"] = json!(
|
||||
"This page renders to a <canvas> (game / WebGL / editor) and exposes almost no \
|
||||
accessibility tree — refs won't help. Use `screenshot` to see it, coordinate \
|
||||
`click <x> <y>` to interact, and `keydown`/`keyup`/`press` for keyboard \
|
||||
(hold-to-move: `keydown d` … `keyup d`)."
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
Ok(out)
|
||||
}
|
||||
|
||||
/// Resolve a (possibly relative) saved-file path to an absolute one so the CLI
|
||||
|
||||
Reference in New Issue
Block a user