diff --git a/cli/src/output.rs b/cli/src/output.rs index 7f43840..bb59fc2 100644 --- a/cli/src/output.rs +++ b/cli/src/output.rs @@ -2778,6 +2778,20 @@ Notes: - Streaming is always enabled. Set AGENT_BROWSER_STREAM_PORT to bind to a specific port instead of the default OS-assigned port. +The WS is BIDIRECTIONAL — the high-throughput way to drive a live/real-time page +(games, canvas apps) instead of one screenshot + one CLI call per action: + - Server -> client (JSON text frames): + {"type":"frame","data":""} live screencast (~60fps) + plus status / tabs messages. + - Client -> server (send JSON text): + {"type":"input_keyboard","eventType":"keyDown|keyUp","key":" ","code":"Space", + "windowsVirtualKeyCode":32} + {"type":"input_mouse","eventType":"mousePressed|mouseReleased|mouseMoved", + "x":640,"y":360,"button":"left","clickCount":1} + {"type":"input_touch","eventType":"touchStart|touchEnd","touchPoints":[...]} + Connect once and run a tight local loop: read frames, send timed input — no + per-action process spawn, no round-trip. Works over the extension relay too. + Global Options: --json Output as JSON --session Use specific session diff --git a/skill-data/core/SKILL.md b/skill-data/core/SKILL.md index 7d4162b..4c3afcf 100644 --- a/skill-data/core/SKILL.md +++ b/skill-data/core/SKILL.md @@ -327,8 +327,25 @@ chrome-use batch "press d --hold 900" "press j" "press j" "wait 200" "press d -- Also try reading real state instead of pixels: `eval` runs in the page's main world, so for a framework/engine game you can often reach its globals (e.g. a Phaser/PIXI/Three instance, a store, `window.__GAME__`) and read positions/score -directly — far better than guessing from a screenshot. Still, a real-time 60fps -action game is not playable frame-perfect over a CLI; expect to script bursts. +directly — far better than guessing from a screenshot. + +**For genuinely real-time driving, drop the CLI entirely and use the WebSocket.** +`chrome-use stream enable` opens a bidirectional WS (`stream status` prints the +`ws://127.0.0.1:`). Connect once and you get a live ~60fps screencast AND +can send input on the same socket — no per-action process spawn, no round-trip, +works over the extension relay: + +```js +// node (global WebSocket): live frames + locally-timed input +const ws = new WebSocket("ws://127.0.0.1:PORT") +ws.onmessage = e => { const m = JSON.parse(e.data); if (m.type==="frame") {/* base64 jpeg */} } +const k = (eventType,key,code,vk) => ws.send(JSON.stringify({type:"input_keyboard",eventType,key,code,windowsVirtualKeyCode:vk})) +k("keyDown"," ","Space",32); setTimeout(()=>k("keyUp"," ","Space",32), 80) // a jump +// also: {type:"input_mouse",eventType:"mousePressed",x,y,button:"left",clickCount:1} +``` + +This is the difference between watching a slideshow and playing the game. Reserve +screenshots for one-off checks; use the WS for any sustained real-time control. ## Waiting (read this)