docs(stream): document the bidirectional WS as the real-time driving path
Dogfooding (driving a canvas game) showed the slow, low-fidelity way — one screenshot + one CLI call per action — when chrome-use already ships the right tool: the session WebSocket is BIDIRECTIONAL. It streams ~60fps screencast frames AND accepts input_keyboard/input_mouse/input_touch on the same socket, straight to CDP Input.dispatch* — verified live over the extension relay (217 frames in 3.4s, ~64fps, and the input drove the game). But the inbound input protocol was undocumented, so agents default to the CLI-per-action grind. Document it in --help (stream) and the core skill: the frame + input message schemas and the 'connect once, read frames, send timed input' loop, with a node snippet. Reserve screenshots for one-off checks; use the WS for sustained real-time control.
This commit is contained in:
@@ -2778,6 +2778,20 @@ Notes:
|
||||
- Streaming is always enabled. Set AGENT_BROWSER_STREAM_PORT to bind to a
|
||||
specific port instead of the default OS-assigned port.
|
||||
|
||||
The WS is BIDIRECTIONAL — the high-throughput way to drive a live/real-time page
|
||||
(games, canvas apps) instead of one screenshot + one CLI call per action:
|
||||
- Server -> client (JSON text frames):
|
||||
{"type":"frame","data":"<base64 jpeg>"} live screencast (~60fps)
|
||||
plus status / tabs messages.
|
||||
- Client -> server (send JSON text):
|
||||
{"type":"input_keyboard","eventType":"keyDown|keyUp","key":" ","code":"Space",
|
||||
"windowsVirtualKeyCode":32}
|
||||
{"type":"input_mouse","eventType":"mousePressed|mouseReleased|mouseMoved",
|
||||
"x":640,"y":360,"button":"left","clickCount":1}
|
||||
{"type":"input_touch","eventType":"touchStart|touchEnd","touchPoints":[...]}
|
||||
Connect once and run a tight local loop: read frames, send timed input — no
|
||||
per-action process spawn, no round-trip. Works over the extension relay too.
|
||||
|
||||
Global Options:
|
||||
--json Output as JSON
|
||||
--session <name> Use specific session
|
||||
|
||||
@@ -327,8 +327,25 @@ chrome-use batch "press d --hold 900" "press j" "press j" "wait 200" "press d --
|
||||
Also try reading real state instead of pixels: `eval` runs in the page's main
|
||||
world, so for a framework/engine game you can often reach its globals (e.g. a
|
||||
Phaser/PIXI/Three instance, a store, `window.__GAME__`) and read positions/score
|
||||
directly — far better than guessing from a screenshot. Still, a real-time 60fps
|
||||
action game is not playable frame-perfect over a CLI; expect to script bursts.
|
||||
directly — far better than guessing from a screenshot.
|
||||
|
||||
**For genuinely real-time driving, drop the CLI entirely and use the WebSocket.**
|
||||
`chrome-use stream enable` opens a bidirectional WS (`stream status` prints the
|
||||
`ws://127.0.0.1:<port>`). Connect once and you get a live ~60fps screencast AND
|
||||
can send input on the same socket — no per-action process spawn, no round-trip,
|
||||
works over the extension relay:
|
||||
|
||||
```js
|
||||
// node (global WebSocket): live frames + locally-timed input
|
||||
const ws = new WebSocket("ws://127.0.0.1:PORT")
|
||||
ws.onmessage = e => { const m = JSON.parse(e.data); if (m.type==="frame") {/* base64 jpeg */} }
|
||||
const k = (eventType,key,code,vk) => ws.send(JSON.stringify({type:"input_keyboard",eventType,key,code,windowsVirtualKeyCode:vk}))
|
||||
k("keyDown"," ","Space",32); setTimeout(()=>k("keyUp"," ","Space",32), 80) // a jump
|
||||
// also: {type:"input_mouse",eventType:"mousePressed",x,y,button:"left",clickCount:1}
|
||||
```
|
||||
|
||||
This is the difference between watching a slideshow and playing the game. Reserve
|
||||
screenshots for one-off checks; use the WS for any sustained real-time control.
|
||||
|
||||
## Waiting (read this)
|
||||
|
||||
|
||||
Reference in New Issue
Block a user