docs(stream): document the bidirectional WS as the real-time driving path

Dogfooding (driving a canvas game) showed the slow, low-fidelity way — one
screenshot + one CLI call per action — when chrome-use already ships the right
tool: the session WebSocket is BIDIRECTIONAL. It streams ~60fps screencast frames
AND accepts input_keyboard/input_mouse/input_touch on the same socket, straight
to CDP Input.dispatch* — verified live over the extension relay (217 frames in
3.4s, ~64fps, and the input drove the game). But the inbound input protocol was
undocumented, so agents default to the CLI-per-action grind.

Document it in --help (stream) and the core skill: the frame + input message
schemas and the 'connect once, read frames, send timed input' loop, with a node
snippet. Reserve screenshots for one-off checks; use the WS for sustained
real-time control.
This commit is contained in:
leeguooooo
2026-06-14 00:54:30 +09:00
parent 81d18bbd2e
commit 7c594820da
2 changed files with 33 additions and 2 deletions
+14
View File
@@ -2778,6 +2778,20 @@ Notes:
- Streaming is always enabled. Set AGENT_BROWSER_STREAM_PORT to bind to a
specific port instead of the default OS-assigned port.
The WS is BIDIRECTIONAL — the high-throughput way to drive a live/real-time page
(games, canvas apps) instead of one screenshot + one CLI call per action:
- Server -> client (JSON text frames):
{"type":"frame","data":"<base64 jpeg>"} live screencast (~60fps)
plus status / tabs messages.
- Client -> server (send JSON text):
{"type":"input_keyboard","eventType":"keyDown|keyUp","key":" ","code":"Space",
"windowsVirtualKeyCode":32}
{"type":"input_mouse","eventType":"mousePressed|mouseReleased|mouseMoved",
"x":640,"y":360,"button":"left","clickCount":1}
{"type":"input_touch","eventType":"touchStart|touchEnd","touchPoints":[...]}
Connect once and run a tight local loop: read frames, send timed input — no
per-action process spawn, no round-trip. Works over the extension relay too.
Global Options:
--json Output as JSON
--session <name> Use specific session