From 7c594820da64f67ca5457e53794558aea571da9a Mon Sep 17 00:00:00 2001
From: leeguooooo <guoli@zhihu.com>
Date: Sun, 14 Jun 2026 00:54:30 +0900
Subject: [PATCH] docs(stream): document the bidirectional WS as the real-time
 driving path
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Dogfooding (driving a canvas game) showed the slow, low-fidelity way — one
screenshot + one CLI call per action — when chrome-use already ships the right
tool: the session WebSocket is BIDIRECTIONAL. It streams ~60fps screencast frames
AND accepts input_keyboard/input_mouse/input_touch on the same socket, straight
to CDP Input.dispatch* — verified live over the extension relay (217 frames in
3.4s, ~64fps, and the input drove the game). But the inbound input protocol was
undocumented, so agents default to the CLI-per-action grind.

Document it in --help (stream) and the core skill: the frame + input message
schemas and the 'connect once, read frames, send timed input' loop, with a node
snippet. Reserve screenshots for one-off checks; use the WS for sustained
real-time control.
---
 cli/src/output.rs        | 14 ++++++++++++++
 skill-data/core/SKILL.md | 21 +++++++++++++++++++--
 2 files changed, 33 insertions(+), 2 deletions(-)

diff --git a/cli/src/output.rs b/cli/src/output.rs
index 7f43840..bb59fc2 100644
--- a/cli/src/output.rs
+++ b/cli/src/output.rs
@@ -2778,6 +2778,20 @@ Notes:
   - Streaming is always enabled. Set AGENT_BROWSER_STREAM_PORT to bind to a
     specific port instead of the default OS-assigned port.
 
+The WS is BIDIRECTIONAL — the high-throughput way to drive a live/real-time page
+(games, canvas apps) instead of one screenshot + one CLI call per action:
+  - Server -> client (JSON text frames):
+      {"type":"frame","data":"<base64 jpeg>"}   live screencast (~60fps)
+      plus status / tabs messages.
+  - Client -> server (send JSON text):
+      {"type":"input_keyboard","eventType":"keyDown|keyUp","key":" ","code":"Space",
+       "windowsVirtualKeyCode":32}
+      {"type":"input_mouse","eventType":"mousePressed|mouseReleased|mouseMoved",
+       "x":640,"y":360,"button":"left","clickCount":1}
+      {"type":"input_touch","eventType":"touchStart|touchEnd","touchPoints":[...]}
+  Connect once and run a tight local loop: read frames, send timed input — no
+  per-action process spawn, no round-trip. Works over the extension relay too.
+
 Global Options:
   --json               Output as JSON
   --session <name>     Use specific session
diff --git a/skill-data/core/SKILL.md b/skill-data/core/SKILL.md
index 7d4162b..4c3afcf 100644
--- a/skill-data/core/SKILL.md
+++ b/skill-data/core/SKILL.md
@@ -327,8 +327,25 @@ chrome-use batch "press d --hold 900" "press j" "press j" "wait 200" "press d --
 Also try reading real state instead of pixels: `eval` runs in the page's main
 world, so for a framework/engine game you can often reach its globals (e.g. a
 Phaser/PIXI/Three instance, a store, `window.__GAME__`) and read positions/score
-directly — far better than guessing from a screenshot. Still, a real-time 60fps
-action game is not playable frame-perfect over a CLI; expect to script bursts.
+directly — far better than guessing from a screenshot.
+
+**For genuinely real-time driving, drop the CLI entirely and use the WebSocket.**
+`chrome-use stream enable` opens a bidirectional WS (`stream status` prints the
+`ws://127.0.0.1:<port>`). Connect once and you get a live ~60fps screencast AND
+can send input on the same socket — no per-action process spawn, no round-trip,
+works over the extension relay:
+
+```js
+// node (global WebSocket): live frames + locally-timed input
+const ws = new WebSocket("ws://127.0.0.1:PORT")
+ws.onmessage = e => { const m = JSON.parse(e.data); if (m.type==="frame") {/* base64 jpeg */} }
+const k = (eventType,key,code,vk) => ws.send(JSON.stringify({type:"input_keyboard",eventType,key,code,windowsVirtualKeyCode:vk}))
+k("keyDown"," ","Space",32); setTimeout(()=>k("keyUp"," ","Space",32), 80)   // a jump
+// also: {type:"input_mouse",eventType:"mousePressed",x,y,button:"left",clickCount:1}
+```
+
+This is the difference between watching a slideshow and playing the game. Reserve
+screenshots for one-off checks; use the WS for any sustained real-time control.
 
 ## Waiting (read this)