diff --git a/README.md b/README.md index bed0b09..8c742b0 100644 --- a/README.md +++ b/README.md @@ -479,6 +479,113 @@ This enables control of: - WebView2 applications - Any browser exposing a CDP endpoint +## Streaming (Browser Preview) + +Stream the browser viewport via WebSocket for live preview or "pair browsing" where a human can watch and interact alongside an AI agent. + +### Enable Streaming + +Set the `AGENT_BROWSER_STREAM_PORT` environment variable: + +```bash +AGENT_BROWSER_STREAM_PORT=9223 agent-browser open example.com +``` + +This starts a WebSocket server on the specified port that streams the browser viewport and accepts input events. + +### WebSocket Protocol + +Connect to `ws://localhost:9223` to receive frames and send input: + +**Receive frames:** +```json +{ + "type": "frame", + "data": "", + "metadata": { + "deviceWidth": 1280, + "deviceHeight": 720, + "pageScaleFactor": 1, + "offsetTop": 0, + "scrollOffsetX": 0, + "scrollOffsetY": 0 + } +} +``` + +**Send mouse events:** +```json +{ + "type": "input_mouse", + "eventType": "mousePressed", + "x": 100, + "y": 200, + "button": "left", + "clickCount": 1 +} +``` + +**Send keyboard events:** +```json +{ + "type": "input_keyboard", + "eventType": "keyDown", + "key": "Enter", + "code": "Enter" +} +``` + +**Send touch events:** +```json +{ + "type": "input_touch", + "eventType": "touchStart", + "touchPoints": [{ "x": 100, "y": 200 }] +} +``` + +### Programmatic API + +For advanced use, control streaming directly via the protocol: + +```typescript +import { BrowserManager } from 'agent-browser'; + +const browser = new BrowserManager(); +await browser.launch({ headless: true }); +await browser.navigate('https://example.com'); + +// Start screencast +await browser.startScreencast((frame) => { + // frame.data is base64-encoded image + // frame.metadata contains viewport info + console.log('Frame received:', frame.metadata.deviceWidth, 'x', frame.metadata.deviceHeight); +}, { + format: 'jpeg', + quality: 80, + maxWidth: 1280, + maxHeight: 720, +}); + +// Inject mouse events +await browser.injectMouseEvent({ + type: 'mousePressed', + x: 100, + y: 200, + button: 'left', +}); + +// Inject keyboard events +await browser.injectKeyboardEvent({ + type: 'keyDown', + key: 'Enter', + code: 'Enter', +}); + +// Stop when done +await browser.stopScreencast(); +``` + ## Architecture agent-browser uses a client-daemon architecture: diff --git a/cli/src/output.rs b/cli/src/output.rs index 080b763..91b02eb 100644 --- a/cli/src/output.rs +++ b/cli/src/output.rs @@ -1197,6 +1197,11 @@ Options: --cdp Connect via CDP (Chrome DevTools Protocol) --debug Debug output +Environment: + AGENT_BROWSER_SESSION Session name (default: "default") + AGENT_BROWSER_EXECUTABLE_PATH Custom browser executable path + AGENT_BROWSER_STREAM_PORT Enable WebSocket streaming on port (e.g., 9223) + Examples: agent-browser open example.com agent-browser snapshot -i # Interactive elements only diff --git a/docs/src/app/streaming/page.tsx b/docs/src/app/streaming/page.tsx new file mode 100644 index 0000000..aa93037 --- /dev/null +++ b/docs/src/app/streaming/page.tsx @@ -0,0 +1,217 @@ +import { CodeBlock } from "@/components/code-block"; + +export default function Streaming() { + return ( +
+
+

Streaming

+

+ Stream the browser viewport via WebSocket for live preview or "pair browsing" + where a human can watch and interact alongside an AI agent. +

+ +

Enable streaming

+

+ Set the AGENT_BROWSER_STREAM_PORT environment variable to start + a WebSocket server: +

+ + +

+ The server streams viewport frames and accepts input events (mouse, keyboard, touch). +

+ +

WebSocket protocol

+

Connect to ws://localhost:9223 to receive frames and send input.

+ +

Frame messages

+

The server sends frame messages with base64-encoded images:

+ ", + "metadata": { + "deviceWidth": 1280, + "deviceHeight": 720, + "pageScaleFactor": 1, + "offsetTop": 0, + "scrollOffsetX": 0, + "scrollOffsetY": 0 + } +}`} /> + +

Status messages

+

Connection and screencast status:

+ + +

Input injection

+

Send input events to control the browser remotely.

+ +

Mouse events

+ + +

Keyboard events

+ + +

Touch events

+ + +

Programmatic API

+

For advanced use, control streaming directly via the TypeScript API:

+ { + console.log('Frame:', frame.metadata.deviceWidth, 'x', frame.metadata.deviceHeight); + // frame.data is base64-encoded image +}, { + format: 'jpeg', // or 'png' + quality: 80, // 0-100, jpeg only + maxWidth: 1280, + maxHeight: 720, + everyNthFrame: 1 +}); + +// Inject mouse event +await browser.injectMouseEvent({ + type: 'mousePressed', + x: 100, + y: 200, + button: 'left', + clickCount: 1 +}); + +// Inject keyboard event +await browser.injectKeyboardEvent({ + type: 'keyDown', + key: 'Enter', + code: 'Enter' +}); + +// Inject touch event +await browser.injectTouchEvent({ + type: 'touchStart', + touchPoints: [{ x: 100, y: 200 }] +}); + +// Check if screencasting +console.log('Active:', browser.isScreencasting()); + +// Stop screencast +await browser.stopScreencast();`} /> + +

Use cases

+
    +
  • Pair browsing - Human watches and assists AI agent in real-time
    • +
  • Remote preview - View browser output in a separate UI
    • +
  • Recording - Capture frames for video generation
    • +
  • Mobile testing - Inject touch events for mobile emulation
    • +
  • Accessibility testing - Manual interaction during automated tests
    • +
+
+
+ ); +} diff --git a/docs/src/components/sidebar.tsx b/docs/src/components/sidebar.tsx index 1d40f51..55a8585 100644 --- a/docs/src/components/sidebar.tsx +++ b/docs/src/components/sidebar.tsx @@ -12,6 +12,7 @@ const navigation = [ { name: "Selectors", href: "/selectors" }, { name: "Sessions", href: "/sessions" }, { name: "Snapshots", href: "/snapshots" }, + { name: "Streaming", href: "/streaming" }, { name: "Agent Mode", href: "/agent-mode" }, { name: "CDP Mode", href: "/cdp-mode" }, ]; diff --git a/src/actions.ts b/src/actions.ts index 747ca1b..e9af532 100644 --- a/src/actions.ts +++ b/src/actions.ts @@ -709,7 +709,7 @@ async function handleTabSwitch( command: TabSwitchCommand, browser: BrowserManager ): Promise> { - const result = browser.switchTo(command.index); + const result = await browser.switchTo(command.index); const page = browser.getPage(); return successResponse(command.id, { ...result, diff --git a/src/browser.test.ts b/src/browser.test.ts index ec61581..80e0737 100644 --- a/src/browser.test.ts +++ b/src/browser.test.ts @@ -378,4 +378,256 @@ describe('BrowserManager', () => { await expect(browser.clearScopedHeaders('https://never-set.com')).resolves.not.toThrow(); }); }); + + describe('CDP session', () => { + it('should create CDP session on demand', async () => { + const cdp = await browser.getCDPSession(); + expect(cdp).toBeDefined(); + }); + + it('should reuse existing CDP session', async () => { + const cdp1 = await browser.getCDPSession(); + const cdp2 = await browser.getCDPSession(); + expect(cdp1).toBe(cdp2); + }); + }); + + describe('screencast', () => { + it('should report screencasting state correctly', () => { + expect(browser.isScreencasting()).toBe(false); + }); + + it('should start screencast', async () => { + const frames: Array<{ data: string }> = []; + await browser.startScreencast((frame) => { + frames.push(frame); + }); + expect(browser.isScreencasting()).toBe(true); + + // Wait a bit for at least one frame + await new Promise((resolve) => setTimeout(resolve, 200)); + + await browser.stopScreencast(); + expect(browser.isScreencasting()).toBe(false); + expect(frames.length).toBeGreaterThan(0); + }); + + it('should start screencast with custom options', async () => { + const frames: Array<{ data: string }> = []; + await browser.startScreencast( + (frame) => { + frames.push(frame); + }, + { + format: 'png', + quality: 100, + maxWidth: 800, + maxHeight: 600, + everyNthFrame: 1, + } + ); + expect(browser.isScreencasting()).toBe(true); + + // Wait for a frame + await new Promise((resolve) => setTimeout(resolve, 200)); + + await browser.stopScreencast(); + expect(frames.length).toBeGreaterThan(0); + }); + + it('should throw when starting screencast twice', async () => { + await browser.startScreencast(() => {}); + await expect(browser.startScreencast(() => {})).rejects.toThrow('Screencast already active'); + await browser.stopScreencast(); + }); + + it('should handle stop when not screencasting', async () => { + // Should not throw + await expect(browser.stopScreencast()).resolves.not.toThrow(); + }); + }); + + describe('tab switch invalidates CDP session', () => { + // Clean up any extra tabs before each test + beforeEach(async () => { + // Close all tabs except the first one + const tabs = await browser.listTabs(); + for (let i = tabs.length - 1; i > 0; i--) { + await browser.closeTab(i); + } + // Ensure we're on tab 0 + await browser.switchTo(0); + // Stop any active screencast + if (browser.isScreencasting()) { + await browser.stopScreencast(); + } + }); + + it('should not invalidate CDP when switching to same tab', async () => { + // Get CDP session for current tab + const cdp1 = await browser.getCDPSession(); + + // Switch to same tab - should NOT invalidate + await browser.switchTo(0); + + // Should be the same session + const cdp2 = await browser.getCDPSession(); + expect(cdp2).toBe(cdp1); + }); + + it('should invalidate CDP session on tab switch', async () => { + // Get CDP session for tab 0 + const cdp1 = await browser.getCDPSession(); + expect(cdp1).toBeDefined(); + + // Create new tab - this switches to the new tab automatically + await browser.newTab(); + + // Get CDP session - should be different since we're on a new page + const cdp2 = await browser.getCDPSession(); + expect(cdp2).toBeDefined(); + + // Sessions should be different objects (different pages have different CDP sessions) + expect(cdp2).not.toBe(cdp1); + }); + + it('should stop screencast on tab switch', async () => { + // Start screencast on tab 0 + await browser.startScreencast(() => {}); + expect(browser.isScreencasting()).toBe(true); + + // Create new tab and switch + await browser.newTab(); + await browser.switchTo(1); + + // Screencast should be stopped (it's page-specific) + expect(browser.isScreencasting()).toBe(false); + }); + }); + + describe('input injection', () => { + it('should inject mouse move event', async () => { + await expect( + browser.injectMouseEvent({ + type: 'mouseMoved', + x: 100, + y: 100, + }) + ).resolves.not.toThrow(); + }); + + it('should inject mouse click events', async () => { + await expect( + browser.injectMouseEvent({ + type: 'mousePressed', + x: 100, + y: 100, + button: 'left', + clickCount: 1, + }) + ).resolves.not.toThrow(); + + await expect( + browser.injectMouseEvent({ + type: 'mouseReleased', + x: 100, + y: 100, + button: 'left', + }) + ).resolves.not.toThrow(); + }); + + it('should inject mouse wheel event', async () => { + await expect( + browser.injectMouseEvent({ + type: 'mouseWheel', + x: 100, + y: 100, + deltaX: 0, + deltaY: 100, + }) + ).resolves.not.toThrow(); + }); + + it('should inject keyboard events', async () => { + await expect( + browser.injectKeyboardEvent({ + type: 'keyDown', + key: 'a', + code: 'KeyA', + }) + ).resolves.not.toThrow(); + + await expect( + browser.injectKeyboardEvent({ + type: 'keyUp', + key: 'a', + code: 'KeyA', + }) + ).resolves.not.toThrow(); + }); + + it('should inject char event', async () => { + // CDP char events only accept single characters + await expect( + browser.injectKeyboardEvent({ + type: 'char', + text: 'h', + }) + ).resolves.not.toThrow(); + }); + + it('should inject keyboard with modifiers', async () => { + await expect( + browser.injectKeyboardEvent({ + type: 'keyDown', + key: 'c', + code: 'KeyC', + modifiers: 2, // Ctrl + }) + ).resolves.not.toThrow(); + }); + + it('should inject touch events', async () => { + await expect( + browser.injectTouchEvent({ + type: 'touchStart', + touchPoints: [{ x: 100, y: 100 }], + }) + ).resolves.not.toThrow(); + + await expect( + browser.injectTouchEvent({ + type: 'touchMove', + touchPoints: [{ x: 150, y: 150 }], + }) + ).resolves.not.toThrow(); + + await expect( + browser.injectTouchEvent({ + type: 'touchEnd', + touchPoints: [], + }) + ).resolves.not.toThrow(); + }); + + it('should inject multi-touch events', async () => { + await expect( + browser.injectTouchEvent({ + type: 'touchStart', + touchPoints: [ + { x: 100, y: 100, id: 0 }, + { x: 200, y: 200, id: 1 }, + ], + }) + ).resolves.not.toThrow(); + + await expect( + browser.injectTouchEvent({ + type: 'touchEnd', + touchPoints: [], + }) + ).resolves.not.toThrow(); + }); + }); }); diff --git a/src/browser.ts b/src/browser.ts index 284783c..23f2585 100644 --- a/src/browser.ts +++ b/src/browser.ts @@ -782,6 +782,9 @@ export class BrowserManager { throw new Error('Browser not launched'); } + // Invalidate CDP session since we're switching to a new page + await this.invalidateCDPSession(); + const context = this.contexts[0]; // Use first context for tabs const page = await context.newPage(); this.pages.push(page); @@ -820,14 +823,36 @@ export class BrowserManager { return { index: this.activePageIndex, total: this.pages.length }; } + /** + * Invalidate the current CDP session (must be called before switching pages) + * This ensures screencast and input injection work correctly after tab switch + */ + private async invalidateCDPSession(): Promise { + // Stop screencast if active (it's tied to the current page's CDP session) + if (this.screencastActive) { + await this.stopScreencast(); + } + + // Detach and clear the CDP session + if (this.cdpSession) { + await this.cdpSession.detach().catch(() => {}); + this.cdpSession = null; + } + } + /** * Switch to a specific tab/page by index */ - switchTo(index: number): { index: number; url: string; title: string } { + async switchTo(index: number): Promise<{ index: number; url: string; title: string }> { if (index < 0 || index >= this.pages.length) { throw new Error(`Invalid tab index: ${index}. Available: 0-${this.pages.length - 1}`); } + // Invalidate CDP session before switching (it's page-specific) + if (index !== this.activePageIndex) { + await this.invalidateCDPSession(); + } + this.activePageIndex = index; const page = this.pages[index]; @@ -852,6 +877,11 @@ export class BrowserManager { throw new Error('Cannot close the last tab. Use "close" to close the browser.'); } + // If closing the active tab, invalidate CDP session first + if (targetIndex === this.activePageIndex) { + await this.invalidateCDPSession(); + } + const page = this.pages[targetIndex]; await page.close(); this.pages.splice(targetIndex, 1); diff --git a/src/protocol.test.ts b/src/protocol.test.ts index c7a0556..720306f 100644 --- a/src/protocol.test.ts +++ b/src/protocol.test.ts @@ -620,6 +620,391 @@ describe('parseCommand', () => { }); }); + describe('screencast', () => { + it('should parse screencast_start with defaults', () => { + const result = parseCommand(cmd({ id: '1', action: 'screencast_start' })); + expect(result.success).toBe(true); + if (result.success) { + expect(result.command.action).toBe('screencast_start'); + } + }); + + it('should parse screencast_start with all options', () => { + const result = parseCommand( + cmd({ + id: '1', + action: 'screencast_start', + format: 'png', + quality: 90, + maxWidth: 1920, + maxHeight: 1080, + everyNthFrame: 2, + }) + ); + expect(result.success).toBe(true); + if (result.success) { + expect(result.command.format).toBe('png'); + expect(result.command.quality).toBe(90); + expect(result.command.maxWidth).toBe(1920); + expect(result.command.maxHeight).toBe(1080); + expect(result.command.everyNthFrame).toBe(2); + } + }); + + it('should reject screencast_start with invalid format', () => { + const result = parseCommand(cmd({ id: '1', action: 'screencast_start', format: 'gif' })); + expect(result.success).toBe(false); + }); + + it('should reject screencast_start with quality out of range', () => { + const result = parseCommand(cmd({ id: '1', action: 'screencast_start', quality: 150 })); + expect(result.success).toBe(false); + }); + + it('should reject screencast_start with negative maxWidth', () => { + const result = parseCommand(cmd({ id: '1', action: 'screencast_start', maxWidth: -100 })); + expect(result.success).toBe(false); + }); + + it('should parse screencast_stop', () => { + const result = parseCommand(cmd({ id: '1', action: 'screencast_stop' })); + expect(result.success).toBe(true); + if (result.success) { + expect(result.command.action).toBe('screencast_stop'); + } + }); + }); + + describe('input injection', () => { + describe('input_mouse', () => { + it('should parse mousePressed event', () => { + const result = parseCommand( + cmd({ + id: '1', + action: 'input_mouse', + type: 'mousePressed', + x: 100, + y: 200, + button: 'left', + }) + ); + expect(result.success).toBe(true); + if (result.success) { + expect(result.command.action).toBe('input_mouse'); + expect(result.command.type).toBe('mousePressed'); + expect(result.command.x).toBe(100); + expect(result.command.y).toBe(200); + expect(result.command.button).toBe('left'); + } + }); + + it('should parse mouseReleased event', () => { + const result = parseCommand( + cmd({ + id: '1', + action: 'input_mouse', + type: 'mouseReleased', + x: 100, + y: 200, + }) + ); + expect(result.success).toBe(true); + }); + + it('should parse mouseMoved event', () => { + const result = parseCommand( + cmd({ + id: '1', + action: 'input_mouse', + type: 'mouseMoved', + x: 150, + y: 250, + }) + ); + expect(result.success).toBe(true); + }); + + it('should parse mouseWheel event with deltas', () => { + const result = parseCommand( + cmd({ + id: '1', + action: 'input_mouse', + type: 'mouseWheel', + x: 100, + y: 200, + deltaX: 0, + deltaY: 100, + }) + ); + expect(result.success).toBe(true); + if (result.success) { + expect(result.command.deltaX).toBe(0); + expect(result.command.deltaY).toBe(100); + } + }); + + it('should parse mouse event with modifiers', () => { + const result = parseCommand( + cmd({ + id: '1', + action: 'input_mouse', + type: 'mousePressed', + x: 100, + y: 200, + modifiers: 6, // Ctrl + Meta + }) + ); + expect(result.success).toBe(true); + if (result.success) { + expect(result.command.modifiers).toBe(6); + } + }); + + it('should parse mouse event with clickCount', () => { + const result = parseCommand( + cmd({ + id: '1', + action: 'input_mouse', + type: 'mousePressed', + x: 100, + y: 200, + clickCount: 2, + }) + ); + expect(result.success).toBe(true); + if (result.success) { + expect(result.command.clickCount).toBe(2); + } + }); + + it('should reject input_mouse with invalid type', () => { + const result = parseCommand( + cmd({ + id: '1', + action: 'input_mouse', + type: 'invalid', + x: 100, + y: 200, + }) + ); + expect(result.success).toBe(false); + }); + + it('should reject input_mouse without x coordinate', () => { + const result = parseCommand( + cmd({ + id: '1', + action: 'input_mouse', + type: 'mousePressed', + y: 200, + }) + ); + expect(result.success).toBe(false); + }); + + it('should reject input_mouse without y coordinate', () => { + const result = parseCommand( + cmd({ + id: '1', + action: 'input_mouse', + type: 'mousePressed', + x: 100, + }) + ); + expect(result.success).toBe(false); + }); + }); + + describe('input_keyboard', () => { + it('should parse keyDown event', () => { + const result = parseCommand( + cmd({ + id: '1', + action: 'input_keyboard', + type: 'keyDown', + key: 'Enter', + code: 'Enter', + }) + ); + expect(result.success).toBe(true); + if (result.success) { + expect(result.command.action).toBe('input_keyboard'); + expect(result.command.type).toBe('keyDown'); + expect(result.command.key).toBe('Enter'); + expect(result.command.code).toBe('Enter'); + } + }); + + it('should parse keyUp event', () => { + const result = parseCommand( + cmd({ + id: '1', + action: 'input_keyboard', + type: 'keyUp', + key: 'a', + }) + ); + expect(result.success).toBe(true); + }); + + it('should parse char event with text', () => { + const result = parseCommand( + cmd({ + id: '1', + action: 'input_keyboard', + type: 'char', + text: 'hello', + }) + ); + expect(result.success).toBe(true); + if (result.success) { + expect(result.command.text).toBe('hello'); + } + }); + + it('should parse keyboard event with modifiers', () => { + const result = parseCommand( + cmd({ + id: '1', + action: 'input_keyboard', + type: 'keyDown', + key: 'c', + modifiers: 2, // Ctrl + }) + ); + expect(result.success).toBe(true); + if (result.success) { + expect(result.command.modifiers).toBe(2); + } + }); + + it('should reject input_keyboard with invalid type', () => { + const result = parseCommand( + cmd({ + id: '1', + action: 'input_keyboard', + type: 'invalid', + }) + ); + expect(result.success).toBe(false); + }); + }); + + describe('input_touch', () => { + it('should parse touchStart event', () => { + const result = parseCommand( + cmd({ + id: '1', + action: 'input_touch', + type: 'touchStart', + touchPoints: [{ x: 100, y: 200 }], + }) + ); + expect(result.success).toBe(true); + if (result.success) { + expect(result.command.action).toBe('input_touch'); + expect(result.command.type).toBe('touchStart'); + expect(result.command.touchPoints).toHaveLength(1); + expect(result.command.touchPoints[0].x).toBe(100); + expect(result.command.touchPoints[0].y).toBe(200); + } + }); + + it('should parse touchEnd event', () => { + const result = parseCommand( + cmd({ + id: '1', + action: 'input_touch', + type: 'touchEnd', + touchPoints: [], + }) + ); + expect(result.success).toBe(true); + }); + + it('should parse touchMove event', () => { + const result = parseCommand( + cmd({ + id: '1', + action: 'input_touch', + type: 'touchMove', + touchPoints: [{ x: 150, y: 250 }], + }) + ); + expect(result.success).toBe(true); + }); + + it('should parse touchCancel event', () => { + const result = parseCommand( + cmd({ + id: '1', + action: 'input_touch', + type: 'touchCancel', + touchPoints: [], + }) + ); + expect(result.success).toBe(true); + }); + + it('should parse multi-touch event', () => { + const result = parseCommand( + cmd({ + id: '1', + action: 'input_touch', + type: 'touchStart', + touchPoints: [ + { x: 100, y: 200, id: 0 }, + { x: 300, y: 400, id: 1 }, + ], + }) + ); + expect(result.success).toBe(true); + if (result.success) { + expect(result.command.touchPoints).toHaveLength(2); + } + }); + + it('should parse touch event with modifiers', () => { + const result = parseCommand( + cmd({ + id: '1', + action: 'input_touch', + type: 'touchStart', + touchPoints: [{ x: 100, y: 200 }], + modifiers: 8, // Shift + }) + ); + expect(result.success).toBe(true); + if (result.success) { + expect(result.command.modifiers).toBe(8); + } + }); + + it('should reject input_touch with invalid type', () => { + const result = parseCommand( + cmd({ + id: '1', + action: 'input_touch', + type: 'invalid', + touchPoints: [], + }) + ); + expect(result.success).toBe(false); + }); + + it('should reject input_touch without touchPoints', () => { + const result = parseCommand( + cmd({ + id: '1', + action: 'input_touch', + type: 'touchStart', + }) + ); + expect(result.success).toBe(false); + }); + }); + }); + describe('invalid commands', () => { it('should reject unknown action', () => { const result = parseCommand(cmd({ id: '1', action: 'unknown' }));