Follow-up to #36 — make the whole interaction surface reach cross-origin OOPIFs
and stop coordinate events drifting onto the user's foreground tab over the relay.
- hover/dblclick/drag now DOM-dispatch over the relay or into an iframe (like
click already did): a coordinate Input event isn't confined to the target tab
on a busy real Chrome and can't map an OOPIF element's box to a top-viewport
point. drag does an HTML5 DnD in the element's frame; cross-frame drag errors
loudly instead of drifting.
- snapshot recurses iframes to MAX_IFRAME_DEPTH (3) instead of one level, so refs
inside nested payment/checkout widgets get a frame_id and resolve into the
right frame.
- relay tab adoption merges several Target.getTargets snapshots — a single flaky
relay snapshot was dropping live tabs (a driven tab vanished after restart).
- `type --key-events` (alias --keys) sends real per-character keyDown/keyUp
instead of Input.insertText, so autocomplete/combobox widgets that ignore the
insertText input event fire (Google address postal lookup; commits Angular
reactive forms so Save enables).
- SKILL: hard "snapshot-first, never default to screenshot+coordinates" rule;
snapshot -i pierces cross-origin iframes since v1.5.12; cross-origin iframe
driving guidance (#37).
#35: `open` auto-reattaches when the bound relay tab is gone — drops the dead
page, opens a fresh tab in the session's group, and navigates it, instead of
only `tab new` recovering.
#36: scroll and click now reach content inside cross-origin OOPIFs:
- scroll dispatches a real wheel at a viewport point (default center, --at x,y,
or --frame n) so it scrolls the iframe under the pointer, which
window.scrollBy on the top document silently no-ops on.
- over the extension relay, clicks always use DOM-dispatch instead of
coordinate Input events — a coordinate event isn't confined to the target tab
on a busy real Chrome (it drifted onto the foreground tab) and an OOPIF
element's box can't be mapped to a top-viewport point.
The connect-mode diagnostic (1.5.5) proved the 'Allow remote debugging?' modal
is NOT Chrome 149 UX (my earlier hypothesis) — it's chrome-use launching a fresh
debug-port Chrome on session=default while the ab-connect relay is up (32 logged
CONSENT-MODAL-RISK launches), almost always from a stray --launch/--no-auto-connect.
A launch now warns loudly when the relay is available, naming the modal and how
to avoid it (drop --launch/--new, don't pass --no-auto-connect), so the modal is
self-explained and the offending caller is fixable.
#33: some sites prepend runs of ZWJ/word-joiner/invisible-times/BOM to
document.title (badging/anti-scrape); left in, they polluted 'tab list', broke
text matching, and wrecked column alignment. sanitize_title() now strips
zero-width/bidi-format chars at every title ingestion point + get_title().
#34: 'screenshot <selector>' (element capture) already worked but was
undocumented; added 'screenshot --clip x,y,w,h' for an explicit pixel region
(CDP captureScreenshot clip), documented both in --help. Verified live.
chrome.debugger forbids DOM.setFileInputFiles, so 'upload' used to hard-fail on
the relay and push users to a --launch/direct-CDP session. Now it falls back to
reconstructing the File entirely in the page (Playwright/Cypress-style: build a
File from the bytes, assign input.files = dataTransfer.files, fire input/change;
for drop/paste composers like X, dispatch synthetic paste+drop with the
DataTransfer). The bytes are streamed in <1 MiB base64 chunks because the relay
tunnels CDP through native messaging (1 MiB/message cap) — a whole image as one
arg closed the channel. Verified live over the relay: an 809 KB PNG lands intact
on a file input with change firing. No more direct-CDP needed for uploads.
Passing a Cloudflare challenge mints an HttpOnly cf_clearance cookie bound to
IP+UA. 'chrome-use cf-status' (aliases cf/cloudflare-status/clearance) reports
whether the active page is currently a CF challenge and whether a still-valid
cf_clearance exists (read via CDP — HttpOnly is invisible to document.cookie),
plus CF_VERIFIED_DEVICE trust, and a recommendation: proceed (already cleared,
don't re-solve) / solve (challenge up, no clearance) / reissue (clearance present
but page still blocks → IP/UA drifted). Lets an agent avoid re-solving what it
already cleared — the persistence optimization. Pure helpers unit-tested; live
-verified on a real cf_clearance.
Driving a busy real Chrome via the relay, a single Target.getTargets call
occasionally returns a different window's tabs ('tab list hops windows'). resync
pruned every tracked page absent from that snapshot — including the agent's
explicitly-adopted (pinned) tab — after which active-target resolution fell back
to active_page_index and eval/click/snapshot drifted onto a foreign tab
(about:blank / chrome-extension:// / the user's page), breaking any 3+ step flow.
prunable_target_ids() now protects the pinned active target from snapshot-based
pruning; a genuine close still arrives as Target.targetDestroyed (event drain) and
removes it properly. Unit-tested.
The consent modal only appears on a raw remote-debugging attach or a browser we
launched with a debug port — never on the ab-connect extension relay. Append one
line per connection to ~/.chrome-use/connect-mode.log (relay | raw-port-attach |
launched | remote-ws), flagging 'CONSENT-MODAL-RISK' when a raw-port/launch path
runs while the relay was available. Lets us tell a code regression from Chrome's
own extension-debugger consent UX when the modal reappears. Best-effort, never
fails a connection. Verified: normal 'open' logs mode=relay (consent-free).
Some injected UI (browser-extension debug panels, web components) renders into a
CLOSED shadow root that eval/innerText cannot read. --pierce walks the CDP DOM
tree (DOM.getDocument depth:-1 pierce:true), which includes closed shadow roots
and child documents, and collects text nodes (skipping script/style/etc).
Review-safe: rides the per-tab debugger session already attached, no new Chrome
permission and no ab-connect/extension change — so it works in extension-relay
mode without touching the published extension. Verified live: a closed-shadow
panel that main-world eval reports HIDDEN is read in full via --pierce.
First slice of #30 (read extension/injected-panel content). Deeper extension
introspection (background SW / chrome.storage) stays a launch-mode / raw-CDP
concern, deliberately NOT done by expanding ab-connect's debugger powers.
So an agent never silently misses iframed content (listing descriptions etc.)
without having to know the --all-frames flag. Single-frame pages are unchanged
(identical to the old body read); multi-frame pages now include child frames —
a strict superset. Skill + help updated to make the default and 'frames'/--main
discoverable.
The changelog step runs under 'bash -e'. section() returned non-zero when a
commit category was empty (grep no-match / empty [ -n ] test), aborting the
script before the closing heredoc delimiter — so any release whose range lacked
a whole category (e.g. 1.5.2: only feat, no fix) failed to attach binaries.
Add '|| true' + 'return 0' so section() always succeeds.
- chrome-use sessions: top-level alias for the daemon inventory (the skill
advertises sessions, so it's a natural guess that used to error).
- Unknown commands now suggest the nearest valid one (Levenshtein + prefix
match), staying silent when nothing is close (e.g. 'clik' -> click,
'sesions' -> sessions, 'xyzzy' -> no suggestion).
- tab <id>: probe the switched session and show a warning indicator instead of
a green check when it isn't responding yet, so a switch onto a re-attaching
(churned-tabId) session no longer reports false success. The #24 targetId
recovery self-heals within ~6s, hence a warning rather than a hard error.
On listing/marketplace pages (Yahoo Auctions, Rakuten, Mercari shops) the
seller's description lives in a child frame or under a related-items sidebar,
so 'get text body' returned only header/nav boilerplate.
- get text --all-frames: aggregate visible text across every reachable frame.
Same-process child frames are read via Page.createIsolatedWorld; OOPIFs via
their auto-attached debugger session (iframe_sessions). Each non-top frame is
labelled with a '----- frame [kind] url -----' separator.
- get text --main: readability-lite — prefer the densest <main>/<article>
region over the whole body, dropping global header/nav/footer chrome.
- frames: enumerate frames (kind + url + per-frame text length) so an agent can
see where a page's text actually lives and pick the right read.
Verified live: inline srcdoc frame text aggregated through --all-frames; Yahoo
Auctions <main> (2881 chars) extracted via --main, stripping the Yahoo header.
Release notes were a flat list of commit subjects — hard to tell at a glance what
was added vs fixed (recurring '看不出改了什么'). Group by conventional-commit type
so every future release auto-shows scannable Features/Fixes sections.
Live-reproduced #24 on 0.4.8 driving the Mercari signin token-exchange hop
(login.jp.mercari.com): the cross-process nav gives the tab a NEW Chrome tabId
while the CDP targetId stays the same. So cb-tab-<oldTabId> can't be recovered —
recoverSessionTab parsed the old tabId, chrome.tabs.get(oldTabId) failed (gone),
and it gave up → permanent 'stale sessionId ... its tab is gone' until the page
settled ~6s later and something re-attached. current/tab <targetId>/daemon
restart all failed because the relay still mapped the targetId to the dead
session.
Fix: remember each session's targetId across detach (sessionTargets map). When
recoverSessionTab can't recover by the encoded tabId, fall back to the STABLE
targetId — chrome.debugger.getTargets() to find the tab now hosting that target,
attach it, and ALIAS the dead cb-tab-<oldTabId> session to the live tab so the
daemon's session id keeps resolving. Longer retry window (~6s) since this hop
takes seconds to settle. Builds on 0.4.6/0.4.8 reattach; covers the tabId-churn
case those missed.
Needs dogfood on the real Mercari flow (can't repro the tabId churn synthetically).
#25 — fill() didn't fire the events framework inputs / site autocomplete need:
it set value directly (bypassing React's value-tracker) and typed via
Input.insertText, so controlled components and input/change/blur listeners (e.g.
Mercari's postal-code → 都道府県 lookup) never ran though the value showed. fill
now emulates a real edit: focus, set through the element's prototype value setter
(React _valueTracker registers), then dispatch input → input → change → blur/
focusout. SELECT and contenteditable handled too. type <sel> <text> remains for
per-keystroke sites. Verified live: an input wired with input/change/blur fired
'IICB' from one fill.
#26 (ergonomics):
- 'close <tab>' now closes just that tab and prints 'Tab [tN] closed'; bare
'close' still closes the browser. Previously 'close t12' ran a browser close
and alarmingly printed 'Browser closed'.
- new 'chrome-use current': prints the active tab's stable handle (tabId + CDP
targetId + url/title), refreshed live — so an agent holds the targetId (which
survives cross-process nav) instead of re-deriving 'which tab is live' from
'tabs' every step. The deeper tab-id churn is the #21/#23 stable-targetId story.
Tests cover fill events (live), close tab-vs-browser parse, and current.
A click on a target=_blank link / window.open opened a new tab, but the active
tab stayed put, so the post-click snapshot showed the OLD page — looking exactly
like the click failed. On the relay the new tab is discovered only via getTargets
(the relay doesn't push target events to the daemon), so it went unsurfaced.
handle_click now snapshots tracked targets before the click and, after, runs a
lightweight BrowserManager::adopt_newly_opened (one getTargets, attaches only the
new target — far cheaper than a full resync) to detect a freshly-opened tab. It's
reported as openedTab {tabId,url,title} in the response (and a '→ opened new tab
[tN] <url>' hint in text mode). Default keeps focus on the current tab (so
multi-tab flows aren't hijacked, per #7/#8.1); 'click <sel> --follow' switches to
the new tab. Verified live: clicking a _blank link prints
'→ opened new tab [t13] https://example.org/'.
Completes the #24 friction items (B/C/D shipped in 770708b).
Three CLI gaps surfaced driving a Mercari signup→checkout flow:
- #24-B (correctness): a bare label like 'click 購入手続きへ' was fed straight to
document.querySelector as CSS and failed as an invalid selector, even though
snapshot listed the button by that exact name. build_find_element_js now tries
CSS first, then falls back to matching an interactive element by visible text
(exact then contains) — nested and non-ASCII labels resolve. 'text=<label>'
forces the text path. CSS still wins when it matches.
- #24-D: 'get text' with no selector now returns the whole page (body).
- #24-C: 'tab <ref> --activate' (alias --front) switches to the tab AND raises it
to the foreground — to surface a specific tab for the human.
Tests cover the text fallback / text= / xpath builder, body default, activate
flag. The core stale-sessionId-after-cross-process-nav bug is the #20/#23 class,
already fixed in ext 0.4.8 — needs that extension deployed.
Dogfooding (driving a canvas game) showed the slow, low-fidelity way — one
screenshot + one CLI call per action — when chrome-use already ships the right
tool: the session WebSocket is BIDIRECTIONAL. It streams ~60fps screencast frames
AND accepts input_keyboard/input_mouse/input_touch on the same socket, straight
to CDP Input.dispatch* — verified live over the extension relay (217 frames in
3.4s, ~64fps, and the input drove the game). But the inbound input protocol was
undocumented, so agents default to the CLI-per-action grind.
Document it in --help (stream) and the core skill: the frame + input message
schemas and the 'connect once, read frames, send timed input' loop, with a node
snippet. Reserve screenshots for one-off checks; use the WS for sustained
real-time control.
Dogfooding by driving a canvas game surfaced that per-action shell round-trips
(keydown; sleep; keyup) are the slowest, lowest-fidelity way to drive anything
timed — each is a process spawn + relay round-trip with ~250ms jitter, so a
'0.8s hold' is anything but.
- 'press <key> --hold <ms>': keyDown, wait, keyUp all inside the daemon, so the
hold duration is precise and it's one round-trip. For games (hold-to-move/
charge) and any press-and-hold.
- Documented the real driving pattern in the core skill + --help: script a timed
sequence in ONE round-trip with 'batch "press d --hold 900" "press j" "wait 200"'
(batch sends each step to the running daemon; --hold/wait block in-daemon), and
prefer reading engine state via main-world 'eval' over guessing from pixels.
Parser test covers plain/held/missing-duration. Builds on the keydown/keyup full
descriptor fix.
claude-in-chrome completes the Rakuten cart→購入手続き→checkout flow that
chrome-use 1.2.3 couldn't, because it binds to the browser-level tabId (survives
renderer-process swaps) rather than a CDP target/sessionId (torn down by the
cross-origin OAuth/SSO nav). chrome-use's relay is already keyed to the stable
tabId (cb-tab-<tabId>, #17) and sends commands by {tabId} — the gap was purely
that the extension treated the session→tab map as the source of truth and only
reactively re-attached after a failed lookup.
Make the tabId the PRIMARY resolution path: derive it straight from the session
id (tabIdFromSession), ensure-attach with short retries across the swap window
(recoverSessionTab now loops), and route every send through sendCdpToTab, which
on a detached-style error drops the stale handle, re-attaches the stable tab, and
retries once. So a cross-process nav never surfaces as a hard error — there's no
'session gone' window, matching claude-in-chrome. Builds on 0.4.5/0.4.6 reattach;
makes it primary + bulletproof rather than a fallback.
`keydown`/`keyup` dispatched a minimal Input.dispatchKeyEvent carrying only
{key}, so games/handlers that read event.code ("KeyD", "ArrowRight") or
event.keyCode saw nothing — a held key set no movement flag and the player
barely moved (dogfood: Dead Cell). They now build the same descriptor `press`
uses (key + code + windows/nativeVirtualKeyCode + printable text on down) via a
shared interaction::dispatch_single_key. Verified live: holding a direction now
drives continuous movement (player ran into an enemy and took damage), where
before it nudged ~80px.
The upgrade story spanned four parts (CLI, daemon, extension, skill) with no
single view and — worst — the extension was a total black box: nothing reported
which build was live, so a user could sit on a stale extension with zero signal.
- ext (0.4.7): on connect the extension sends a `hello` with
chrome.runtime.getManifest().version; the native-messaging host records it to a
`relay-ext-version` sidecar (next to relay-cdp-url, removed on exit).
- build.rs embeds the shipped extension version (AB_CONNECT_VERSION, read from the
ext manifest at compile time) so the CLI knows what extension it expects.
- `chrome-use doctor` gains a Versions section: CLI (vs the cached latest from the
background update check), extension (connected version vs the bundled expected —
warns + tells you to reload it in Chrome if behind), and skill (bundled, version-
locked; `skills add` copies may be stale). Daemon coherence was already covered.
So 'which of the four parts is on what version, and what needs upgrading' is now
one command. Verified: doctor warns on a simulated old extension and passes on a
current one; gracefully shows 'not connected / predates reporting' when the host
hasn't learned a version yet.
Dogfooding a canvas game over the extension relay surfaced three issues:
1. (serious) A fresh relay session's first `open` navigated one of the USER's
existing tabs instead of opening its own — in testing it replaced a
half-filled form with the target site. On connect the daemon passively
attaches to the user's tabs and pinned one as active; navigate() then drove
it. Now: on the relay (agent_group set), if the active tab isn't one this
session created, navigate() opens its own tab in the session's group first.
Off the relay (a browser we launched) reusing the active tab stays correct.
Pure helper active_index_is_owned() + regression tests.
2. (discoverability) `keydown <key>` / `keyup <key>` (hold-to-move, essential
for games/shortcuts) already existed as commands+daemon handlers but were
absent from --help and the skill, so they were undiscoverable. Documented in
--help, the core skill, and the canvas-app hint.
3. (UX) Canvas/WebGL pages expose almost no a11y tree, so `snapshot` is empty
and agents get stuck hunting refs. snapshot now detects a viewport-dominating
canvas with a sparse tree and prints a hint pointing at the screenshot +
coordinate-click + keydown/keyup path.
Verified live over the relay: `open` now lands the game in its own new tab with
the user's tabs (incl. the Rakuten recovery form) untouched; the canvas hint
fires on the game page; `close` cleans up only the session's own tab.
Multi-session over one relayed Chrome had a tab-identity fracture: each daemon
discovered targets ONCE at connect and assigned its own t<N> indices, so a tab
filled in session A was unreachable from session B — B saw a disjoint/blank set
and rebinding via 'open' piled up duplicate tabs. A stranded, still-filled tab
could not be finished from any other session.
- 'tab list' now re-syncs the live target set on every call: adopts tabs other
sessions opened (or that re-attached after a cross-process nav), drops gone
ones (clears phantom rows), and refreshes url/title from each live tab via
Target.getTargetInfo (the relay only stamps target_info on attach, so it goes
stale/blank after navigation — which made rows indistinguishable).
- 'tab list --full' now prints each tab's stable CDP targetId. Unlike t<N>
(per-session, reassigned each connect), targetId is stable across every session
on the relayed Chrome.
- 'tab <targetId>' adopts a specific pre-existing tab — including another
session's — WITHOUT reloading, so a half-filled form survives. handle_tab_switch
resyncs first, then resolves a raw targetId before falling back to t<N>/label.
- 'open <url> --reuse-tab' (alias --reuse) switches to an existing tab already on
that URL (matched by origin+path, ignoring volatile query/fragment) instead of
spawning a duplicate.
Verified live over the extension relay: a fresh session's 'tab list --full' lists
the user's real tabs with correct titles + full URLs + targetIds, and
'tab <targetId>' lands on and reads the exact stranded Rakuten account-recovery
form from the report. Unit tests cover URL normalization + --reuse-tab parsing;
full suite green. Docs: --help Tabs section + core skill multi-session guidance.
When a tab navigates across render processes (e.g. an SSO redirect to another
origin like login.account.rakuten.com), the debugger handle detaches and the
session drops out of the relay maps, so the next command dead-ends with
'stale sessionId ... its tab is gone' — even open/navigate, which should always
be able to drive the tab. But cb-tab-<tabId> encodes the STABLE Chrome tabId
(#17), and the tab itself usually survives the nav.
So before throwing, recoverSessionTab() parses the tabId out of the session,
checks the tab still exists + is eligible, and re-attaches (attachTab re-mints
the identical cb-tab-<tabId> session, keeping the daemon's binding valid), then
the in-flight command retries against the recovered tab. Complements the 0.4.5
onDetach proactive re-attach: that heals on the detach event, this heals lazily
on the next command if the event was missed. Falls back to the original error
only when the tab is genuinely gone (closed/restricted).
A mid-session 'chrome-use upgrade' (or a crashed worker) can leave per-session
daemons holding stale/cross-leaked tab handles, and the only fix was hunting
PIDs with pgrep/kill. Add a first-class command:
- 'daemon restart' kills every session daemon worker (SIGTERM→SIGKILL +
sidecar cleanup) but leaves the Chrome-launched __nm-host bridge alone, so
the extension relay stays up — the next command spins a fresh, clean daemon
against the same live Chrome. Closes no tabs.
- 'daemon status' lists running session daemons (pid + version) and relay state.
Wires connection::restart_all_daemons(), skips the command in the update-notify
nag, documents it in --help and the core skill. Unit tests cover the empty case
and a live-session kill (spawns a real child, asserts it's reaped + sidecars
cleaned). Issue #20.
v1.2.3's stable per-tab session id (#17) fixed sessionId STABILITY, but nothing
re-attached when an origin swaps the render process (e.g. the
login.account.rakuten.com SSO redirect — full-page nav + OOPIF). chrome.debugger
detached, the tab survived, but only onUpdated('complete') could re-attach — and
for that flow it didn't, so the session went permanently stale (even
open/navigate failed, retries didn't recover).
onDetach now proactively re-attaches the surviving tab (retry w/ backoff for the
swapped-in process to settle; skips user/DevTools-initiated detaches), so the
stable cb-tab-<tabId> session is restored and commands self-heal. Extension
0.4.4 → 0.4.5; needs a Web Store republish + dogfood on the Rakuten flow.
The changelog step lived in the separate `release` job (needs: build), which
had no checkout — so git ran with no repo ('fatal: not a git repository') and
the body came out empty. Add a fetch-depth:0 checkout to that job; drop the
now-pointless fetch-depth:0 from the build job.
Two SPA-SSO debugging gaps:
- The core skill referenced `bringToFront` but the CLI parser never mapped it
(the daemon handler existed) → 'Unknown command'. Wire up
bringToFront / bring-to-front / bringtofront → the existing action.
- 'stale sessionId — re-open your target URL' recovery was impossible because
`tab list` truncates long URLs with '…', cutting client_id/state out of SSO
links. Add `tab list --full` (also `tab --full`) to print untruncated URLs;
SKILL.md documents the recovery (full URL + re-open the stable entry URL).
The stale-session itself auto-recovers via the stable per-tab relay session id
(#17, extension 0.4.4). Parse tests for both new forms; verified live.
v1.2.2's auto-changelog came out empty: in a detached-HEAD tag checkout the
tag refs git describe/git log need aren't reliably present even with
fetch-depth:0. Fetch them explicitly first.
Release binaries / Build macOS ARM64 (push) Has been cancelled
Release binaries / Build macOS x64 (push) Has been cancelled
Release binaries / Build Linux ARM64 (push) Has been cancelled
Release binaries / Build Linux musl ARM64 (push) Has been cancelled
Release binaries / Build Linux musl x64 (push) Has been cancelled
Release binaries / Build Linux x64 (push) Has been cancelled
Release binaries / Build Windows x64 (push) Has been cancelled
Release binaries / Attach binaries to GitHub Release (push) Has been cancelled
- feat(cli): non-blocking update-available notice (stderr, once/day, opt-out) so users learn to upgrade
- ci(release): auto-generate changelog from commit log on every release
leeguooooo