feat: rich-editor fill, box centers, screenshot downscale, disabled+docs (#41-#45)
Release binaries / Build macOS ARM64 (push) Has been cancelled
Release binaries / Build macOS x64 (push) Has been cancelled
Release binaries / Build Linux ARM64 (push) Has been cancelled
Release binaries / Build Linux musl ARM64 (push) Has been cancelled
Release binaries / Build Linux musl x64 (push) Has been cancelled
Release binaries / Build Linux x64 (push) Has been cancelled
Release binaries / Build Windows x64 (push) Has been cancelled
Release binaries / Attach binaries to GitHub Release (push) Has been cancelled

Dogfooding backlog from this session's embedded-form/editor work.

#41 fill on rich editors: detect CodeMirror 5 / Monaco / ProseMirror /
contenteditable and set via their own API or execCommand('insertText') so
beforeinput/input fire (a raw .value/textContent write no-op'd juejin's
CodeMirror and skipped React composers). Response echoes the `engine` used.
`fill <sel> --file <path>` / `--stdin` set large multiline text without
shell-escaping. `get value` now reads CodeMirror/Monaco/contenteditable too.

#42 screenshot --max-width/--max-height/--scale, plus a default 2000px
longest-edge cap (AGENT_BROWSER_SCREENSHOT_MAX_EDGE; 0 disables) so retina
full-page shots fit an agent's image reader and --scale 0.5 makes screenshot
px line up with click px. Annotated shots are never downscaled.

#43 `box @ref` (already a top-level alias of `get box`) now also returns
centerX/centerY/inViewport in CSS px — feed straight into `click x y` when a
ref-click no-ops (e.g. a button in a cross-origin iframe).

#44 no code change needed — disabled elements already list as
`button "Save" [disabled, ref=eN]`; the reporter's missing button was
DOM-gated on validity. Added a skill note: `find text` can't reach into a
cross-origin iframe — target those by snapshot @ref.

#45 core skill now distinguishes screenshot-to-locate (discouraged) from
screenshot-to-capture a reusable image asset via `screenshot [--clip] <file>`
(encouraged), so agents stop over-reading the prohibition.

#40 (group-scoped relay) stays deferred — needs an ab-connect extension change.

Verified live: fill --file round-trips multiline+CJK+backticks; contenteditable
engine=contenteditable + get value reads it back; box gives centerX/centerY/
inViewport; screenshot of retina example.com → 2000px; disabled button shows
[disabled]. 856 tests pass.
This commit is contained in:
leeguooooo
2026-06-17 17:49:38 +09:00
parent 50b27ac0e0
commit a83d1b1df9
9 changed files with 315 additions and 33 deletions
+18
View File
@@ -59,6 +59,18 @@ next ref interaction.
> anyway.) Driving off pixels on the relay also risks a coordinate event drifting
> onto the user's foreground tab — refs never do. See issue #37.
> **Two different intents — only one is discouraged.** The rule above is about
> *screenshot-to-locate* (using a picture to find/hit an element) — that's the bug.
> *screenshot-to-capture* — saving a region or element to a file as a **reusable
> image asset** (maps, charts, og-images, visual-diff baselines, report figures) —
> is fully supported and encouraged: `screenshot [selector] [--clip x,y,w,h] <file>`.
> Capturing a rendered map region to a PNG for a blog post is the right tool, not a
> smell. Screenshots are auto-downscaled to ≤2000px (longest edge) so they fit an
> image reader and their pixels line up with `click x y`; override with
> `--max-width`/`--max-height`/`--scale`. To click something you couldn't hit by
> ref, `box @ref` gives the element's CSS-px box + `centerX/centerY` to feed
> straight into `click <centerX> <centerY>` — no screenshot needed.
## Before you automate: pick the cheapest tool
Driving a browser is the heavy option. chrome-use earns its keep when you
@@ -373,6 +385,12 @@ foreground, so prefer refs. For below-the-fold content in such a frame, scroll i
with `scroll down N --at x,y` (a pixel over the frame) or `--frame n`. For a
postal/autocomplete box inside the frame, `type @e "…" --key-events`.
> **Caveat: `find text "…"` can't reach into a cross-origin iframe** — it errors
> "Element not found" even though `snapshot -i` lists those nodes and
> `get text` reads them. Inside cross-origin iframes, target elements by their
> **snapshot `@ref`**, not by `find`. (`box @ref` also works on iframe refs when
> you need a coordinate fallback.)
### When refs don't work or you don't want to snapshot
Use semantic locators: