oimwiodev/chrome-use
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

feat: enhance snapshot usability by reducing AI cognitive load of semantic noise and -C flag (#968)

Browse Source 
* fix: add ref for cursor-interactive content roles

* fix: format

* feat: always include cursor-interactive elements in snapshot, -C is deprecated

* feat: process StaticText aggregation and deduplication

* update test

* clean up

* fix: escape text of elements in snapshot

* fix: redundant slicing

* fix: cargo fmt

* feat: deduplicate redundant StaticText

---------

Co-authored-by: 羲洋 <lipengyang.lpy@alibaba-inc.com>
This commit is contained in:
Lppy
2026-03-23 09:35:43 -05:00
committed by GitHub
co-authored by 羲洋
parent 9c0955ca99
commit b48c3e9dd2
11 changed files with 160 additions and 116 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
-4
View File
@@ -511,7 +511,6 @@ The `snapshot` command supports filtering to reduce output size:
```bash
agent-browser snapshot # Full accessibility tree
agent-browser snapshot -i # Interactive elements only (buttons, inputs, links)
agent-browser snapshot -i -C # Include cursor-interactive elements (divs with onclick, etc.)
agent-browser snapshot -c # Compact (remove empty structural elements)
agent-browser snapshot -d 3 # Limit depth to 3 levels
agent-browser snapshot -s "#main" # Scope to CSS selector
@@ -521,13 +520,10 @@ agent-browser snapshot -i -c -d 5 # Combine options
| Option | Description |
| ---------------------- | ----------------------------------------------------------------------- |
| `-i, --interactive` | Only show interactive elements (buttons, links, inputs) |
| `-C, --cursor` | Include cursor-interactive elements (cursor:pointer, onclick, tabindex) |
| `-c, --compact` | Remove empty structural elements |
| `-d, --depth <n>` | Limit tree depth |
| `-s, --selector <sel>` | Scope to CSS selector |
The `-C` flag is useful for modern web apps that use custom clickable elements (divs, spans) instead of standard buttons/links.
## Annotated Screenshots
The `--annotate` flag overlays numbered labels on interactive elements in the screenshot. Each label `[N]` corresponds to ref `@eN`, so the same refs work for both visual and text-based workflows.
cli/src/commands.rs
+1
View File
@@ -533,6 +533,7 @@ pub fn parse_command(args: &[String], flags: &Flags) -> Result<Value, ParseError
obj.insert("compact".to_string(), json!(true));
}
"-C" | "--cursor" => {
// deprecated, cursor-interactive elements are referred by default now
obj.insert("cursor".to_string(), json!(true));
}
"-d" | "--depth" => {
cli/src/native/actions.rs
-2
View File
@@ -1734,7 +1734,6 @@ async fn handle_snapshot(cmd: &Value, state: &mut DaemonState) -> Result<Value,
.get("maxDepth")
.and_then(|v| v.as_u64())
.map(|d| d as usize),
cursor: cmd.get("cursor").and_then(|v| v.as_bool()).unwrap_or(false),
};
state.ref_map.clear();
@@ -1846,7 +1845,6 @@ async fn handle_screenshot(cmd: &Value, state: &mut DaemonState) -> Result<Value
&session_id,
&SnapshotOptions {
interactive: true,
cursor: true,
..SnapshotOptions::default()
},
&mut state.ref_map,
cli/src/native/e2e_tests.rs
+16 -21
View File
@@ -2274,12 +2274,12 @@ async fn e2e_material_checkbox_check_uncheck() {
// ---------------------------------------------------------------------------
// Issue #841 – snapshot -C and screenshot --annotate must not hang over WSS
// (PS: -C is deprecated, cursor-interactive elements are referred by default now)
// ---------------------------------------------------------------------------
/// Verifies that `snapshot -C` (cursor-interactive mode) detects elements with
/// cursor:pointer / onclick / tabindex, produces the correct v0.19.0-compatible
/// output format, deduplicates against the ARIA tree, and completes in bounded
/// time (no sequential CDP round-trip explosion).
/// Verifies that `snapshot` detects elements with cursor:pointer / onclick / tabindex,
/// produces the correct v0.19.0-compatible output format, deduplicates against the ARIA
/// tree, and completes in bounded time (no sequential CDP round-trip explosion).
#[tokio::test]
#[ignore]
async fn e2e_snapshot_cursor_interactive() {
@@ -2293,7 +2293,7 @@ async fn e2e_snapshot_cursor_interactive() {
assert_success(&resp);
// Page with:
// - <button> and <a> (standard interactive – ARIA tree, NOT in cursor section)
// - <button> and <a> (standard interactive – ARIA tree)
// - <div cursor:pointer onclick> (clickable – cursor section)
// - <div tabindex=0> (focusable – cursor section)
// - <span cursor:pointer> (clickable – cursor section)
@@ -2316,10 +2316,10 @@ async fn e2e_snapshot_cursor_interactive() {
.await;
assert_success(&resp);
// snapshot -i -C: interactive tree + cursor section
// snapshot -i: interactive tree
let start = std::time::Instant::now();
let resp = execute_command(
&json!({ "id": "3", "action": "snapshot", "interactive": true, "cursor": true }),
&json!({ "id": "3", "action": "snapshot", "interactive": true }),
&mut state,
)
.await;
@@ -2366,7 +2366,7 @@ async fn e2e_snapshot_cursor_interactive() {
// Must complete quickly (< 5s), not hit the 30s CDP timeout
assert!(
elapsed.as_secs() < 5,
"snapshot -C took {:?}, expected < 5s (Issue #841 regression)",
"snapshot took {:?}, expected < 5s (Issue #841 regression)",
elapsed,
);
@@ -2433,7 +2433,7 @@ async fn e2e_screenshot_annotate_many_elements() {
assert_success(&resp);
}
/// Verifies `snapshot -C` with many cursor-interactive elements completes in
/// Verifies `snapshot` with many cursor-interactive elements completes in
/// bounded time. Direct regression test for Issue #841's root cause: N×2
/// sequential CDP round-trips per cursor-interactive element.
#[tokio::test]
@@ -2468,7 +2468,7 @@ async fn e2e_snapshot_cursor_many_elements() {
let start = std::time::Instant::now();
let resp = execute_command(
&json!({ "id": "3", "action": "snapshot", "interactive": true, "cursor": true }),
&json!({ "id": "3", "action": "snapshot", "interactive": true }),
&mut state,
)
.await;
@@ -2492,7 +2492,7 @@ async fn e2e_snapshot_cursor_many_elements() {
// Must complete quickly
assert!(
elapsed.as_secs() < 10,
"snapshot -C with 100 cursor elements took {:?}, expected < 10s (Issue #841)",
"snapshot with 100 cursor elements took {:?}, expected < 10s (Issue #841)",
elapsed,
);
@@ -2504,7 +2504,7 @@ async fn e2e_snapshot_cursor_many_elements() {
/// the actual text content from parent elements.
#[tokio::test]
#[ignore]
async fn e2e_snapshot_inline_text_box_filtered() {
async fn e2e_snapshot_continuous_static_text() {
let mut state = DaemonState::new();
let resp = execute_command(
@@ -2514,7 +2514,7 @@ async fn e2e_snapshot_inline_text_box_filtered() {
.await;
assert_success(&resp);
// Simple HTML with text content that would generate InlineTextBox nodes
// Simple HTML with text content that would generate InlineTextBox nodes and sperate to multiple StaticText nodes
let html =
"data:text/html,<html><body><div><span>Hello</span> <span>World</span></div></body></html>";
@@ -2525,7 +2525,7 @@ async fn e2e_snapshot_inline_text_box_filtered() {
.await;
assert_success(&resp);
// Take snapshot to capture full output and verify InlineTextBox filtering
// Take snapshot to capture full output and verify InlineTextBox filtering and StaticText aggregation
let start = std::time::Instant::now();
let resp = execute_command(&json!({ "id": "3", "action": "snapshot" }), &mut state).await;
assert_success(&resp);
@@ -2542,13 +2542,8 @@ async fn e2e_snapshot_inline_text_box_filtered() {
// Verify that the actual text content is preserved
assert!(
snapshot_output.contains("Hello"),
"Snapshot should contain 'Hello': {}",
snapshot_output
);
assert!(
snapshot_output.contains("World"),
"Snapshot should contain 'World': {}",
snapshot_output.contains("Hello World"),
"Snapshot should contain 'Hello World': {}",
snapshot_output
);
cli/src/native/snapshot.rs
+141 -47
View File
@@ -65,13 +65,21 @@ const STRUCTURAL_ROLES: &[&str] = &[
"RootWebArea",
];
const INVISIBLE_CHARS: &[char] = &[
'\u{FEFF}', // BOM / Zero Width No-Break Space
'\u{200B}', // Zero Width Space
'\u{200C}', // Zero Width Non-Joiner
'\u{200D}', // Zero Width Joiner
'\u{2060}', // Word Joiner
'\u{00A0}', // Non-Breaking Space (&nbsp;)
];
#[derive(Default)]
pub struct SnapshotOptions {
pub selector: Option<String>,
pub interactive: bool,
pub compact: bool,
pub depth: Option<usize>,
pub cursor: bool,
}
struct TreeNode {
@@ -90,8 +98,51 @@ struct TreeNode {
has_ref: bool,
ref_id: Option<String>,
depth: usize,
/// Cursor-interactive information (only set when options.cursor is true)
cursor_info: Option<CursorElementInfo>,
cursor_info: Option<CursorElementInfo>, // cursor-interactive information
}
impl TreeNode {
// Create an empty node
fn empty() -> Self {
Self {
role: String::new(),
name: String::new(),
level: None,
checked: None,
expanded: None,
selected: None,
disabled: None,
required: None,
value_text: None,
backend_node_id: None,
children: Vec::new(),
parent_idx: None,
has_ref: false,
ref_id: None,
depth: 0,
cursor_info: None,
}
}
// Clear node content
fn clear(&mut self) {
self.role = String::new();
self.name = String::new();
self.level = None;
self.checked = None;
self.expanded = None;
self.selected = None;
self.disabled = None;
self.required = None;
self.value_text = None;
self.backend_node_id = None;
self.children.clear();
self.parent_idx = None;
self.has_ref = false;
self.ref_id = None;
self.depth = 0;
self.cursor_info = None;
}
}
/// Information about a cursor-interactive element (elements with cursor:pointer, onclick, tabindex, etc.)
@@ -263,30 +314,30 @@ pub async fn take_snapshot(
let mut nodes_with_refs: Vec<(usize, usize)> = Vec::new();
// When cursor mode is enabled, pre-collect cursor-interactive elements
// so we can mark them with refs during tree building
let cursor_elements: HashMap<i64, CursorElementInfo> = if options.cursor {
// Pre-collect cursor-interactive elements so we can mark them with refs during tree building
let cursor_elements: HashMap<i64, CursorElementInfo> =
find_cursor_interactive_elements(client, session_id)
.await
.unwrap_or_default()
} else {
HashMap::new()
};
.unwrap_or_default();
for (idx, node) in tree_nodes.iter().enumerate() {
let role = node.role.as_str();
let should_ref = if INTERACTIVE_ROLES.contains(&role) {
let mut should_ref = if INTERACTIVE_ROLES.contains(&role) {
true
} else if CONTENT_ROLES.contains(&role) {
!node.name.is_empty()
} else if options.cursor {
// In cursor mode, also ref elements that are cursor-interactive
node.backend_node_id
.is_some_and(|bid| cursor_elements.contains_key(&bid))
} else {
false
};
if node
.backend_node_id
.is_some_and(|bid| cursor_elements.contains_key(&bid))
{
// ref elements that are cursor-interactive
should_ref = true;
}
if should_ref {
let nth = tracker.track(role, &node.name, idx);
nodes_with_refs.push((idx, nth));
@@ -321,13 +372,11 @@ pub async fn take_snapshot(
tree_nodes[*idx].ref_id = Some(ref_id);
}
// Populate cursor_info for ref-bearing nodes when cursor mode is enabled
if options.cursor {
for (idx, _) in &nodes_with_refs {
if let Some(bid) = tree_nodes[*idx].backend_node_id {
if let Some(cursor_info) = cursor_elements.get(&bid) {
tree_nodes[*idx].cursor_info = Some((*cursor_info).clone());
}
// Populate cursor_info for ref-bearing nodes
for (idx, _) in &nodes_with_refs {
if let Some(bid) = tree_nodes[*idx].backend_node_id {
if let Some(cursor_info) = cursor_elements.get(&bid) {
tree_nodes[*idx].cursor_info = Some((*cursor_info).clone());
}
}
}
@@ -726,24 +775,7 @@ fn build_tree(nodes: &[AXNode]) -> (Vec<TreeNode>, Vec<usize>) {
extract_properties(&node.properties);
if (node.ignored.unwrap_or(false) && role != "RootWebArea") || role == "InlineTextBox" {
tree_nodes.push(TreeNode {
role: String::new(),
name: String::new(),
level: None,
checked: None,
expanded: None,
selected: None,
disabled: None,
required: None,
value_text: None,
backend_node_id: None,
children: Vec::new(),
parent_idx: None,
has_ref: false,
ref_id: None,
depth: 0,
cursor_info: None,
});
tree_nodes.push(TreeNode::empty());
id_to_idx.insert(node.node_id.clone(), i);
continue;
}
@@ -781,6 +813,58 @@ fn build_tree(nodes: &[AXNode]) -> (Vec<TreeNode>, Vec<usize>) {
}
}
// Process StaticText aggregation
for i in 0..tree_nodes.len() {
if tree_nodes[i].role.is_empty() || tree_nodes[i].children.is_empty() {
continue;
}
let children_indices: Vec<usize> = tree_nodes[i].children.clone();
// Continuous StaticText nodes at the same level are an artifact of HTML structure rather than semantic meaning.
// They typically represent a single continuous piece of text on the page that was split due to inline elements, formatting tags, or other structural reasons.
// Thus, continuous StaticText children are aggregated into the first one.
let mut start = 0;
while start < children_indices.len() {
// Skip non-StaticText nodes
if tree_nodes[children_indices[start]].role != "StaticText" {
start += 1;
continue;
}
// Find the end of the current StaticText sequence
let mut end = start + 1;
while end < children_indices.len()
&& tree_nodes[children_indices[end]].role == "StaticText"
{
end += 1;
}
// If we have a sequence of at least two StaticText
if end > start + 1 {
// Collect and aggregate all names from the sequence
let aggregated_name: String = (start..end)
.map(|idx| tree_nodes[children_indices[idx]].name.clone())
.collect();
// Always aggregate into the first node of the sequence
tree_nodes[children_indices[start]].name = aggregated_name;
// Clear the rest of the nodes in the sequence (from start+1 to end-1)
for j in (start + 1)..end {
tree_nodes[children_indices[j]].clear();
}
}
start = end;
}
// Deduplicate redundant StaticText
if children_indices.len() == 1
&& tree_nodes[children_indices[0]].role == "StaticText"
&& tree_nodes[i].name == tree_nodes[children_indices[0]].name
{
tree_nodes[children_indices[0]].clear();
}
}
// Set depths
let mut root_indices = Vec::new();
let children_exist: Vec<bool> = nodes.iter().map(|_| false).collect();
@@ -820,7 +904,11 @@ fn render_tree(
) {
let node = &nodes[idx];
if node.role.is_empty() {
// Reduce unnecessary indentation and rendering
if node.role.is_empty()
|| (node.role == "generic" && !node.has_ref && node.children.len() <= 1)
|| (node.role == "StaticText" && node.name.replace(INVISIBLE_CHARS, "").is_empty())
{
// Ignored node -- still render children
for &child in &node.children {
render_tree(nodes, child, indent, output, options);
@@ -855,16 +943,22 @@ fn render_tree(
let prefix = " ".repeat(indent);
let mut line = format!("{}- {}", prefix, role);
// Use ARIA name if available, otherwise fall back to cursor-interactive textContent
let display_name = if !node.name.is_empty() {
// Use ARIA name if available, only fall back to cursor-interactive textContent in interactive mode since their visible text in child nodes is filtered out
let unescaped_display_name = if !node.name.is_empty() {
&node.name
} else if let Some(ref ci) = node.cursor_info {
&ci.text
} else if options.interactive {
if let Some(ref ci) = node.cursor_info {
&ci.text
} else {
&node.name
}
} else {
&node.name
};
if !display_name.is_empty() {
line.push_str(&format!(" \"{}\"", display_name));
if !unescaped_display_name.is_empty() {
if let Ok(display_name) = serde_json::to_string(&unescaped_display_name) {
line.push_str(&format!(" {}", display_name.replace(INVISIBLE_CHARS, "")));
}
}
// Properties
cli/src/output.rs
-2
View File
@@ -1459,7 +1459,6 @@ Designed for AI agents to understand page structure.
Options:
-i, --interactive Only include interactive elements
-C, --cursor Include cursor-interactive elements (cursor:pointer, onclick, tabindex)
-c, --compact Remove empty structural elements
-d, --depth <n> Limit tree depth
-s, --selector <sel> Scope snapshot to CSS selector
@@ -1471,7 +1470,6 @@ Global Options:
Examples:
agent-browser snapshot
agent-browser snapshot -i
agent-browser snapshot -i -C # Interactive + cursor-interactive elements
agent-browser snapshot --compact --depth 5
agent-browser snapshot -s "#main-content"
"##
docs/src/app/snapshots/page.mdx
-21
View File
@@ -9,7 +9,6 @@ Filter output to reduce size:
```bash
agent-browser snapshot # Full accessibility tree
agent-browser snapshot -i # Interactive elements only (recommended)
agent-browser snapshot -i -C # Include cursor-interactive elements
agent-browser snapshot -c # Compact (remove empty elements)
agent-browser snapshot -d 3 # Limit depth to 3 levels
agent-browser snapshot -s "#main" # Scope to CSS selector
@@ -22,32 +21,12 @@ agent-browser snapshot -i -c -d 5 # Combine options
</thead>
<tbody>
<tr><td><code>-i, --interactive</code></td><td>Only interactive elements (buttons, links, inputs)</td></tr>
<tr><td><code>-C, --cursor</code></td><td>Include cursor-interactive elements (cursor:pointer, onclick, tabindex)</td></tr>
<tr><td><code>-c, --compact</code></td><td>Remove empty structural elements</td></tr>
<tr><td><code>-d, --depth</code></td><td>Limit tree depth</td></tr>
<tr><td><code>-s, --selector</code></td><td>Scope to CSS selector</td></tr>
</tbody>
</table>
## Cursor-interactive elements
Many modern web apps use custom clickable elements (divs, spans) instead of standard buttons or links.
The `-C` flag detects these by looking for:
- `cursor: pointer` CSS style
- `onclick` attribute or handler
- `tabindex` attribute (keyboard focusable)
```bash
agent-browser snapshot -i -C
# Output includes:
# @e1 [button] "Submit"
# @e2 [link] "Learn more"
# Cursor-interactive elements:
# @e3 [clickable] "Menu Item" [cursor:pointer, onclick]
# @e4 [clickable] "Card" [cursor:pointer]
```
## Output format
The default text output is compact and AI-friendly:
skills/agent-browser/SKILL.md
-1
View File
@@ -113,7 +113,6 @@ agent-browser close # Close browser
# Snapshot
agent-browser snapshot -i # Interactive elements with refs (recommended)
agent-browser snapshot -i -C # Include cursor-interactive elements (divs with onclick, cursor:pointer)
agent-browser snapshot -s "#selector" # Scope to CSS selector
# Interaction (use @refs from snapshot)
skills/electron/SKILL.md
-1
View File
@@ -217,7 +217,6 @@ AGENT_BROWSER_COLOR_SCHEME=dark agent-browser connect 9222
### Elements not appearing in snapshot
- The app may use multiple webviews. Use `agent-browser tab` to list targets and switch to the right one
- Use `agent-browser snapshot -i -C` to include cursor-interactive elements (divs with onclick handlers)
### Cannot type in input fields
skills/slack/SKILL.md
-9
View File
@@ -235,15 +235,6 @@ agent-browser console
agent-browser errors
```
### View raw HTML of an element
```bash
# Snapshot shows the accessibility tree. If an element isn't there,
# it may not be interactive (e.g., div instead of button)
# Use snapshot -i -C to include cursor-interactive divs
agent-browser snapshot -i -C
```
### Get current page state
```bash
skills/slack/references/slack-tasks.md
+2 -8
View File
@@ -334,19 +334,13 @@ If you can't find an element:
agent-browser snapshot -i
```
3. **Try snapshot with extended range**
```bash
# Include cursor-interactive elements (divs with onclick handlers)
agent-browser snapshot -i -C
```
4. **Check current URL**
3. **Check current URL**
```bash
agent-browser get url
# Verify you're in the right section
```
5. **Wait for page to load**
4. **Wait for page to load**
```bash
agent-browser wait --load networkidle
agent-browser wait 1000