Chrome Remote Browser Guide

Guide for AI agents on how to use Chrome Remote Debugging (CDP on port 9222) to automate browser interactions. Covers the full lifecycle — connecting, navigating, taking screenshots, reading page structure, clicking/typing/scrolling, executing JavaScript, and handling common pitfalls. Use this when an AI agent needs to interact with web pages in Chrome, especially login-gated or JavaScript-rendered pages that web_fetch cannot handle.

GuangxianLiu@lgx-00

Install

openclaw skills install @lgx-00/chrome-remote-browser

Chrome Remote Browser — AI Agent Usage Guide

A comprehensive guide for AI agents on how to control Chrome via Remote Debugging Protocol (CDP) on port 9222. This enables interaction with authenticated sessions (Google, GitHub, dashboards, etc.) without needing separate login credentials.

Overview

What is Chrome Remote Debugging?

Chrome can expose a debugging interface on a local TCP port (default: 9222). When enabled, external tools can:

Open/close tabs
Navigate to URLs
Read page content (DOM, accessibility tree)
Take screenshots
Click buttons, fill forms, scroll
Execute JavaScript

When to Use the Browser Tool

Scenario	Use
Public facts, search queries	`web_search` (simpler, faster)
Fetching public page content	`web_fetch` (lightweight, no browser needed)
Login-gated pages (Google Console, GitHub, dashboards)	`browser`
JavaScript-rendered content (SPAs)	`browser`
Visual inspection (charts, layouts, screenshots)	`browser`
Interactive tasks (clicking, form filling, multi-step flows)	`browser`

Rule of thumb: Try web_fetch first for public pages. Use browser when you need authentication, JavaScript rendering, or interaction.

Prerequisites

Chrome must be running with remote debugging enabled:

text

--remote-debugging-port=9222

The user should see something like:

text

Server running at: 127.0.0.1:9222

Core Workflow

Every browser automation task follows this cycle:

text

start → open(url) → screenshot + snapshot → act/execute_js → screenshot + snapshot → ... → done

Golden rule: Always take a screenshot and/or snapshot before and after every action. This is how you "see" and "understand" the page.

API Reference

1. `status` — Check Connection

Check if the browser is connected and list open tabs.

json

{"action": "status"}

Returns: Connection status and list of currently open tabs with their targetIds.

2. `open` — Open a URL

Navigate to a URL in a new tab.

json

{"action": "open", "targetUrl": "https://example.com"}

Returns: A targetId (string) that you use for all subsequent actions on this tab.

IMPORTANT: Save the targetId — you need it for every other action.

3. `screenshot` — Capture Visual State

Take a screenshot of the page. This is how you "see" the page.

json

{"action": "screenshot", "targetId": "<targetId>"}

Optionally save to a file:

json

{"action": "screenshot", "targetId": "<targetId>", "savePath": "/path/to/output.png"}

Returns: A CDN URL of the screenshot image that you can view.

When to screenshot:

After opening a page (to see its initial state)
After clicking a button (to see the result)
After scrolling (to see new content)
When you need to read visual data (charts, images, layouts)
Before reporting results to the user

4. `snapshot` — Read Page Structure (Accessibility Tree)

Get a text representation of the page's DOM structure. Interactive elements are assigned reference IDs (ref).

json

{"action": "snapshot", "targetId": "<targetId>"}

Returns: An accessibility tree like:

text

[ref=e1] link "Home"
[ref=e2] button "Sign In"
[ref=e3] textbox "Email address"
[ref=e4] textbox "Password"
[ref=e5] button "Submit"
heading "Welcome to Example"
text "Some content here..."
[ref=e6] link "Learn more"

Key concept: The ref values (e.g., e1, e2, e3) are your handles for interacting with elements. You use these in act commands.

When to snapshot:

After opening a page (to find interactive elements)
After any page state change (new elements may appear)
When you need to find a specific button, link, or form field
Before clicking (to get the correct ref for the target element)

5. `act` — Interact with Page Elements

Perform actions on elements identified by their ref from a snapshot.

Click

json

{"action": "act", "targetId": "<targetId>", "request": {"kind": "click", "ref": "e2"}}

Type (into a focused/clicked element)

json

{"action": "act", "targetId": "<targetId>", "request": {"kind": "type", "ref": "e3", "text": "hello@example.com"}}

With Enter key submission:

json

{"action": "act", "targetId": "<targetId>", "request": {"kind": "type", "ref": "e3", "text": "search query", "submit": true}}

Fill (multiple fields at once)

json

{"action": "act", "targetId": "<targetId>", "request": {"kind": "fill", "fields": [
  {"ref": "e3", "value": "hello@example.com"},
  {"ref": "e4", "value": "password123"}
]}}

Scroll

json

{"action": "act", "targetId": "<targetId>", "request": {"kind": "scroll", "direction": "down", "amount": 500}}

Directions: "up", "down", "left", "right"

Press a key

json

{"action": "act", "targetId": "<targetId>", "request": {"kind": "press", "key": "Enter"}}

Common keys: "Enter", "Tab", "Escape", "ArrowDown", "ArrowUp"

Wait

json

{"action": "act", "targetId": "<targetId>", "request": {"kind": "wait", "timeout": 3000}}

Wait for a specified time in milliseconds. Use after navigation or actions that trigger page loads.

6. `execute_js` — Run JavaScript

Execute arbitrary JavaScript in the page context. Extremely powerful for extracting data that isn't visible in the accessibility tree.

json

{"action": "execute_js", "targetId": "<targetId>", "script": "document.title"}

Returns: The result of the JavaScript expression.

Common use cases:

Extract text content:

javascript

document.querySelector('#result-stats').textContent

Extract data from tables:

javascript

JSON.stringify(Array.from(document.querySelectorAll('table tr')).map(r => Array.from(r.cells).map(c => c.textContent.trim())))

Get all links:

javascript

JSON.stringify(Array.from(document.querySelectorAll('a[href]')).map(a => ({text: a.textContent.trim(), href: a.href})))

Scroll to bottom:

javascript

window.scrollTo(0, document.body.scrollHeight)

Extract chart/SVG data:

javascript

JSON.stringify(Array.from(document.querySelectorAll('[aria-label]')).map(el => el.getAttribute('aria-label')))

7. `navigate` — Navigate in Current Tab

Navigate the current tab to a new URL (without opening a new tab).

json

{"action": "navigate", "targetId": "<targetId>", "url": "https://example.com/page2"}

8. `tabs` — List Open Tabs

json

{"action": "tabs"}

9. `close` — Close a Tab

json

{"action": "close", "targetId": "<targetId>"}

Step-by-Step Example: Filling a Google Form

Here's a complete example workflow:

text

Step 1: Open the page
  → browser.open("https://console.cloud.google.com/...")
  → Get targetId: "ABCD1234"

Step 2: See what's on the page
  → browser.screenshot("ABCD1234")
  → See: A form with various fields
  → browser.snapshot("ABCD1234")
  → See: [ref=e1] textbox "Name" / [ref=e2] dropdown "Type" / [ref=e5] button "Create"

Step 3: Fill in the name
  → browser.act("ABCD1234", {kind: "click", ref: "e1"})
  → browser.act("ABCD1234", {kind: "type", ref: "e1", text: "My Application"})

Step 4: Select from dropdown
  → browser.act("ABCD1234", {kind: "click", ref: "e2"})
  → browser.snapshot("ABCD1234")  ← MUST re-snapshot to see dropdown options!
  → See: [ref=e10] option "Web application" / [ref=e11] option "Desktop app"
  → browser.act("ABCD1234", {kind: "click", ref: "e10"})

Step 5: Submit
  → browser.act("ABCD1234", {kind: "click", ref: "e5"})
  → browser.screenshot("ABCD1234")  ← Verify the result

Critical Rules & Best Practices

Rule 1: ALWAYS Snapshot Before Acting

text

❌ BAD:  open → click e5 (from memory or assumption)
✅ GOOD: open → snapshot → find ref → click ref → snapshot (verify)

The ref IDs change every time the page state changes. Never reuse refs from a previous snapshot after any navigation or interaction.

Rule 2: ALWAYS Screenshot After State Changes

After clicking, submitting, or navigating, always take a screenshot to verify the result before proceeding. Things that can go wrong:

The click didn't register
A loading spinner appeared
An error dialog popped up
The page navigated somewhere unexpected

Rule 3: Re-snapshot After EVERY Page Change

When you click a button, open a dropdown, or navigate — the DOM changes. Old ref IDs become invalid. You MUST take a new snapshot to get fresh refs.

text

click dropdown → snapshot (get new refs for options) → click option

Rule 4: Handle Page Load Delays

Many pages (especially SPAs like Google Cloud Console) load content asynchronously. After navigating:

text

navigate → wait(2000-3000) → screenshot + snapshot

If the page still looks empty or loading, wait longer and retry.

Rule 5: Handle Google Account Switching

Google services may show a different account than expected. Check for account indicators:

Look for user avatar/email in the top-right corner
If wrong account, look for account switcher or use ?authuser=N URL parameter:
- authuser=0 = first Google account
- authuser=1 = second Google account
- Add to any Google URL: https://console.cloud.google.com/...?authuser=1

Rule 6: Element Not Found Errors

If you get "Element ref not found":

The page likely changed since your last snapshot
Take a fresh snapshot
Find the new ref for the element you want
Retry with the new ref

Rule 7: Don't Over-rely on Screenshots Alone

Screenshots show what the page looks like, but:

You can't click on screenshot pixels
You need snapshot to get interactive element refs
Use screenshots for verification, snapshots for interaction

Rule 8: Sequential, Not Parallel

Browser actions are inherently sequential — one tab, one state at a time. Do NOT try to parallelize browser operations on the same tab.

Rule 9: Minimize Actions

Each action has latency and potential for failure. Plan your workflow:

Use fill for multiple form fields instead of individual type calls
Use execute_js for data extraction instead of clicking through pagination
Use URL parameters instead of clicking through menus when possible

Rule 10: File Downloads

Chrome remote debugging cannot reliably intercept file downloads. If you need to download a file:

Try to get the download URL via execute_js and use web_fetch instead
Or note the download URL/path for the user to handle manually

Common Pitfalls

Problem	Cause	Solution
"Element e11 not found"	Page changed after snapshot	Re-snapshot, find new ref
Page looks blank after navigate	Content loading asynchronously	`wait(3000)` then screenshot
Wrong Google account	Multiple accounts logged in	Add `?authuser=N` to URL
CAPTCHA appeared	Too many automated actions	Slow down, add delays between actions
Dropdown options not visible	Haven't re-snapshotted after click	Click dropdown → snapshot → click option
Form submit didn't work	Button ref was stale	Re-snapshot, find fresh button ref, click
Can't find a button	Element is below fold / in scrollable area	Scroll down, then re-snapshot
Screenshot shows loading spinner	Page still loading	Wait 2-5 seconds, retry screenshot
JavaScript returns undefined	Selector doesn't match	Check the page source, try different selectors
Tab closed unexpectedly	Page navigated away (e.g., OAuth redirect)	List tabs, find the new tab/targetId

Advanced Patterns

Pattern: Extracting Data from Google Charts

Google's charts are often rendered as SVG/Canvas. To extract underlying data:

javascript

// Try aria-labels on chart elements
JSON.stringify(Array.from(document.querySelectorAll('[aria-label]')).map(el => {
  var label = el.getAttribute('aria-label');
  if (label && /\d/.test(label)) return label;
  return null;
}).filter(Boolean))

javascript

// Try embedded data scripts
JSON.stringify(Array.from(document.querySelectorAll('script')).filter(s => 
  s.textContent.includes('AF_initDataCallback')
).map(s => s.textContent.substring(0, 500)))

Pattern: Handling Multi-Page Workflows

For wizards/multi-step forms:

text

1. snapshot → fill step 1 fields → click "Next"
2. wait(2000) → snapshot → fill step 2 fields → click "Next"
3. wait(2000) → snapshot → verify summary → click "Submit"
4. wait(3000) → screenshot → verify confirmation

Pattern: Navigating Google Cloud Console

Google Cloud Console is a complex SPA. Tips:

Use direct URLs whenever possible instead of clicking through menus
Example: https://console.cloud.google.com/apis/library/searchconsole.googleapis.com?project=PROJECT_ID
Always include ?project=PROJECT_ID to avoid project-switching issues
Pages may take 3-5 seconds to load; always wait before interacting
The sidebar menu may cover content; close it if it's in the way

Pattern: Extracting Search Results Count

For Google Search (site:example.com):

javascript

document.querySelector('#result-stats')?.textContent || 'No result stats found'

Workflow Template

Use this template for any browser automation task:

text

1. PLAN: What do I need to accomplish? What URL do I start at?
2. OPEN: browser.open(url) → save targetId
3. OBSERVE: screenshot + snapshot → understand the page
4. ACT: Identify the right ref → perform action (click/type/fill)
5. VERIFY: screenshot + snapshot → did it work?
6. REPEAT: Steps 3-5 until task is complete
7. EXTRACT: Use execute_js or snapshot to collect results
8. REPORT: Present findings to user with screenshots as evidence

Quick Reference

Action	When to Use	Returns
`open(url)`	Start: open a page	`targetId`
`screenshot(targetId)`	See the page visually	CDN image URL
`snapshot(targetId)`	Read page structure, get element refs	Accessibility tree text
`act(click, ref)`	Click buttons, links, dropdowns	—
`act(type, ref, text)`	Type into input fields	—
`act(fill, fields)`	Fill multiple fields at once	—
`act(scroll, direction)`	Scroll the page	—
`act(press, key)`	Press keyboard keys	—
`act(wait, timeout)`	Wait for page loads	—
`execute_js(script)`	Extract data, run custom logic	Script result
`navigate(url)`	Go to new URL in same tab	—
`tabs()`	List open tabs	Tab list
`close(targetId)`	Close a tab	—

Chrome Remote Browser Guide

Install

Chrome Remote Browser — AI Agent Usage Guide

Overview

What is Chrome Remote Debugging?

When to Use the Browser Tool

Prerequisites

Core Workflow

API Reference

1. status — Check Connection

2. open — Open a URL

3. screenshot — Capture Visual State

4. snapshot — Read Page Structure (Accessibility Tree)

5. act — Interact with Page Elements

Click

Type (into a focused/clicked element)

Fill (multiple fields at once)

Scroll

Press a key

Wait

6. execute_js — Run JavaScript

7. navigate — Navigate in Current Tab

8. tabs — List Open Tabs

9. close — Close a Tab

Step-by-Step Example: Filling a Google Form

Critical Rules & Best Practices

Rule 1: ALWAYS Snapshot Before Acting

Rule 2: ALWAYS Screenshot After State Changes

Rule 3: Re-snapshot After EVERY Page Change

Rule 4: Handle Page Load Delays

Rule 5: Handle Google Account Switching

Rule 6: Element Not Found Errors

Rule 7: Don't Over-rely on Screenshots Alone

Rule 8: Sequential, Not Parallel

Rule 9: Minimize Actions

Rule 10: File Downloads

Common Pitfalls

Advanced Patterns

Pattern: Extracting Data from Google Charts

Pattern: Handling Multi-Page Workflows

Pattern: Navigating Google Cloud Console

Pattern: Extracting Search Results Count

Workflow Template

Quick Reference

Related skills

1. `status` — Check Connection

2. `open` — Open a URL

3. `screenshot` — Capture Visual State

4. `snapshot` — Read Page Structure (Accessibility Tree)

5. `act` — Interact with Page Elements

6. `execute_js` — Run JavaScript

7. `navigate` — Navigate in Current Tab

8. `tabs` — List Open Tabs

9. `close` — Close a Tab