Auto Scraping to CSV

Scrape any webpage using text-based DOM manipulation and export structured data to CSV. The agent handles complex page nuances — infinite scroll, pagination, popups, lazy loading — and asks clarifying questions when data is ambiguous. No external LLM needed.

Auto Scraping to CSV — Agent-Driven Web Scraping

Scrape any webpage using text-based DOM manipulation and export structured data to CSV. The agent handles complex page nuances — infinite scroll, pagination, popups, lazy loading — and asks clarifying questions when data is ambiguous. No external LLM required.

Philosophy: Let the Agent Figure It Out

Traditional scraping requires you to inspect HTML, write CSS selectors, handle edge cases, and debug when the site changes. This skill flips that:

You say what you want. The agent handles the how.

text

You: "Scrape product catalog"
Agent: "I see 50 products across 5 pages with infinite scroll. 
        I also see 'Price', 'Sale Price', and 'Member Price' columns.
        Which price field should I extract?"
You: "Sale Price"
Agent: [handles scrolling, pagination, extracts 50 rows] → products.csv

The agent will:

1. Explore the page structure via text-based DOM
2. Detect complexity — scroll, pagination, tabs, filters
3. Ask questions when ambiguous (multiple price fields, missing data, format choices)
4. Handle edge cases — dismiss popups, wait for lazy loading, retry on errors
5. Export CSV — clean, structured, ready to use

When to Use

• Product catalogs: "Scrape all laptops with prices and ratings"
• News/articles: "Get latest blog posts with titles, dates, authors"
• Directory listings: "Extract all company names, emails, and websites"
• Table data: "Get the pricing table from this SaaS page"
• Real estate: "Scrape listings with price, beds, baths, square footage"
• Job boards: "Get job titles, companies, locations, and salary ranges"
• Social feeds: "Extract posts with engagement counts (likes, comments, shares)"
• Research data: "Get citation counts, authors, publication dates from this index"

How It Works

text

Claude (Host Model)
    ↕  HTTP
Bridge Server (Node.js + Playwright)
    ↕  page.evaluate()
Browser (Chromium) ← Page-Agent injected

1. Bridge launches a local Chromium browser via Playwright
2. Page-Agent is injected as an IIFE script from CDN into the target page
3. Page-Agent indexes the DOM and generates a simplified text representation:

   text

   [5]<a >Widget Pro Laptop /></a>
   [12]<div >$1,299.99 /></div>
   [18]<div >4.5 stars (128 reviews) /></div>
   

4. Claude receives the text state, understands the page structure, decides actions
5. Agent asks questions when data is ambiguous or complex
6. Loop continues until all data is extracted or user says stop
7. CSV export converts structured JSON to CSV

---

First-Time Setup

1. Install Playwright

bash

npm install -D playwright
npx playwright install chromium

2. Place the Bridge Script

bash

# Copy from this skill folder to your agents directory
cp .claude/skills/auto-scraping-to-csv/page-agent-bridge.mjs .claude/agents/

3. Start the Bridge

bash

node .claude/agents/page-agent-bridge.mjs

Default port: 9876.

4. Verify Health

bash

curl http://localhost:9876/health

---

Agent-Driven Scraping Workflow

Step 1: Request What You Want

text

/scrape-to-csv https://example.com/products
  "Extract all products with name, price, rating, and number of reviews"

Step 2: Agent Explores and Asks Questions

The agent will fetch the DOM state and may ask:

text

Agent: "I found 24 products on the current page. The site has infinite scroll 
        and I can see more products loading as I scroll. 
        How many products do you want total? (type 'all' for everything)"
You: "50"

Agent: "I see three price fields:
        - Original Price: $1,299
        - Sale Price: $999
        - Member Price: $899
        Which one should I extract?"
You: "Sale Price"

Agent: "Some products are out of stock and show 'N/A' for price.
        Should I:
        A) Include them with 'N/A'
        B) Skip those rows
        C) Mark as 'Out of Stock' in a separate column"
You: "C"

Step 3: Agent Handles Complexity

The agent automatically:

• Scrolls to load more content
• Clicks pagination buttons
• Waits for lazy-loaded images/data
• Dismisses cookie banners and popups
• Retries if elements are slow to appear
• Stops gracefully if the site blocks scraping

Step 4: Preview and Confirm

text

Agent: "Extracted 50 products. Here's a preview:

| name | price | rating | reviews | stock_status |
|------|-------|--------|---------|-------------|
| Widget Pro | $999 | 4.5 | 128 | In Stock |
| Gadget Mini | $499 | 4.2 | 89 | In Stock |
| ... | ... | ... | ... | ... |

Save as 'products.csv'? (yes/no/change columns)"
You: "yes"

Step 5: CSV Delivered

text

Agent: "Saved to ./products.csv — 50 rows, 5 columns"

---

Handling Complex Scenarios

Infinite Scroll

text

You: "Scrape all posts from this feed"
Agent: "The page loads 10 posts at a time via infinite scroll.
        I've scrolled 3 times and found 30 posts so far.
        Should I continue scrolling? (yes/no/scroll N more times)"
You: "scroll 5 more times"
Agent: [scrolls] → "Found 80 total posts. Saved to feed.csv"

Pagination

text

You: "Get all products from this catalog"
Agent: "I see pagination: Page 1 of 12. 
        I can scrape all 12 pages (may take 2-3 minutes) or a subset.
        How many pages?"
You: "all"
Agent: [clicks through 12 pages] → "360 products saved to catalog.csv"

Login Required

text

You: "Scrape my order history"
Agent: "The page redirects to a login screen. 
        I need credentials to proceed.
        Please provide:
        - Username/email
        - Password
        Or type 'skip' to abort."
You: "skip"
Agent: "Aborted. No data extracted."

Multiple Data Formats

text

You: "Get prices from this page"
Agent: "I found prices in multiple formats:
        - '$19.99' (US format)
        - '€17.99' (EU format)
        - '¥2,980' (JP format)
        Which format should I standardize to?"
You: "Keep as-is, add a 'currency' column"
Agent: [extracts with currency detection] → prices.csv

Missing / Inconsistent Data

text

You: "Extract employee data from this directory"
Agent: "I see 100 employee cards. Some are missing:
        - 12 missing phone numbers
        - 8 missing job titles
        - 3 missing photos
        How should I handle missing fields?
        A) Leave blank
        B) Fill with 'N/A'
        C) Skip those rows entirely"
You: "B"
Agent: [extracts 100 rows with 'N/A' for missing fields] → employees.csv

---

Natural Language Commands

/scrape-to-csv <url> <description>

General scraping with CSV export.

text

/scrape-to-csv https://news.ycombinator.com
  "Get top 30 stories with title, URL, points, and comment counts"

/scrape-to-csv https://www.anthropic.com/news
  "Latest blog posts: title, date, category, URL"

/scrape-to-csv https://example.com/realestate
  "Listings: address, price, beds, baths, sqft, listing agent"

/scrape-table <url> <selector_or_description>

Extract a specific table.

text

/scrape-table https://example.com/pricing
  "The comparison table with Basic/Pro/Enterprise columns"

/scrape-table https://example.com/sales
  "Q4 2024 revenue breakdown table"

/scrape-news <url>

Optimized for news/blog scraping.

text

/scrape-news https://techcrunch.com
  "Latest 20 articles: title, author, date, excerpt, URL"

/scrape-news https://blog.openai.com
  "All posts from 2024: title, date, tags, URL"

/scrape-products <url>

Optimized for e-commerce.

text

/scrape-products https://amazon.com/s?k=laptops
  "Laptops: name, brand, price, rating, prime eligible, URL"

/scrape-products https://shopify-store.com/collections/all
  "All products: name, price, compare-at price, availability"

---

Output Format

The agent produces a structured markdown report:

markdown

## Scraping Report — example.com/products
**Session:** a1b2c3d4 | **Duration:** 2m 14s | **Rows:** 50

### Task
Extract all products with name, price, rating, and number of reviews

### Agent Decisions
- **Pagination**: Detected infinite scroll, scrolled 5 times
- **Price field**: Chose "Sale Price" per user request
- **Missing data**: Filled out-of-stock prices with "N/A" per user request
- **Columns**: name, sale_price, rating, review_count, stock_status

### Sample Data

| name | sale_price | rating | review_count | stock_status |
|------|-----------|--------|-------------|-------------|
| Widget Pro | $999 | 4.5 | 128 | In Stock |
| Gadget Mini | $499 | 4.2 | 89 | In Stock |
| Super Gizmo | $1,299 | 4.8 | 256 | Out of Stock |

### File
`./products.csv` — 50 rows, 5 columns

---

CSV Conversion Options

Option A — Python (recommended)

python

import json, csv, re

# Bridge returns: "✅ Executed JavaScript. Result: [{...}, {...}]"
msg = """PASTE_BRIDGE_RESPONSE_HERE"""
match = re.search(r'Result: (\[.*\])', msg)
if match:
    data = json.loads(match.group(1))
    with open('output.csv', 'w', newline='') as f:
        writer = csv.DictWriter(f, fieldnames=data[0].keys())
        writer.writeheader()
        writer.writerows(data)
    print(f"Wrote {len(data)} rows to output.csv")

Option B — Node.js

javascript

const fs = require('fs');
const data = JSON.parse(fs.readFileSync('data.json', 'utf8'));
const headers = Object.keys(data[0]);
const csv = [
  headers.join(','),
  ...data.map(row => headers.map(h => `"${(row[h]||'').replace(/"/g,'""')}"`).join(','))
].join('\n');
fs.writeFileSync('output.csv', csv);

---

Troubleshooting

Bridge won't start

text

Error: Cannot find module 'playwright'

Fix: npm install -D playwright && npx playwright install chromium

Site blocks scraping

Agent detects: "The site returned 403 Forbidden. This may be bot protection." Options:

• Try headless: false (looks more like a real user)
• Add delays between requests
• Use a different user agent

Page loads but no data found

Agent detects: "The page loaded but I see mostly navigation elements. Content may be behind a login or loaded dynamically." Agent asks: "Should I wait longer, scroll down, or do you have login credentials?"

Data looks wrong

Agent detects: "Prices show as 'NaN' or empty. The site may use JavaScript to render prices." Agent asks: "Should I try executing JavaScript to extract the real values, or skip this field?"

---

Comparison with Other Tools

Tool	Setup	Selectors	Complex Pages	Agent Questions	Best For
Auto Scraping to CSV	npm install	None needed	Handles automatically	Yes, clarifies ambiguity	One-off extraction, exploratory scraping
BeautifulSoup	pip install	Required	Manual handling	No	Known structure, repeated scraping
Scrapy	Project setup	Required	Middleware needed	No	Large-scale crawling, pipelines
Playwright E2E	npm install	Required	Manual handling	No	Testing, automation
Browser-Use	API key	None	Partial	Limited	Multi-page research

Use this skill when:

• You want to scrape without writing selectors
• The page structure is complex or unknown
• You need the agent to handle edge cases (scroll, popups, pagination)
• You want clarifying questions when data is ambiguous
• You need quick one-off extraction to CSV

---

Bridge API Reference

POST /sessions

Launch a new browser session.

Body:

json

{ "url": "https://example.com", "headless": false }

Response: { "id": "abc123", "url": "https://example.com" }

GET /sessions/:id/state

Get text-based DOM state.

Response: { url, title, header, content, footer }

POST /sessions/:id/act

Execute an action.

Body:

json

{ "action": "executeJavascript", "params": { "script": "return document.title;" } }

DELETE /sessions/:id

Close session.

POST /shutdown

Stop bridge.

---

Skill: auto-scraping-to-csv v1.0.0 | Bridge: page-agent-bridge.mjs | Powered by Alibaba Page-Agent + Playwright