Data Scraper

Web page data collection and structured text extraction

Web Data Scraper — Extract structured data from web pages using curl + parsing. Lightweight, no browser required. Supports HTML-to-text, table extraction, price monitoring, and batch scraping.

When to Use

• Extract text content from web pages (articles, blogs, docs)
• Scrape product prices, reviews, or listings
• Monitor pages for changes (price drops, new content)
• Batch-collect data from multiple URLs
• Convert HTML tables to structured formats (JSON/CSV)

Quick Start

bash

# Extract readable text from URL
data-scraper fetch "https://example.com/article"

# Extract specific elements
data-scraper extract "https://example.com" --selector "h2, .price"

# Monitor for changes
data-scraper watch "https://example.com/product" --interval 3600

Extraction Modes

Text Mode (default)

Fetches page and extracts readable content, stripping HTML tags, scripts, and styles. Similar to reader mode.

bash

data-scraper fetch URL
# Output: clean markdown text

Selector Mode

Target specific CSS selectors for precise extraction.

bash

data-scraper extract URL --selector ".product-title, .price, .rating"
# Output: matched elements as structured data

Table Mode

Extract HTML tables into structured formats.

bash

data-scraper table URL --index 0
# Output: JSON array of row objects (header → value mapping)

Link Mode

Extract all links from a page with optional filtering.

bash

data-scraper links URL --filter "*.pdf"
# Output: filtered list of absolute URLs

Batch Scraping

bash

# Scrape multiple URLs
data-scraper batch urls.txt --output results/

# With rate limiting
data-scraper batch urls.txt --delay 2000 --output results/

urls.txt format:

text

https://site1.com/page
https://site2.com/page
https://site3.com/page

Change Monitoring

bash

# Watch for changes, alert on diff
data-scraper watch URL --selector ".price" --interval 3600

# Compare with previous snapshot
data-scraper diff URL

Stores snapshots in data-scraper/snapshots/ with timestamps. Alerts via notification-hub when changes detected.

Output Formats

Format	Flag	Use Case
Text	--format text	Reading, summarization
JSON	--format json	Data processing
CSV	--format csv	Spreadsheets
Markdown	--format md	Documentation

Headers & Auth

bash

# Custom headers
data-scraper fetch URL --header "Authorization: Bearer TOKEN"

# Cookie-based auth
data-scraper fetch URL --cookie "session=abc123"

# User-Agent override
data-scraper fetch URL --ua "Mozilla/5.0..."

Rate Limiting & Ethics

• Default: 1 request per second per domain
• Respects robots.txt when --polite flag is set
• Configurable delay between requests
• Stops on 429 (Too Many Requests) and backs off

Error Handling

Error	Behavior
404	Log and skip
403/401	Warn about auth requirement
429	Exponential backoff (max 3 retries)
Timeout	Retry once with longer timeout
SSL error	Warn, option to proceed with --insecure

Integration

• web-claude: Use as fallback when web_fetch isn't enough
• competitor-watch: Feed scraped data into competitor analysis
• seo-audit: Scrape competitor pages for SEO comparison
• performance-tracker: Collect social metrics from public profiles