Web Scraper

Extract structured data from websites using browser automation. Use when scraping product listings, articles, contact info, prices, or any web content. Supports single pages, pagination, infinite scroll, and dynamic content. Outputs to CSV, JSON, or Excel.

Overview

Professional web scraping skill using agent-browser. Extract structured data from any website with support for JavaScript-rendered content, pagination, and complex selectors.

Use Cases

• E-commerce: Product listings, prices, reviews, inventory
• Real Estate: Property listings, prices, agent contacts
• Job Boards: Job postings, salaries, requirements
• News/Media: Articles, headlines, publication dates
• Directories: Business listings, contact information
• Competitor Monitoring: Prices, products, content changes

Quick Start

Scrape Single Page

python scripts/scrape_page.py \
  --url "https://example.com/products" \
  --fields "title= h2.title,price=.price,link=a.href" \
  --output products.csv

Scrape with Pagination

python scripts/scrape_paginated.py \
  --url "https://example.com/products?page={page}" \
  --pages 10 \
  --fields "title,price,description" \
  --output all_products.csv

Scripts

scrape_page.py

Scrape a single page or static list.

Arguments:

• --url - Target URL
• --fields - Field definitions (name=selector format, comma-separated)
• --output - Output file (CSV, JSON, or XLSX)
• --format - Output format (csv, json, xlsx)
• --wait - Wait time for dynamic content (seconds)

Field Definition Format:

fieldname=css_selector

Examples:

title=h1.product-title
price=.price-tag
description=.product-description
image=img.product-image.src
link=a.product-link.href

scrape_paginated.py

Scrape multiple pages with pagination.

Arguments:

• --url - URL pattern (use {page} for page number)
• --pages - Number of pages to scrape
• --fields - Field definitions
• --output - Output file
• --delay - Delay between pages (seconds)
• --next-selector - CSS selector for "next page" button (alternative to URL pattern)

scrape_infinite_scroll.py

Scrape pages with infinite scroll loading.

Arguments:

• --url - Target URL
• --scrolls - Number of scroll actions
• --fields - Field definitions
• --output - Output file
• --scroll-delay - Delay between scrolls (ms)

scrape_dynamic.py

Scrape JavaScript-heavy sites with custom interactions.

Arguments:

• --url - Target URL
• --actions - JSON file with interaction sequence
• --fields - Field definitions
• --output - Output file

Configuration

Actions JSON Format (for dynamic scraping)

{
  "actions": [
    {"type": "click", "selector": "#load-more"},
    {"type": "wait", "ms": 2000},
    {"type": "scroll", "direction": "down", "pixels": 500},
    {"type": "fill", "selector": "#search", "value": "keyword"},
    {"type": "press", "key": "Enter"}
  ]
}

Output Formats

CSV:

title,price,link,url
"Product A",29.99,https://...,https://...
"Product B",39.99,https://...,https://...

JSON:

[
  {
    "title": "Product A",
    "price": "29.99",
    "link": "https://...",
    "scraped_at": "2026-03-07T16:00:00"
  }
]

Excel (XLSX):

• Same as CSV but with formatting options
• Multiple sheets support
• Auto-fit columns

Examples

Example 1: Scrape E-commerce Products

python scripts/scrape_paginated.py \
  --url "https://example.com/shop?page={page}" \
  --pages 5 \
  --fields "name=.product-name,price=.price,rating=.stars,reviews=.review-count,url=a.href" \
  --output products.csv \
  --delay 3

Example 2: Scrape News Articles

python scripts/scrape_page.py \
  --url "https://news-site.com/latest" \
  --fields "headline=h2.article-title,summary=.article-summary,author=.byline,date=.publish-date,url=a.read-more.href" \
  --output articles.json \
  --format json

Example 3: Scrape Job Postings

python scripts/scrape_infinite_scroll.py \
  --url "https://jobs-site.com/search" \
  --scrolls 10 \
  --fields "title=.job-title,company=.company-name,location=.location,salary=.salary,posted=.date-posted,url=a.job-link.href" \
  --output jobs.csv \
  --scroll-delay 1500

Example 4: Scrape Real Estate Listings

python scripts/scrape_paginated.py \
  --url "https://realestate.com/listings?page={page}" \
  --pages 20 \
  --fields "address=.property-address,price=.listing-price,beds=.bedrooms,baths=.bathrooms,sqft=.square-feet,url=a.property-link.href" \
  --output listings.xlsx \
  --format xlsx \
  --delay 5

Best Practices

1. Respect robots.txt - Check and follow site rules
2. Rate limiting - Add delays between requests (2-5s recommended)
3. Error handling - Handle missing elements gracefully
4. User-Agent - Use realistic browser headers
5. Retry logic - Implement retries for failed requests
6. Data validation - Validate extracted data before saving
7. Storage - Save intermediate results for long scrapes

Anti-Scraping Measures

Some sites employ anti-scraping techniques:

Measure	Countermeasure
IP blocking	Use proxies, rotate IPs
CAPTCHA	Manual solving or CAPTCHA services
Rate limiting	Increase delays, randomize timing
JavaScript challenges	Use browser automation (agent-browser)
Honeypot traps	Avoid hidden fields, validate selectors

Legal Considerations

• Public data: Generally legal to scrape
• Terms of Service: Review site ToS before scraping
• Copyright: Don't republish copyrighted content
• Personal data: GDPR/privacy laws may apply
• Commercial use: May require permission

Disclaimer: This skill is for educational purposes. Users are responsible for compliance with applicable laws and website terms.

Troubleshooting

• Elements not found: Verify CSS selectors with browser dev tools
• Empty results: Check if content is JavaScript-rendered (use dynamic scraping)
• Timeout errors: Increase wait time or check network
• Blocked requests: Add delays, rotate user agents, or use proxies
• Incomplete data: Verify pagination or scroll handling

References

CSS Selector Guide

See references/css-selectors.md for comprehensive selector examples.

Common Website Patterns

See references/website-patterns.md for common HTML structures and selectors.