Web Scraper

Extract and export structured web data like text, tables, and images in JSON, CSV, Markdown, or SQL formats while respecting ethical and legal scraping pract...

rupertnt034@rupertnt034

Install

openclaw skills install @rupertnt034/rupert-web-scraper

Web Scraper Skill

Overview

Extract data from websites efficiently and ethically.

Capabilities

1. Data Extraction

Extract text content
Pull structured data
Capture tables
Get images/media

2. Formats

JSON output
CSV export
Markdown
SQL inserts

3. Features

Rate limiting
Caching
Retry logic
Error handling
Proxy support

4. Ethical Scraping

Respect robots.txt
Rate limits
User agent rotation
Legal compliance

Usage

Commands

scrape [URL] for [data]
extract [element] from [URL]
get table from [URL]
crawl [website] depth [n]
export [URL] to [format]

Examples

Input: "scrape example.com for product names and prices" Output:

json

{
  "products": [
    {"name": "Product A", "price": "$19.99"},
    {"name": "Product B", "price": "$29.99"}
  ]
}

Configuration

Rate Limits

Default: 1 request/second
Configurable: 0.1-10 req/s
Respect site limits

Output Options

JSON (default)
CSV
Markdown
SQL
Custom template

Best Practices

Always identify yourself
Cache responses
Handle errors gracefully
Stay within legal bounds
Don't overwhelm servers