Web Scraper as a Service

Build client-ready web scrapers with clean data output. Use when creating scrapers for clients, extracting data from websites, or delivering scraping projects.

Security Scan

The skill is internally consistent: its instructions, required tools, and outputs align with a web-scraping service and it does not request unrelated credentials, installs, or elevated persistence.

Purpose & Capability

The name/description (web scraper service) match the SKILL.md: it enumerates scraping templates, generation of a Python scraper, data cleaning, and delivery packaging. There are no unrelated environment variables, binaries, or config paths requested that would be disproportionate to building scrapers.

Instruction Scope

The runtime instructions are focused on collecting page structure, generating scraper code, running scrapes, cleaning data, and packaging results. They explicitly recommend respecting robots.txt and ToS and avoiding personal data unless authorized. The instructions do require fetching target URLs (expected for scraping) but do not instruct reading unrelated local files or exfiltrating secrets.

Install Mechanism

This is an instruction-only skill with no install spec and no code files included. That minimizes code-on-disk risk; the instructions recommend standard Python packages (requests, BeautifulSoup, playwright) which is proportionate for the stated purpose.

Credentials

No environment variables, credentials, or config paths are requested; the requirement set is minimal and appropriate for a scraper template/instruction skill. The skill does not ask for unrelated secrets or broad system access.

Persistence & Privilege

The skill does not request always: true and is user-invocable only. It does not instruct modifying other skills or system-wide agent settings. Autonomous invocation is enabled by default but not combined with other elevated privileges.

Assessment

This skill appears coherent and focused on building scrapers, but consider these practical precautions before using it: (1) The skill will fetch arbitrary target URLs — ensure you (and your client) are authorized to scrape the target and that scraping does not violate site terms or laws. (2) Some sites use CAPTCHAs or anti-bot protections; the guidance mentions detecting these but does not instruct bypassing them — avoid attempting evasive or unauthorized bypass techniques. (3) If you execute generated scrapers, run them in an isolated environment (container/VM) and watch network usage and stored data to avoid accidental leakage of client or system secrets. (4) The skill has no stated source or homepage — if you need long-term support or guarantees, prefer vetted libraries or an authored/published implementation. (5) If scraping requires authenticated access to a target, expect you will need to supply credentials separately; the skill does not request or manage credentials itself.

Turn scraping briefs into deliverable scraping projects. Generates the scraper, runs it, cleans the data, and packages everything for the client.

How to Use

/web-scraper-as-a-service "Scrape all products from example-store.com — need name, price, description, images. CSV output."
/web-scraper-as-a-service https://example.com --fields "title,price,rating,url" --format csv
/web-scraper-as-a-service brief.txt

Scraper Generation Pipeline

Step 1: Analyze the Target

Before writing any code:

1. Fetch the target URL to understand the page structure
2. Identify:
   • Is the site server-rendered (static HTML) or client-rendered (JavaScript/SPA)?
   • What anti-scraping measures are visible? (Cloudflare, CAPTCHAs, rate limits)
   • Pagination pattern (URL params, infinite scroll, load more button)
   • Data structure (product cards, table rows, list items)
   • Total estimated volume (number of pages/items)
3. Choose the right tool:
   • Static HTML → Python + requests + BeautifulSoup
   • JavaScript-rendered → Python + playwright
   • API available → Direct API calls (check network tab patterns)

Step 2: Build the Scraper

Generate a complete Python script in scraper/ directory:

scraper/
  scrape.py           # Main scraper script
  requirements.txt    # Dependencies
  config.json         # Target URLs, fields, settings
  README.md           # Setup and usage instructions for client

scrape.py must include:

# Required features in every scraper:

# 1. Configuration
import json
config = json.load(open('config.json'))

# 2. Rate limiting (ALWAYS — be respectful)
import time
DELAY_BETWEEN_REQUESTS = 2  # seconds, adjustable in config

# 3. Retry logic
MAX_RETRIES = 3
RETRY_DELAY = 5

# 4. User-Agent rotation
USER_AGENTS = [
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36...",
    # ... at least 5 user agents
]

# 5. Progress tracking
print(f"Scraping page {current}/{total} — {items_collected} items collected")

# 6. Error handling
# - Log errors but don't crash on individual page failures
# - Save progress incrementally (don't lose data on crash)
# - Write errors to error_log.txt

# 7. Output
# - Save data incrementally (append to file, don't hold in memory)
# - Support CSV and JSON output
# - Clean and normalize data before saving

# 8. Resume capability
# - Track last successfully scraped page/URL
# - Can resume from where it left off if interrupted

Step 3: Data Cleaning

After scraping, clean the data:

1. Remove duplicates (by unique identifier or composite key)
2. Normalize text (strip extra whitespace, fix encoding issues, consistent capitalization)
3. Validate data (no empty required fields, prices are numbers, URLs are valid)
4. Standardize formats (dates to ISO 8601, currency to numbers, consistent units)
5. Generate data quality report:

   Data Quality Report
   ───────────────────
   Total records: 2,487
   Duplicates removed: 13
   Empty fields filled: 0
   Fields with issues: price (3 records had non-numeric values — cleaned)
   Completeness: 99.5%
   

Step 4: Client Deliverable Package

Generate a complete deliverable:

delivery/
  data.csv                    # Clean data in requested format
  data.json                   # JSON alternative
  data-quality-report.md      # Quality metrics
  scraper-documentation.md    # How the scraper works
  README.md                   # Quick start guide

scraper-documentation.md includes:

• What was scraped and from where
• How many records collected
• Data fields and their descriptions
• How to re-run the scraper
• Known limitations
• Date of scraping

Step 5: Output to User

Present:

1. Summary: X records scraped from Y pages, Z% data quality
2. Sample data: First 5 rows of the output
3. File locations: Where the deliverables are saved
4. Client handoff notes: What to tell the client about the data

Scraper Templates

Based on the target type, use the appropriate template:

E-commerce Product Scraper

Fields: name, price, original_price, discount, description, images, category, sku, rating, review_count, availability, url

Real Estate Listings

Fields: address, price, bedrooms, bathrooms, sqft, lot_size, listing_type, agent, description, images, url

Job Listings

Fields: title, company, location, salary, job_type, description, requirements, posted_date, url

Directory/Business Listings

Fields: business_name, address, phone, website, category, rating, review_count, hours, description

News/Blog Articles

Fields: title, author, date, content, tags, url, image

Ethical Scraping Rules

1. Always respect robots.txt — check before scraping
2. Rate limit — minimum 2 second delay between requests
3. Identify yourself — use realistic but honest User-Agent
4. Don't scrape personal data (emails, phone numbers) unless explicitly authorized by the client AND the data is publicly displayed
5. Cache responses — don't re-scrape pages unnecessarily
6. Check ToS — note if the site's terms prohibit scraping and inform the client