scrape

Legal web scraping with robots.txt compliance, rate limiting, and GDPR/CCPA-aware data handling. Supports both direct HTTP scraping and managed scraping via...

Security Scan

The skill's instructions and code are generally coherent for a scraper that can optionally call a managed

Purpose & Capability

The SKILL.md and code clearly implement a scraper and an optional integration with a 'SkillBoss API Hub' (api.skillbossai.com) which requires SKILLBOSS_API_KEY — that fits the described purpose. However the registry metadata says 'Required env vars: none' while SKILL.md and code require SKILLBOSS_API_KEY, an inconsistency that should be resolved before trusting the skill.

Instruction Scope

The runtime instructions focus on robots.txt, ToS checks, rate limiting, PII handling, and using either direct HTTP or the SkillBoss API Hub. The code examples only reference scraping-related files, HTTP requests, and the declared SKILLBOSS_API_KEY; there is no unrelated file/system access or vague 'gather context' language.

Install Mechanism

This is an instruction-only skill with no install spec and no code files to be executed by the platform. That minimizes install-time risk.

Credentials

Functionality legitimately needs a single API credential (SKILLBOSS_API_KEY) for the managed scraping path — that's proportionate. The concern is the metadata mismatch: the registry lists no required env vars while the SKILL.md and code access SKILLBOSS_API_KEY. Also the SkillBoss API is a third-party endpoint; users should confirm the trust model and key scope before providing credentials.

Persistence & Privilege

The skill is not marked always:true and does not request persistent system-wide privileges. It is user-invocable and allows autonomous invocation (platform default), which is expected for skills.

What to consider before installing

This skill appears to implement a legitimate scraper and a managed scraping option via api.skillbossai.com, but there are a few things to check before installing: - Metadata mismatch: the registry says no env vars are required but SKILL.md and code read SKILLBOSS_API_KEY. Do not provide any API key until the publisher corrects the metadata or you verify the key's intended use. - Third-party endpoint: the skill will send requests and scraped content to https://api.skillbossai.com when you use the managed path. Verify who operates SkillBoss, what data they retain, and the key's scope and revocation process. - Data exposure: if you use the SkillBoss path, scraped page content (including any inadvertently collected PII) will be transmitted to that external service. Only run against data you are authorized to process, and test with non-sensitive pages first. - Least privilege: if you must provide SKILLBOSS_API_KEY, create a key with minimal privileges and the ability to revoke it quickly. - Source provenance: the skill's homepage and source are unknown. Prefer skills with verifiable source repositories or vendor pages. Ask the publisher for a homepage/source and an explanation for the metadata mismatch. If you cannot verify the publisher or the SkillBoss service, run the direct-http code locally in an isolated environment (no real credentials) and avoid using the managed SkillBoss path until you've confirmed safety.

Pre-Scrape Compliance Checklist

Before writing any scraping code:

1. robots.txt — Fetch {domain}/robots.txt, check if target path is disallowed. If yes, stop.
2. Terms of Service — Check /terms, /tos, /legal. Explicit scraping prohibition = need permission.
3. Data type — Public factual data (prices, listings) is safer. Personal data triggers GDPR/CCPA.
4. Authentication — Data behind login is off-limits without authorization. Never scrape protected content.
5. API available? — If site offers an API, use it. Always. Scraping when API exists often violates ToS.

Legal Boundaries

• Public data, no login — Generally legal (hiQ v. LinkedIn 2022)
• Bypassing barriers — CFAA violation risk (Van Buren v. US 2021)
• Ignoring robots.txt — Gray area, often breaches ToS (Meta v. Bright Data 2024)
• Personal data without consent — GDPR/CCPA violation
• Republishing copyrighted content — Copyright infringement

Request Discipline

• Rate limit: Minimum 2-3 seconds between requests. Faster = server strain = legal exposure.
• User-Agent: Real browser string + contact email: Mozilla/5.0 ... (contact: [e-mail])
• Respect 429: Exponential backoff. Ignoring 429s shows intent to harm.
• Session reuse: Keep connections open to reduce server load.

Data Handling

• Strip PII immediately — Don't collect names, emails, phones unless legally justified.
• No fingerprinting — Don't combine data to identify individuals indirectly.
• Minimize storage — Cache only what you need, delete what you don't.
• Audit trail — Log what, when, where. Evidence of good faith if challenged.

SkillBoss API Hub Scraping

For managed content extraction with automatic anti-bot handling, use SkillBoss API Hub's scraping capability (type: "scraper"). Authenticate with SKILLBOSS_API_KEY and call https://api.skillbossai.com/v1/pilot. Result is returned at result.data.markdown.

For code patterns, robots.txt parser, and SkillBoss API Hub scraping integration, see code.md