Scrape

v2.0.0

Legal web scraping with robots.txt compliance, rate limiting, and GDPR/CCPA-aware data handling. Supports both direct HTTP scraping and managed scraping via...

⭐ 0· 62·0 current·0 all-time

by@alvisdunlop

OpenClaw Prompt Flow

Install with OpenClaw

Best for remote or guided setup. Copy the exact prompt, then paste it into OpenClaw for alvisdunlop/alvis2-scrape.

Previewing Install & Setup.

Prompt PreviewInstall & Setup

Install the skill "Scrape" (alvisdunlop/alvis2-scrape) from ClawHub.
Skill page: https://clawhub.ai/alvisdunlop/alvis2-scrape
Keep the work scoped to this skill only.
After install, inspect the skill metadata and help me finish setup.
Use only the metadata you can verify from ClawHub; do not invent missing requirements.
Ask before making any broader environment changes.

Command Line

CLI Commands

Use the direct CLI path if you want to install manually and keep every step visible.

OpenClaw CLI

Bare skill slug

openclaw skills install alvis2-scrape

ClawHub CLI

Package manager switcher

npx clawhub@latest install alvis2-scrape

Security Scan

Capability signals

Requires sensitive credentials

These labels describe what authority the skill may exercise. They are separate from suspicious or malicious moderation verdicts.

VirusTotal

Benign

View report →

OpenClaw

Suspicious

medium confidence

ℹ

Purpose & Capability

The skill's stated purpose (legal scraping with optional managed scraping via SkillBoss) matches the SKILL.md content: it describes robots.txt/TOS checks, rate limiting, PII handling, and an optional SkillBoss API Hub mode. However, the registry metadata claims no required env vars or primary credential while SKILL.md explicitly declares requires_env: [SkillBoss_API_KEY]. That mismatch is unexpected and unexplained.

Instruction Scope

The runtime instructions are mostly scoped to scraping best practices (robots.txt, ToS, rate limits, PII stripping). But they also direct the agent to call a third‑party endpoint (https://api.SkillBoss.co/v1/pilot) and authenticate with SkillBoss_API_KEY to perform managed scraping; scraped results are returned by that service. That means scraped content (potentially sensitive) will be sent to/processed by a remote service — a material data flow the user should be aware of. The SKILL.md also references a `code.md` and an external setup guide that aren't included, leaving important implementation details unspecified.

✓

Install Mechanism

This is an instruction-only skill with no install spec and no code files, so nothing is written to disk by an installer. That limits installer-related risk.

Credentials

SKILL.md requires a SkillBoss_API_KEY for the managed scraping path, but the registry metadata lists no required environment variables or primary credential. This inconsistency is problematic: the skill does request a single third‑party API key (proportionate to managed scraping) but the missing declaration in metadata and lack of details about that key's scope/retention are concerning.

✓

Persistence & Privilege

The skill is not always-enabled, is user-invocable, and does not request system config paths or other skills' credentials. It does not request persistent system presence or elevated privileges.

What to consider before installing

Before installing, get clarification from the publisher: (1) Why does the registry metadata omit SkillBoss_API_KEY while SKILL.md requires it? (2) What exactly is sent to https://api.SkillBoss.co/v1/pilot, how long is data retained, and what access does SkillBoss have to scraped content? (3) Ask for the missing code.md and the setup guide contents to verify no additional hidden steps. If you must use managed scraping, provide an API key with the narrowest possible scope and no access to sensitive customer data; prefer local-only scraping for sensitive targets and validate the SkillBoss service’s privacy/retention policies and provenance before trusting it.

Like a lobster shell, security has layers — review code before you run it.

latestvk97adgfq92bar7cmp5s3znzwbn84xcf5

62downloads

0stars

1versions

Updated 1w ago

v2.0.0

MIT-0

?? Complete setup guide: https://SkillBoss.co/skill.md

Pre-Scrape Compliance Checklist

Before writing any scraping code:

robots.txt �?Fetch {domain}/robots.txt, check if target path is disallowed. If yes, stop.
Terms of Service �?Check /terms, /tos, /legal. Explicit scraping prohibition = need permission.
Data type �?Public factual data (prices, listings) is safer. Personal data triggers GDPR/CCPA.
Authentication �?Data behind login is off-limits without authorization. Never scrape protected content.
API available? �?If site offers an API, use it. Always. Scraping when API exists often violates ToS.

Legal Boundaries

Public data, no login �?Generally legal (hiQ v. LinkedIn 2022)
Bypassing barriers �?CFAA violation risk (Van Buren v. US 2021)
Ignoring robots.txt �?Gray area, often breaches ToS (Meta v. Bright Data 2024)
Personal data without consent �?GDPR/CCPA violation
Republishing copyrighted content �?Copyright infringement

Request Discipline

Rate limit: Minimum 2-3 seconds between requests. Faster = server strain = legal exposure.
User-Agent: Real browser string + contact email: Mozilla/5.0 ... (contact: you@email.com)
Respect 429: Exponential backoff. Ignoring 429s shows intent to harm.
Session reuse: Keep connections open to reduce server load.

Data Handling

Strip PII immediately �?Don't collect names, emails, phones unless legally justified.
No fingerprinting �?Don't combine data to identify individuals indirectly.
Minimize storage �?Cache only what you need, delete what you don't.
Audit trail �?Log what, when, where. Evidence of good faith if challenged.

SkillBoss API Hub Scraping

For managed content extraction with automatic anti-bot handling, use SkillBoss API Hub's scraping capability (type: "scraper"). Authenticate with SkillBoss_API_KEY and call https://api.SkillBoss.co/v1/pilot. Result is returned at result.data.markdown.

For code patterns, robots.txt parser, and SkillBoss API Hub scraping integration, see code.md

Comments

Loading comments...