Crawler
v3.0.0Web crawling and scraping reference — robots.txt protocol, Scrapy framework, anti-bot detection, headless browsers, and legal considerations
⭐ 0· 573·1 current·1 all-time
MIT-0
Download zip
LicenseMIT-0 · Free to use, modify, and redistribute. No attribution required.
Security Scan
OpenClaw
Benign
high confidencePurpose & Capability
The name/description (web crawling & scraping reference) match the provided SKILL.md and the included script which only emits reference text. There are no unrelated required binaries, env vars, or config paths.
Instruction Scope
SKILL.md explicitly states outputs are plain-text reference via heredoc and no external API calls or credentials are required. The bundled script.sh functions simply print documentation. Note: the documentation discusses techniques (proxies, captcha-solving services, anti-bot workarounds) that can enable contentious scraping behavior — the skill itself only documents these topics and does not perform them.
Install Mechanism
No install spec is provided (instruction-only). The only code file is a local shell script that prints docs; nothing is downloaded or extracted during install.
Credentials
The skill declares no required environment variables or credentials and the runtime instructions do not attempt to read secrets. References to proxy providers and services in the docs are informational only and do not create a credential requirement.
Persistence & Privilege
No elevated persistence requested (always: false). The skill does not modify other skills or system config and does not request permanent presence or special privileges.
Assessment
This skill appears to be documentation-only and does not ask for credentials or install external software. Before installing: (1) review the script.sh yourself (it prints the docs) to confirm behavior; (2) be aware that the content includes instructions about proxies and captcha-solving services which can enable scraping that may violate websites' terms or laws—use the guidance only for lawful, ethical purposes; (3) if you allow the agent to execute shell scripts, ensure your agent runtime is sandboxed because any skill that can run shell commands could be modified to run arbitrary code. If you need stricter guarantees, request a version with only SKILL.md (no executable files) or audit the full script contents line-by-line.Like a lobster shell, security has layers — review code before you run it.
latestvk977xm5khc7xs31gj0ftfvf8q983f1y4
License
MIT-0
Free to use, modify, and redistribute. No attribution required.