Scrapeless Webunlocker Skill

Bypass website blocks and scrape web content using Scrapeless Universal Scraping API.

Security Scan

The skill largely does what it claims (calls Scrapeless API to bypass blocks) but the package metadata omits the required X_API_TOKEN and the tool will transmit full target URLs, request headers, and request bodies to a third‑party service — review provenance, billing, and privacy before installing.

Purpose & Capability

The skill's name/description, SKILL.md, README, and script are coherent: it calls the Scrapeless Universal Scraping API and needs an API token. However the registry metadata reported at the top lists no required env vars while SKILL.md and scripts require X_API_TOKEN, an inconsistency that should be resolved.

Instruction Scope

Runtime instructions and the script are scoped to building a payload and POSTing it to https://api.scrapeless.com/api/v2/unlocker/request. This means any URL, custom headers, POST data, or selectors you provide will be transmitted to the external Scrapeless service (and billed). That behavior matches the stated purpose but has privacy/billing implications that users must consider.

Install Mechanism

No install spec in the registry (instruction-only with packaged Python script). requirements.txt lists only 'requests' and 'python-dotenv' — no suspicious download/install URLs or archive extractions were present.

Credentials

Requesting a single API credential (X_API_TOKEN) is proportional to the described functionality. The inconsistency between registry metadata (no env vars) and SKILL.md/script (requires X_API_TOKEN) is the main concern. The skill reads .env (repo root) which is normal but means tokens in that file will be used and potentially transmitted to the external API.

Persistence & Privilege

The skill is not marked 'always: true' and does not request special persistent privileges. It does not modify other skills or system settings based on the provided files.

What to consider before installing

Before installing: - Confirm the skill's source and that scrapeless.com is the intended provider (repository source was 'unknown'). - Expect that every target URL, headers, and request body you give will be sent to Scrapeless and could be stored or billed — do not send sensitive private URLs or credentials. - Provide a dedicated, limited-scope API token (not reuse high-privilege secrets) and keep it out of public repos (.env should not be committed). - Resolve the metadata inconsistency: the registry should declare X_API_TOKEN as required (SKILL.md already does). - If you care about data locality/privacy, review Scrapeless's privacy/billing policies and TLS endpoint (api.scrapeless.com) and consider testing with non-sensitive targets first. - If uncertain about provenance, run the script in an isolated environment or review the full code (the main network call is a single POST to api.scrapeless.com) before granting the token.

WebUnlocker OpenClaw Skill

Use this skill to bypass website blocks and scrape web content using the Scrapeless Universal Scraping API. It supports JavaScript rendering, CAPTCHA solving, IP rotation, and intelligent request retries.

Authentication: Set X_API_TOKEN in your environment or in a .env file in the repo root.

Errors: On failure the script writes a JSON error to stderr and exits with code 1.

---

Usage

Command:

python3 scripts/webunlocker.py --url "https://example.com"

Examples:

# Scrape HTML content
python3 scripts/webunlocker.py --url "https://httpbin.io/get"

# Scrape plain text
python3 scripts/webunlocker.py --url "https://example.com" --response-type plaintext

# Scrape as Markdown
python3 scripts/webunlocker.py --url "https://example.com" --response-type markdown

# Take a screenshot
python3 scripts/webunlocker.py --url "https://example.com" --response-type png

# Capture network requests
python3 scripts/webunlocker.py --url "https://example.com" --response-type network

# Extract specific content types
python3 scripts/webunlocker.py --url "https://example.com" --response-type content --content-types emails,links,images

# Use a specific country proxy
python3 scripts/webunlocker.py --url "https://example.com" --country US

# Use POST method
python3 scripts/webunlocker.py --url "https://httpbin.org/post" --method POST --data '{"key": "value"}'

# Add custom headers
python3 scripts/webunlocker.py --url "https://example.com" --headers '{"User-Agent": "Mozilla/5.0"}'

# Use custom proxy
python3 scripts/webunlocker.py --url "https://example.com" --proxy-url "http://your-proxy-url:port"

# Enable JavaScript rendering
python3 scripts/webunlocker.py --url "https://example.com" --js-render

# Enable JavaScript rendering with headless mode
python3 scripts/webunlocker.py --url "https://example.com" --js-render --headless

# Enable JavaScript rendering and wait for specific element
python3 scripts/webunlocker.py --url "https://example.com" --js-render --wait-selector "body > div > p:nth-child(3) > a"

# Bypass Cloudflare protection with JavaScript rendering
python3 scripts/webunlocker.py --url "https://example.com" --js-render

# Bypass Cloudflare Turnstile challenge
python3 scripts/webunlocker.py --url "https://2captcha.com/demo/cloudflare-turnstile-challenge" --js-render --headless --response-type markdown

---

Summary

Argument	Description	Default
--url	Target URL	Required
--method	HTTP method	GET
--redirect	Allow redirects	False
--headers	Custom headers as JSON string	None
--data	Request data as JSON string	None
--response-type	Response type (html, plaintext, markdown, png, jpeg, network, content)	html
--content-types	Content types to extract (comma-separated)	None
--country	Country code for proxy	ANY
--proxy-url	Custom proxy URL	None
--js-render	Enable JavaScript rendering	False
--headless	Run browser in headless mode	False
--wait-selector	Wait for element with this selector to appear	None

Output: All commands return JSON objects with the scraped content or Cloudflare bypass results.

---

Response Types

HTML

Returns the HTML content of the page as an escaped string.

Plaintext

Returns the plain text content of the page, removing all HTML tags.

Markdown

Returns the page content formatted as Markdown for better readability.

PNG/JPEG

Returns a base64 encoded string of the page screenshot.

Network

Returns all network requests made during page load, including URLs, methods, status codes, and headers.

Content

Returns specific content types extracted from the page, such as emails, phone numbers, headings, images, audios, videos, links, hashtags, metadata, tables, and favicon.

---

Notes

⚠️ Timeout Policy:

• Page load timeout: 30 seconds
• Global execution timeout: 180 seconds

⚠️ Supported CAPTCHAs:

• reCaptcha V2
• Cloudflare Turnstile
• Cloudflare Challenge

⚠️ Rate Limits:

• 429 errors indicate rate limit exceeded. Reduce request frequency or upgrade plan.

⚠️ Billing:

• Charges are applied on a per-request basis
• Only successful requests will be billed