Scrapling Fetch

支持自动绕过 Cloudflare Turnstile 和微信公众号反爬机制的网页内容抓取工具，输出干净Markdown或纯文本。

Security Scan

The skill largely matches its scraping purpose, but it contains hard-coded billing credentials and a few mismatches (venv path / install spec) that are disproportionate or inconsistent and warrant caution before use.

Purpose & Capability

The name/description (anti-bot web fetch) aligns with included scripts that use Scrapling, Playwright, and Jina Reader. However, the skill claims no required credentials/config but includes a paid mode that expects an API key; references/skill.json lists dependencies/install commands although the registry install spec is empty. These inconsistencies (no declared env vars but an embedded billing key; install command present only in files) reduce coherence.

Instruction Scope

SKILL.md and the scripts instruct the agent to run local Python scripts that fetch pages, call r.jina.ai, and contact skillpay.me for billing. The runtime instructions do not request unrelated system files or secrets. They do, however, point at a fixed virtualenv path and show commands that assume local filesystem layout (e.g., ~/.openclaw/workspace/.venv).

Install Mechanism

There is no install spec in the registry (instruction-only), which is the lowest-risk case, but repository files (README and references/skill.json) include pip/playwright install commands. This mismatch means installation is manual and the skill expects external packages (scrapling, playwright) to be present; that is reasonable for the stated purpose but the missing official install step is an operational inconsistency.

Credentials

Although the declared requirements list no env vars/credentials, scripts/fetch_paid.py include a hard-coded billing API key (BILLING_API_KEY) and a SKILL_ID. Embedding a secret in code is disproportionate and risky: the key could be abused by anyone with the skill bundle, and the skill will call billing endpoints using that key. The script also hard-codes VENV_PYTHON to a specific user path (/Users/gaolei/...), which is brittle and inconsistent with the declared venv path in references/skill.json.

Persistence & Privilege

The skill is not always-enabled and uses normal agent invocation. It does not request elevated system privileges or attempt to modify other skill configurations. No persistent installation mechanism is enforced by the registry metadata.

What to consider before installing

This skill appears to do the scraping it claims, but there are red flags you should address before installing or running it: - Hard-coded billing API key: scripts/fetch_paid.py contains a visible BILLING_API_KEY value. Do not use the paid script as-is — that key may belong to someone else, could be revoked, or could be abused to query/charge the billing service. Replace it with your own key (or modify the script to read the key from an environment variable) before using paid mode. - VENV path mismatch: both scripts call a hard-coded VENV_PYTHON (/Users/gaolei/...) which likely won't exist on your machine. Update the path to your environment or run the scripts with your python interpreter to avoid unintentionally invoking an unexpected interpreter. - Missing/ambiguous install step: the repo mentions pip install and playwright install, but the registry has no install spec. Ensure dependencies (scrapling, playwright, requests, etc.) are installed in an isolated virtualenv before running. - Network & legal considerations: the tool bypasses anti-bot protections. Verify you have legal/rightful permission to scrape target sites and review terms of service before using bypass techniques. - Operational safety: run initial tests against safe/public pages. If you must use paid mode, prefer a modified script that reads BILLING_API_KEY from an environment variable (not hard-coded), verify the billing endpoint behavior, and confirm the SKILL_ID and payment flows are legitimate. If you are not comfortable making these code changes, avoid enabling the paid mode and prefer the free/fast path (Jina) after installing dependencies in a controlled environment. If you plan to share the skill, remove embedded secrets and fix the venv/path assumptions first.

Scrapling Fetch - 自适应网页抓取

快速开始

微信公众号（自动绕过反爬）

~/.openclaw/workspace/.venv/bin/python3 \
  ~/.openclaw/workspace/skills/scrapling-fetch/scripts/fetch.py \
  "https://mp.weixin.qq.com/s/xxxxxx"

普通网页（快速模式）

~/.openclaw/workspace/.venv/bin/python3 \
  ~/.openclaw/workspace/skills/scrapling-fetch/scripts/fetch.py \
  "https://example.com" \
  --fast

输出格式

默认输出 JSON：

{
  "title": "文章标题",
  "author": "作者",
  "content": "正文内容（Markdown格式）",
  "word_count": 1234
}

加 --text 参数只输出纯文本。

---

工具选择策略

网站类型	推荐工具	原因
微信公众号	Scrapling	自动绕过反爬 ✅
Medium/Substack	Scrapling	绕过反爬机制
Cloudflare 保护	Scrapling	支持 Turnstile 绕过
普通技术博客	Jina（--fast）	速度快（1.4秒）
GitHub/静态页面	web_fetch	无需额外工具

---

脚本说明

fetch.py

核心功能：

• 自动检测网站类型
• 绕过主流反爬机制
• 提取干净的 Markdown 内容
• 支持图片链接保留

参数：

• url - 必填，目标网址
• --fast - 使用 Jina Reader（适合普通网页）
• --text - 只输出纯文本（默认 JSON）
• --max-chars - 最大字符数（默认 50000）

示例：

# 抓取微信文章
python3 fetch.py "https://mp.weixin.qq.com/s/xxxxx"

# 快速抓取普通网页
python3 fetch.py "https://blog.example.com" --fast

# 限制长度
python3 fetch.py "https://example.com" --max-chars 10000

---

使用场景

1. AI 内容创作流程

用户发链接 → scrapling-fetch 抓取 → AI 消化 → 生成内容

2. 研究资料收集

# 批量抓取多篇公众号文章
for url in "url1" "url2" "url3"; do
  python3 fetch.py "$url" >> articles.jsonl
done

3. 失败重试策略

# web_fetch 失败 → 自动切换 Scrapling
if ! web_fetch "$url"; then
  python3 fetch.py "$url"
fi

---

技术原理

Scrapling 核心特性：

1. StealthyFetcher - 模拟真实浏览器，绕过反爬
2. 自适应选择器 - 网站改版也能定位元素
3. Playwright 引擎 - 无头浏览器渲染

Jina Reader：

• API: https://r.jina.ai/{url}
• 自动提取正文、去广告
• 速度约 1.4 秒
• 免费限额 200 次/天

---

注意事项

1. 微信文章 - Scrapling 成功率 >95%
2. 推特/微博 - 需要登录，此工具无法抓取
3. 付费内容 - 仅能抓取公开部分
4. 频率限制 - 建议 2-3 秒间隔，避免被封

---

💰 付费版本（SkillPay 计费）

快速开始

# 付费模式（需要 user_id）
python3 fetch_paid.py "https://mp.weixin.qq.com/s/xxxxx" --user-id user123

# 免费模式（不计费）
python3 fetch_paid.py "https://mp.weixin.qq.com/s/xxxxx" --free

计费说明

• 单价: $0.01 USDT / 次
• 计费模型: 1 USDT = 1000 tokens，每次调用消耗 1 token
• 最低充值: 8 USDT
• 支付方式: BNB Chain USDT

配置步骤

1. 获取 API Key

   • 访问 https://skillpay.me/dashboard/config
   • 复制 API Key（以 sk_ 开头）

2. 修改配置

   # 编辑 fetch_paid.py
   BILLING_API_KEY = 'sk_your_api_key_here'
   

3. 测试扣费

   python3 fetch_paid.py "https://example.com" --user-id test_user
   

返回示例（余额不足）

{
  "error": "余额不足",
  "balance": 0.005,
  "payment_url": "https://skillpay.me/pay/xxx",
  "hint": "请充值后继续使用"
}

API 接口

接口	方法	说明
/api/v1/billing/balance	GET	查询余额
/api/v1/billing/charge	POST	扣费
/api/v1/billing/payment-link	POST	生成充值链接

---

📊 收益预期

场景	调用量/天	日收入	月收入
个人使用	10 次	$0.10	$3
小团队	100 次	$1.00	$30
公开服务	1000 次	$10.00	$300

---

相关技能

• web_fetch - 轻量级网页抓取（无反爬绕过）
• browser - 浏览器自动化（需要手动操作）
• xurl - 推特 API（需配置）

---

参考

• Scrapling GitHub: https://github.com/D4Vinci/Scrapling
• Jina Reader: https://jina.ai/reader