Web Fetcher

Fetch web pages and extract readable content for AI use. Use when reading, summarizing, or crawling a specific URL or small set of URLs. Prefer low-friction...

Security Scan

The skill does what it claims (fetch and extract web pages) but it relies on third-party URL-to-Markdown proxy services (r.jina.ai, markdown.new, defuddle.md) to fetch content without explicitly warning about forwarding user-supplied pages to external services — a privacy/exfiltration risk that should be considered before install.

Purpose & Capability

Name, description, SKILL.md and the bundled script all align: the skill fetches and extracts readable web content and implements a clear fallback chain. Using external URL-to-Markdown services is consistent with the 'low-friction' design, but it is a design choice that has privacy implications (see instruction_scope).

Instruction Scope

The runtime instructions and bundled script explicitly send the target URL to third-party conversion services (r.jina.ai, markdown.new, defuddle.md) and return the fetched content. That means the full target page (including any sensitive data present on it) may be proxied through those external services. SKILL.md does not require or document user consent, nor does it warn about possible storage/retention or confidentiality implications. Aside from this, the instructions stay within the stated purpose and do not ask to read unrelated files or environment variables.

Install Mechanism

Instruction-only skill with one small bundled Python script. There is no install spec, no downloads from external URLs, and nothing is written to disk beyond the included file. This is low installer risk.

Credentials

The skill requests no environment variables, credentials, or config paths. It does not require any unrelated secrets or permissions.

Persistence & Privilege

always is false and the skill is user-invocable. It does not request permanent presence or system-wide changes. The skill can be invoked autonomously by the agent (default behavior) but that is normal and not by itself a red flag.

What to consider before installing

This skill will forward the requested page URL to third‑party URL-to-Markdown services (r.jina.ai, markdown.new, defuddle.md) as its primary retrieval method. Before installing or using: (1) avoid fetching pages that contain private, confidential, or sensitive information because that content may be proxied and possibly stored by those external services; (2) consider telling users and obtaining consent before fetching private pages; (3) if privacy is important, prefer a browser-based/local headless fetch or a self-hosted reader instead of public proxy services; (4) be aware the simple classifier in the bundled script uses heuristics (marker phrases and length) and can misclassify pages as blocked or 'thin'; (5) if you need stronger guarantees about where data goes, request the skill owner to add explicit warnings and an option to disable third-party proxies or to use self-hosted alternatives. If you want a stricter assessment, provide the deployment environment (where the agent runs) and whether you expect to fetch authenticated/private pages — that would change the risk evaluation.

Fetch readable web content with a reliability-first fallback chain.

Core rule

Do not promise direct access to every site. Some sites use Cloudflare, login walls, bot detection, or legal restrictions. In those cases, switch to the next fallback instead of insisting the first method should work.

Preferred fetch order

1) Direct readable fetch

Try lightweight conversion services first:

1. r.jina.ai

   https://r.jina.ai/http://example.com
   

2. markdown.new

   https://markdown.new/https://example.com
   

3. defuddle

   https://defuddle.md/https://example.com
   

For deterministic retries, use the bundled script:

python {baseDir}/scripts/fetch_url.py "https://example.com/article"

The script returns JSON with:

• chosen method
• attempt history
• blocked/thin-content detection
• final content when successful

Use these when the user wants article text, page summaries, or structured extraction from normal public pages.

2) Detect failure modes early

Treat the fetch as failed or unreliable if you see signs like:

• Just a moment...
• Performing security verification
• Enable JavaScript and cookies
• CAPTCHA / challenge pages
• login wall instead of target content
• obvious truncation / missing article body

When this happens, stop treating the result as the page content.

3) Browser fallback for protected sites

For sites blocked behind Cloudflare or requiring real browser execution:

• Prefer a real browser session via OpenClaw browser tools when available.
• If the user is using the Chrome relay/extension, ask them to attach the tab and then inspect the live rendered page.
• Snapshot the page and extract only the needed fields.

Use browser fallback for:

• JS-heavy pages
• Cloudflare-protected pages
• sites that render key content after load
• pages where the direct markdown services return verification screens

4) Search / indexed fallback

If direct fetch and browser fetch are not available or still fail:

• search for the exact page / journal / article title
• use search snippets, publisher mirror pages, cached summaries, or secondary sources
• prefer official publisher pages when search can surface the needed field
• clearly label data as secondary-source derived if it was not read directly from the target page

This is often enough for metadata tasks like:

• editor-in-chief names
• journal impact factors
• publication frequency
• ISSN
• institutional affiliations

5) Partial-completion mode

If a site is inconsistent, return a mixed result instead of stalling:

• fill the rows that can be verified directly
• mark blocked / unresolved rows clearly
• explain what failed and which fallback was used

Practical extraction strategy

For one page

1. Try r.jina.ai
2. If blocked, try markdown.new
3. If blocked, try defuddle
4. If still blocked, use browser tools
5. If browser unavailable, use search/indexed fallback
6. Report confidence level

For many similar pages

1. Fetch the index/list page first
2. Extract all target URLs or codes
3. Process pages in batches
4. Record success/failure per row
5. Retry only failures with stronger fallback methods
6. Deliver the best complete table possible

Output guidance

When extracting structured data, prefer columns like:

• source URL
• extraction method (direct, browser, search, secondary)
• confidence (high, medium, low)
• note for blocked/unverified rows

Examples

• User: "Read this article" → direct fetch first
• User: "What does this page say?" → direct fetch, then browser fallback if blocked
• User: "Crawl this journal site" → index page first, then batched extraction with fallback chain
• User: "Cloudflare blocked it" → switch to browser or search fallback, do not keep retrying the same failed method