Web Crawling API

Fetch raw HTML, rendered HTML, or clean Markdown from public webpages through Just Serp API.

Web Crawling

Use this skill when the user needs webpage retrieval rather than a search engine result. It fits crawling, scraping preparation, readable content extraction, and page structure inspection for a known URL.

When To Use It

• The user already has a target webpage URL and wants its raw HTML, rendered HTML, or cleaned Markdown.
• The task is about content extraction, page inspection, scrape preparation, or converting a page into LLM-friendly text.
• The user can provide a direct url to crawl.
• The user needs page content from the source URL itself, not Google search results about that URL.

Representative Operations

• html: Crawl Webpage (HTML) — Retrieve the raw HTML response for a page.
• renderedHtml: Crawl Webpage (Rendered HTML) — Retrieve DOM output after rendering for JavaScript-heavy pages.
• markdown: Crawl Webpage (Markdown) — Extract the main readable content as clean Markdown for summarization or downstream processing.

Request Pattern

• 3 read-only GET operations are available in this skill.
• All operations require a direct url query parameter.
• No operation in this skill requires a request body.
• Choose renderedHtml for dynamic pages, html for raw source, and markdown for readable content extraction.

How To Work

1. Read generated/operations.md before choosing an endpoint.
2. Start with one of these operations when it matches the user's request: html, renderedHtml, markdown.
3. Pick the smallest matching operation instead of guessing.
4. Ask the user for any missing required parameter. Do not invent values.
5. Call the helper with:

bash

node {baseDir}/bin/run.mjs --operation "<operation-id>" --api-key "$JUST_SERP_API_KEY" --params-json '{"key":"value"}'

Environment

• Required: JUST_SERP_API_KEY
• This skill uses JUST_SERP_API_KEY only for authenticated Just Serp API requests.
• Keep JUST_SERP_API_KEY private. Do not paste it into chat messages, screenshots, or logs.
• Project site: Just Serp API.
• Authentication details: Just Serp API Docs.

Output Rules

• Start with what was fetched: raw HTML, rendered HTML, or cleaned Markdown.
• Echo the target URL so the crawl scope is explicit.
• For markdown, surface the extracted readable content or key sections before raw JSON.
• For HTML-oriented requests, mention whether the user asked for source HTML or rendered output.
• If the backend errors, include the backend payload and the exact operation ID.