XCrawl Crawl

Use this skill for XCrawl crawl tasks, including bulk site crawling, crawler rule design, async status polling, and delivery of crawl output for downstream s...

Security Scan

The skill's requirements and runtime instructions match its stated purpose (calling XCrawl APIs using a local API key and curl/node); nothing requests unrelated credentials or odd installs.

Purpose & Capability

Name/description (XCrawl crawl) align with instructions: it calls XCrawl endpoints, needs an XCrawl API key stored at ~/.xcrawl/config.json, and expects curl/node to be available. Required bins and the local config file are proportionate to the stated purpose.

Instruction Scope

SKILL.md instructs the agent to read ~/.xcrawl/config.json for XCRAWL_API_KEY and to call https://run.xcrawl.com. That is appropriate for this integration. Minor note: the allowed-tools list includes Write/Edit in addition to Read/Grep — the documentation never requires writing arbitrary files, so Write/Edit appears unnecessary and could be tightened.

Install Mechanism

Instruction-only skill with no install spec and no downloads — lowest-risk install footprint.

Credentials

No global environment variables or unrelated credentials are requested. The API key is read from a local config file (~/.xcrawl/config.json), which the SKILL.md documents and the metadata reflects (apiKeySource: local_config). This is proportionate to the skill's function.

Persistence & Privilege

always is false and the skill has no install hooks or system-wide changes. Autonomous invocation is allowed (platform default) but not combined with any broad privileges or unrelated credential access.

Assessment

This skill appears coherent for using XCrawl: it reads an API key from ~/.xcrawl/config.json and uses curl/node to call run.xcrawl.com. Before installing, verify you trust the publisher and the homepage (https://www.xcrawl.com/), ensure the local config file is created with correct file permissions (restrict to your user), and confirm you are comfortable storing the API key in that file. Consider removing Write/Edit permissions if you want the skill strictly read-only. Also be aware of crawl costs (credits) and that some crawl options (e.g., webhook settings or submitting private URLs) can expose data to external endpoints — review request bodies and webhook targets before use. Finally, note defaults like skip_tls_verification=true appear in the API schema; if you need secure TLS verification, override that option when calling the API.

Overview

This skill orchestrates full-site or scoped crawling with XCrawl Crawl APIs. Default behavior is raw passthrough: return upstream API response bodies as-is.

Required Local Config

Before using this skill, the user must create a local config file and write XCRAWL_API_KEY into it.

Path: ~/.xcrawl/config.json

{
  "XCRAWL_API_KEY": "<your_api_key>"
}

Read API key from local config file only. Do not require global environment variables.

Credits and Account Setup

Using XCrawl APIs consumes credits. If the user does not have an account or available credits, guide them to register at https://dash.xcrawl.com/. After registration, they can activate the free 1000 credits plan before running requests.

Tool Permission Policy

Request runtime permissions for curl and node only. Do not request Python, shell helper scripts, or other runtime permissions.

API Surface

• Start crawl: POST /v1/crawl
• Read result: GET /v1/crawl/{crawl_id}
• Base URL: https://run.xcrawl.com
• Required header: Authorization: Bearer <XCRAWL_API_KEY>

Usage Examples

cURL (create + result)

API_KEY="$(node -e "const fs=require('fs');const p=process.env.HOME+'/.xcrawl/config.json';const k=JSON.parse(fs.readFileSync(p,'utf8')).XCRAWL_API_KEY||'';process.stdout.write(k)")"

CREATE_RESP="$(curl -sS -X POST "https://run.xcrawl.com/v1/crawl" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer ${API_KEY}" \
  -d '{"url":"https://example.com","crawler":{"limit":100,"max_depth":2},"output":{"formats":["markdown","links"]}}')"

echo "$CREATE_RESP"

CRAWL_ID="$(node -e 'const s=process.argv[1];const j=JSON.parse(s);process.stdout.write(j.crawl_id||"")' "$CREATE_RESP")"

curl -sS -X GET "https://run.xcrawl.com/v1/crawl/${CRAWL_ID}" \
  -H "Authorization: Bearer ${API_KEY}"

Node

node -e '
const fs=require("fs");
const apiKey=JSON.parse(fs.readFileSync(process.env.HOME+"/.xcrawl/config.json","utf8")).XCRAWL_API_KEY;
const body={url:"https://example.com",crawler:{limit:300,max_depth:3,include:["/docs/.*"],exclude:["/blog/.*"]},request:{locale:"ja-JP"},output:{formats:["markdown","links","json"]}};
fetch("https://run.xcrawl.com/v1/crawl",{
  method:"POST",
  headers:{"Content-Type":"application/json",Authorization:`Bearer ${apiKey}`},
  body:JSON.stringify(body)
}).then(async r=>{console.log(await r.text());});
'

Request Parameters

Request endpoint and headers

• Endpoint: POST https://run.xcrawl.com/v1/crawl
• Headers:
• Content-Type: application/json
• Authorization: Bearer <api_key>

Request body: top-level fields

Field	Type	Required	Default	Description
url	string	Yes	-	Site entry URL
crawler	object	No	-	Crawler config
proxy	object	No	-	Proxy config
request	object	No	-	Request config
js_render	object	No	-	JS rendering config
output	object	No	-	Output config
webhook	object	No	-	Async callback config

crawler

Field	Type	Required	Default	Description
limit	integer	No	100	Max pages
include	string[]	No	-	Include only matching URLs (regex supported)
exclude	string[]	No	-	Exclude matching URLs (regex supported)
max_depth	integer	No	3	Max depth from entry URL
include_entire_domain	boolean	No	false	Crawl full site instead of only subpaths
include_subdomains	boolean	No	false	Include subdomains
include_external_links	boolean	No	false	Include external links
sitemaps	boolean	No	true	Use site sitemap

proxy

Field	Type	Required	Default	Description
location	string	No	US	ISO-3166-1 alpha-2 country code, e.g. US / JP / SG
sticky_session	string	No	Auto-generated	Sticky session ID; same ID attempts to reuse exit

request

Field	Type	Required	Default	Description
locale	string	No	en-US,en;q=0.9	Affects Accept-Language
device	string	No	desktop	desktop / mobile; affects UA and viewport
cookies	object map	No	-	Cookie key/value pairs
headers	object map	No	-	Header key/value pairs
only_main_content	boolean	No	true	Return main content only
block_ads	boolean	No	true	Attempt to block ad resources
skip_tls_verification	boolean	No	true	Skip TLS verification

js_render

Field	Type	Required	Default	Description
enabled	boolean	No	true	Enable browser rendering
wait_until	string	No	load	load / domcontentloaded / networkidle
viewport.width	integer	No	-	Viewport width (desktop 1920, mobile 402)
viewport.height	integer	No	-	Viewport height (desktop 1080, mobile 874)

output

Field	Type	Required	Default	Description
formats	string[]	No	["markdown"]	Output formats
screenshot	string	No	viewport	full_page / viewport (only if formats includes screenshot)
json.prompt	string	No	-	Extraction prompt
json.json_schema	object	No	-	JSON Schema

output.formats enum:

• html
• raw_html
• markdown
• links
• summary
• screenshot
• json

webhook

Field	Type	Required	Default	Description
url	string	No	-	Callback URL
headers	object map	No	-	Custom callback headers
events	string[]	No	["started","completed","failed"]	Events: started / completed / failed

Response Parameters

Create response (POST /v1/crawl)

Field	Type	Description
crawl_id	string	Task ID
endpoint	string	Always crawl
version	string	Version
status	string	Always pending

Result response (GET /v1/crawl/{crawl_id})

Field	Type	Description
crawl_id	string	Task ID
endpoint	string	Always crawl
version	string	Version
status	string	pending / crawling / completed / failed
url	string	Entry URL
data	object[]	Per-page result array
started_at	string	Start time (ISO 8601)
ended_at	string	End time (ISO 8601)
total_credits_used	integer	Total credits used

data[] fields follow output.formats:

• html, raw_html, markdown, links, summary, screenshot, json
• metadata (page metadata)
• traffic_bytes
• credits_used
• credits_detail

Workflow

1. Confirm business objective and crawl boundary.

• What content is required, what content must be excluded, and what is the completion signal.

2. Draft a bounded crawl request.

• Prefer explicit limits and path constraints.

3. Start crawl and capture task metadata.

• Record crawl_id, initial status, and request payload.

4. Poll GET /v1/crawl/{crawl_id} until terminal state.

• Track pending, crawling, completed, or failed.

5. Return raw create/result responses.

• Do not synthesize derived summaries unless explicitly requested.

Output Contract

Return:

• Endpoint flow (POST /v1/crawl + GET /v1/crawl/{crawl_id})
• request_payload used for the create request
• Raw response body from create call
• Raw response body from result call
• Error details when request fails

Do not generate summaries unless the user explicitly requests a summary.

Guardrails

• Never run an unbounded crawl without explicit constraints.
• Do not present speculative page counts as final coverage.
• Do not hardcode provider-specific tool schemas in core logic.
• Highlight policy, legal, or website-usage risks when relevant.