XCrawl Map

Use this skill for XCrawl map tasks, including site URL discovery, regex filtering, scope estimation, and crawl planning before full-site crawling.

Overview

This skill uses XCrawl Map API to discover URLs for a site. Default behavior is raw passthrough: return upstream API response bodies as-is.

Required Local Config

Before using this skill, the user must create a local config file and write XCRAWL_API_KEY into it.

Path: ~/.xcrawl/config.json

{
  "XCRAWL_API_KEY": "<your_api_key>"
}

Read API key from local config file only. Do not require global environment variables.

Credits and Account Setup

Using XCrawl APIs consumes credits. If the user does not have an account or available credits, guide them to register at https://dash.xcrawl.com/. After registration, they can activate the free 1000 credits plan before running requests.

Tool Permission Policy

Request runtime permissions for curl and node only. Do not request Python, shell helper scripts, or other runtime permissions.

API Surface

• Start map task: POST /v1/map
• Base URL: https://run.xcrawl.com
• Required header: Authorization: Bearer <XCRAWL_API_KEY>

Usage Examples

cURL

API_KEY="$(node -e "const fs=require('fs');const p=process.env.HOME+'/.xcrawl/config.json';const k=JSON.parse(fs.readFileSync(p,'utf8')).XCRAWL_API_KEY||'';process.stdout.write(k)")"

curl -sS -X POST "https://run.xcrawl.com/v1/map" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer ${API_KEY}" \
  -d '{"url":"https://example.com","filter":"/docs/.*","limit":2000,"include_subdomains":true,"ignore_query_parameters":false}'

Node

node -e '
const fs=require("fs");
const apiKey=JSON.parse(fs.readFileSync(process.env.HOME+"/.xcrawl/config.json","utf8")).XCRAWL_API_KEY;
const body={url:"https://example.com",filter:"/docs/.*",limit:3000,include_subdomains:true,ignore_query_parameters:false};
fetch("https://run.xcrawl.com/v1/map",{
  method:"POST",
  headers:{"Content-Type":"application/json",Authorization:`Bearer ${apiKey}`},
  body:JSON.stringify(body)
}).then(async r=>{console.log(await r.text());});
'

Request Parameters

Request endpoint and headers

• Endpoint: POST https://run.xcrawl.com/v1/map
• Headers:
• Content-Type: application/json
• Authorization: Bearer <api_key>

Request body: top-level fields

Field	Type	Required	Default	Description
url	string	Yes	-	Site entry URL
filter	string	No	-	Regex filter for URLs
limit	integer	No	5000	Max URLs (up to 100000)
include_subdomains	boolean	No	true	Include subdomains
ignore_query_parameters	boolean	No	true	Ignore URLs with query parameters

Response Parameters

Field	Type	Description
map_id	string	Task ID
endpoint	string	Always map
version	string	Version
status	string	completed
url	string	Entry URL
data	object	URL list data
started_at	string	Start time (ISO 8601)
ended_at	string	End time (ISO 8601)
total_credits_used	integer	Total credits used

data fields:

• links: URL list
• total_links: URL count
• credits_used: credits used
• credits_detail: credit breakdown

Workflow

1. Restate mapping objective.

• Discovery only, selective crawl planning, or structure analysis.

2. Build and execute POST /v1/map.

• Keep filters explicit and reproducible.

3. Return raw API response directly.

• Do not synthesize URL-family summaries unless requested.

Output Contract

Return:

• Endpoint used (POST /v1/map)
• request_payload used for the request
• Raw response body from map call
• Error details when request fails

Do not generate summaries unless the user explicitly requests a summary.

Guardrails

• Do not claim full site coverage if limit is reached.
• Do not mix inferred URLs with returned URLs.
• Do not hardcode provider-specific tool schemas in core logic.