Cloudflare Crawl

Crawl websites using Cloudflare's Browser Rendering API. Use when you need to scrape entire sites, build knowledge bases, extract content from multiple pages...

Crawl entire websites using Cloudflare's Browser Rendering /crawl API. Async job-based crawling with JS rendering.

When to Use

• Scrape entire sites (not just single pages)
• Build knowledge bases or RAG datasets
• Research across multiple pages
• Sites protected by Cloudflare (CF won't block itself)
• Need Markdown or structured JSON output

Prerequisites

Get Cloudflare API credentials:

1. Go to https://dash.cloudflare.com/profile/api-tokens
2. Create token with Account.Browser Rendering permission
3. Get your Account ID from dashboard URL

Set environment variables:

export CLOUDFLARE_API_TOKEN="your_token"
export CLOUDFLARE_ACCOUNT_ID="your_account_id"

Quick Start

# Start a crawl job
node scripts/crawl.js start https://example.com --limit 50

# Check status
node scripts/crawl.js status <job_id>

# Get results as markdown
node scripts/crawl.js results <job_id> --format markdown

API Overview

1. Start Crawl Job

curl -X POST "https://api.cloudflare.com/client/v4/accounts/$CLOUDFLARE_ACCOUNT_ID/browser-rendering/crawl" \
  -H "Authorization: Bearer $CLOUDFLARE_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com",
    "limit": 50,
    "depth": 3,
    "formats": ["markdown"]
  }'

Returns: { "success": true, "result": "job_id_here" }

2. Poll for Completion

curl "https://api.cloudflare.com/client/v4/accounts/$CLOUDFLARE_ACCOUNT_ID/browser-rendering/crawl/$JOB_ID?limit=1" \
  -H "Authorization: Bearer $CLOUDFLARE_API_TOKEN"

Status values: running, completed, errored, cancelled_due_to_timeout

3. Get Results

curl "https://api.cloudflare.com/client/v4/accounts/$CLOUDFLARE_ACCOUNT_ID/browser-rendering/crawl/$JOB_ID" \
  -H "Authorization: Bearer $CLOUDFLARE_API_TOKEN"

Parameters

Parameter	Type	Default	Description
url	string	required	Starting URL
limit	number	10	Max pages to crawl (max 100,000)
depth	number	100000	Max link depth from start URL
source	string	"all"	URL discovery: all, sitemaps, links
formats	array	["html"]	Output: html, markdown, json
render	boolean	true	Execute JS (false = fast HTML only)

Output Formats

Markdown (best for AI)

{
  "url": "https://example.com/page",
  "status": "completed",
  "markdown": "# Page Title\n\nContent here...",
  "metadata": { "title": "Page Title", "status": 200 }
}

JSON (AI-extracted)

Uses Workers AI to extract structured data. Requires jsonOptions.prompt:

{
  "formats": ["json"],
  "jsonOptions": {
    "prompt": "Extract product name, price, and description"
  }
}

Pricing

Plan	Free Tier	Overage
Workers Free	10 min/day	N/A
Workers Paid	10 hrs/month	$0.09/hour

Limits

• Max 100,000 pages per crawl
• 7 day max runtime
• Results available 14 days
• Free plan: 10 concurrent, 100 pages max

Example: Crawl for RAG

// Crawl docs site for knowledge base
const job = await startCrawl({
  url: 'https://docs.example.com',
  limit: 500,
  formats: ['markdown'],
  source: 'sitemaps' // Use sitemap for efficiency
});

// Wait for completion
const results = await waitForCrawl(job.id);

// Save markdown files for RAG
for (const page of results.records) {
  if (page.status === 'completed') {
    fs.writeFileSync(`docs/${slugify(page.url)}.md`, page.markdown);
  }
}

vs Browserbase/Stagehand

Use Case	Cloudflare Crawl	Browserbase
Full site scrape	✅ Best	❌ Manual
Interactive (forms)	❌ No	✅ Best
CF-protected sites	✅ Native	⚠️ Cloud bypass
AI extraction	✅ Built-in	✅ Stagehand
Session management	✅ Async jobs	❌ Manual
Cost	$0.09/hr	Credits

Use Cloudflare Crawl for bulk content extraction. Use Browserbase for interactive automation.