FlowCrawl — Stealth Web Scraper That Bypasses Everything

Stealth web scraper. Give it any URL and it punches through Cloudflare, bot detection, and WAFs automatically using a 3-tier cascade (plain HTTP → TLS spoof...

FlowCrawl

Scrape any website. Bypass any bot protection. Free.

Install Scrapling First

pip install scrapling

Scrapling installs Playwright automatically on first run. That's the only dependency.

Quick Usage

# Single URL — prints clean markdown to stdout
python3 ~/clawd/skills/flowcrawl/scripts/flowcrawl.py https://example.com

# Spider the whole site
python3 ~/clawd/skills/flowcrawl/scripts/flowcrawl.py https://example.com --deep

# Deep crawl with limits, save and combine
python3 ~/clawd/skills/flowcrawl/scripts/flowcrawl.py https://example.com --deep --limit 30 --combine

# JSON output — pipe into anything
python3 ~/clawd/skills/flowcrawl/scripts/flowcrawl.py https://example.com --json

Add Alias (Recommended)

echo 'alias flowcrawl="python3 ~/clawd/skills/flowcrawl/scripts/flowcrawl.py"' >> ~/.zshrc
source ~/.zshrc

Then just: flowcrawl https://example.com

How It Works

FlowCrawl uses a 3-tier fetcher cascade. Starts fast, escalates only when blocked:

Tier	Method	Handles
1	Plain HTTP	Most sites, instant
2	Stealth + TLS spoof	Cloudflare, Imperva, basic WAFs
3	Full JS execution	SPAs, heavy JS, aggressive bot detection

Auto-detects blocking (403, 503, "Just a moment...") and escalates silently.

All Options

Flag	Description	Default
--deep	Spider whole site following internal links	off
--depth N	Max hop depth from start URL	3
--limit N	Max pages to crawl	50
--combine	Merge all pages into one file	off
--format md|txt	Output format	md
--output DIR	Output directory	./flowcrawl-output
--json	Structured JSON output	off
--quiet	Suppress progress logs	off