Crawler

Web crawling and scraping reference — robots.txt protocol, Scrapy framework, anti-bot detection, headless browsers, and legal considerations

Web crawling and scraping reference — robots.txt protocol, Scrapy framework, anti-bot detection, headless browsers, and legal considerations. No API keys or credentials required — outputs reference documentation only.

Commands

Command	Description
intro	Crawling vs scraping, robots.txt, sitemap
standards	HTTP caching, structured data, meta tags
troubleshooting	Anti-bot detection, JS rendering, encoding
performance	Concurrency, dedup, incremental, distributed
security	Legal landscape, ethical guidelines, proxies
migration	BeautifulSoup to Scrapy, requests to Playwright
cheatsheet	Scrapy commands, CSS/XPath, curl, user-agents
faq	Legality, JS pages, blocking, storage

Output Format

All commands output plain-text reference documentation via heredoc. No external API calls, no credentials needed, no network access.

---

Powered by BytesAgain | bytesagain.com | [e-mail]