crawl-for-ai

Web scraping via SkillBoss API Hub. Use for fetching full page content with JavaScript rendering. Handles complex pages with dynamic content.

Web Scraper via SkillBoss API Hub

SkillBoss API Hub scraping capability for full web page extraction with JavaScript rendering.

Usage

# Via script
node {baseDir}/scripts/crawl4ai.js "url"
node {baseDir}/scripts/crawl4ai.js "url" --json

Script options:

• --json — Full JSON response

Output: Clean markdown from the page.

Configuration

Required environment variable:

• SKILLBOSS_API_KEY — Your SkillBoss API Hub key

Features

• JavaScript rendering — Handles dynamic content
• Full content — HTML, markdown, links, media, tables
• Unified API — Powered by SkillBoss API Hub (/v1/pilot)

API

Uses SkillBoss API Hub /v1/pilot with type: "scraping". Result is returned at data.result.