ClawHub

Web Site or Domain Name Basic Information Scanner

Security

Comprehensive website analysis and scanning tool. Analyzes IP (IPv4/IPv6), DNS records, WHOIS data, website content (including Schema.org JSON-LD, robots.txt, llms.txt, sitemap.xml), SEO metrics, and third-party data (Google index). Supports deep scanning with Playwright for client-side rendered pages. Generates detailed technical reports.

Install

openclaw skills install website-scanner

Website Scanner

Comprehensive website analysis tool that scans domains and generates detailed technical reports.

When to Use

Use this skill when you need to:

  • Analyze a website's technical infrastructure (IP, DNS, WHOIS)
  • Audit website content and SEO
  • Extract structured data (Schema.org JSON-LD)
  • Check robots.txt, llms.txt, sitemap.xml
  • Perform deep scanning of multiple pages
  • Generate comprehensive website reports

Prerequisites

Required System Tools

  • dig - DNS lookup utility
  • whois - WHOIS query tool

Python Dependencies

pip install requests beautifulsoup4 aiohttp

Optional (for deep scanning)

  • Playwright (for JavaScript-rendered pages)
pip install playwright
playwright install chromium

Usage

Basic Scan

python3 scripts/scan.py example.com

Deep Scan (follow sitemap)

python3 scripts/scan.py example.com --deep --max-pages 10

Save Reports

python3 scripts/scan.py example.com --output report.json --markdown report.md

Features

1. Network Analysis

  • IP Resolution: IPv4 and IPv6 addresses
  • Geolocation: Server location, ASN, organization
  • DNS Records: A, AAAA, MX, NS, TXT, CNAME, SOA
  • WHOIS Data: Registrar, creation/expiration dates, status

2. Content Analysis

  • Homepage Metadata: Title, description, viewport, charset
  • Heading Structure: H1-H6 distribution
  • Links: Internal vs external count
  • Images: Alt text coverage
  • Scripts/Styles: Resource count

3. Structured Data

  • Schema.org JSON-LD: Extract all structured data schemas
  • Open Graph: Meta tag analysis
  • Twitter Cards: Social media metadata

4. SEO Analysis

  • SEO Score: 0-100 rating
  • Issues: Missing tags, optimization problems
  • Recommendations: Actionable improvements

5. Crawler Files

  • robots.txt: Fetch and parse
  • llms.txt: AI/LLM instructions
  • sitemap.xml: Parse URL structure

6. Third-Party Data

  • Google Index: Check indexed pages (approximate)
  • Server Detection: Technology stack identification

7. Deep Scanning

  • Follow sitemap URLs
  • Analyze multiple pages
  • Client-side rendering support (with Playwright)

Output Format

Console Output

Human-readable markdown report with:

  • IP and network information
  • DNS record summary
  • WHOIS details
  • Content statistics
  • SEO score and issues
  • Third-party metrics

JSON Export

{
  "scan_info": {
    "url": "https://example.com",
    "domain": "example.com",
    "scan_date": "2026-04-02T10:00:00"
  },
  "ip_info": { ... },
  "dns_info": { ... },
  "whois_info": { ... },
  "content_analysis": { ... },
  "seo": { ... },
  ...
}

Markdown Export

Formatted report suitable for documentation or sharing.

Examples

Example 1: Quick Scan

python3 scripts/scan.py hugogu.cn

Output includes:

  • IP addresses and location
  • DNS configuration
  • WHOIS registration info
  • Homepage SEO analysis

Example 2: Full Audit

python3 scripts/scan.py hugogu.cn --deep --max-pages 20 \
  --output audit.json --markdown audit.md

Performs comprehensive analysis including:

  • All network information
  • Deep page crawling
  • SEO audit
  • Structured data extraction
  • Full report export

Limitations

  1. Rate Limiting: Some queries (WHOIS, DNS) may be rate-limited
  2. JavaScript Rendering: Basic scan doesn't execute JS (use --deep with Playwright)
  3. Third-Party APIs: Traffic estimation requires paid API keys
  4. Google Index: Approximate only (scrapes search results)

Troubleshooting

"dig command not found"

# Ubuntu/Debian
sudo apt-get install dnsutils

# macOS
brew install bind

# CentOS/RHEL
sudo yum install bind-utils

"whois command not found"

# Ubuntu/Debian
sudo apt-get install whois

# macOS
brew install whois

DNS Resolution Fails

Check if domain is accessible:

nslookup example.com
dig example.com

Architecture

┌─────────────────┐
│ WebsiteScanner  │
└────────┬────────┘
         │
    ┌────┴────┬──────────┬──────────┐
    │         │          │          │
┌───▼───┐ ┌──▼────┐ ┌───▼───┐ ┌───▼────┐
│ IP    │ │ DNS   │ │ WHOIS │ │ Content│
│Analyzer│ │Analyzer│ │Analyzer│ │Analyzer│
└───────┘ └───────┘ └───────┘ └───────┘
                                   │
                          ┌────────┼────────┐
                          │        │        │
                      ┌───▼───┐ ┌─▼────┐ ┌─▼────┐
                      │ SEO   │ │ JSON │ │ Deep │
                      │Analyzer│ │ -LD  │ │ Scan │
                      └───────┘ └──────┘ └──────┘

Integration with OpenClaw

This skill can be invoked from OpenClaw to:

  1. Scan competitor websites
  2. Audit your own sites
  3. Research domain infrastructure
  4. Generate technical documentation

Example OpenClaw workflow:

User: "Scan hugogu.cn and tell me about its infrastructure"
→ Run scanner
→ Analyze results
→ Generate summary
More in Security