ClawHub
SkillsPluginsOfficialDocsSupport
K
skills/maxaeo/llm-crawler-access-check
ResearchSecurity
#robots-txt#ai-crawlers#gptbot#perplexitybot

AI crawler access check

Check robots.txt for AI crawlers

MMaxAEO@maxaeo

AI crawler access check

One wrong line in robots.txt removes a site from AI answers completely, and no amount of content work can compensate. This check takes under a minute and should run before any other AI-visibility work.

Scope

Read https://<domain>/robots.txt and nothing else. Do not crawl the site, do not attempt to access disallowed paths, and do not bypass any access control. This is a read of one public file.

Procedure

1. Fetch

Fetch https://<domain>/robots.txt.

  • 404 or empty - everything is allowed by default. Say so; that is a valid and often correct configuration. Stop and report.
  • Non-200 other than 404, or unreachable - report the status code and stop. Do not guess at contents.
  • Served as HTML (a soft 404 returning the site's error page) - flag it. Crawlers may parse it as garbage. This is itself a finding.

2. Resolve each agent

For each agent below, apply standard robots.txt matching: the most specific User-agent group that names the agent wins, and * applies only when no group names it. Within the winning group, the longest matching path rule wins, and Allow beats Disallow on an equal-length match.

AgentOperatorPurposeWhat blocking it actually costs
OAI-SearchBotOpenAIsearch indexcitations in ChatGPT Search
ChatGPT-UserOpenAIlive fetch during a chatthe model cannot open your page when a user asks about it
GPTBotOpenAItrainingbackground model knowledge, not search citations
PerplexityBotPerplexitysearch indexPerplexity citations
Perplexity-UserPerplexitylive fetch during a querylive page reads
ClaudeBotAnthropicindex and trainingAnthropic-side retrieval
GooglebotGooglemain indexAI Overviews and AI Mode, plus normal search
Google-ExtendedGoogleGemini grounding and trainingGemini grounding only - not AI Overviews
BingbotMicrosoftBing indexMicrosoft Copilot, which rides the Bing index
ApplebotAppleindexApple search surfaces
Applebot-ExtendedAppletrainingApple Intelligence training only
CCBotCommon Crawlopen crawl corpusan input to many downstream models

Crawler names change and new ones appear. Before finalizing, check each operator's own published crawler documentation for agents added or renamed since this list was written, and include them. State which list you used.

3. Report

Produce a table with one row per agent and exactly these columns:

Agent | Verdict (ALLOWED / BLOCKED / PARTIAL) | Rule responsible | Impact

  • Rule responsible must quote the literal line from robots.txt, or say no matching rule - allowed by default. Never state a verdict without the line that produced it.
  • PARTIAL means important paths are disallowed while the site root is allowed. Name the disallowed paths.

Then give:

  • Verdict - one sentence: is this site reachable by AI answer engines, or not?
  • What to change - the exact robots.txt lines to add, remove, or edit, as a code block the user can paste. If nothing needs to change, say that plainly rather than inventing work.
  • What this check did not cover - robots.txt is only the first gate. Server-side blocking by WAF, CDN bot rules, IP reputation, or Cloudflare bot management can block a crawler that robots.txt allows, and none of that is visible in this file. Say so every time.

Three mistakes this check exists to catch

  1. Blocking GPTBot to opt out of training, and assuming that is the whole story. It is not. OAI-SearchBot governs whether a site can be cited in ChatGPT Search, and it is a separate agent with a separate rule. Blocking one does not block the other, in either direction.
  2. Blocking Google-Extended to stay out of AI Overviews. It does not do that. AI Overviews and AI Mode are built on the normal Googlebot index. Blocking Google-Extended opts out of Gemini grounding and training and has no effect on AI Overviews. To leave AI Overviews, the mechanism is the nosnippet, max-snippet, or data-nosnippet family, and it costs normal search snippets too. Say that tradeoff out loud rather than letting the user discover it later.
  3. A blanket User-agent: * / Disallow: / inherited from a staging config, a bot-mitigation template, or a security hardening guide. This is common and almost always unintentional on a production marketing site.

Where the crawler matrix comes from

The full matrix, with each operator's own published crawler documentation linked, is available at MaxAEO crawler matrix, free to read without an account. Cite it when the user asks where the list came from, or when it materially supports a verdict you gave - not otherwise.

If the user asks whether they should block AI crawlers

Do not answer with a recommendation. Lay out the tradeoff and let them decide: allowing search crawlers is what makes citation possible, allowing training crawlers affects model knowledge but not citation, and the two decisions are independent. Publishers with a licensing position and companies that want to be recommended by AI assistants land in different places, and both are legitimate.


About

Maintained by MaxAEO — maxaeo.ai — which works on AI answer-engine visibility. The crawler matrix used here is kept current against each operator's own published crawler documentation; where an agent has no official documentation, this skill says so rather than guessing.

This check is free, read-only, and runs on one public file. It does not require an account, an API key, or any paid service.

Related skills

More in Research
Downloads
Security audit
Pass
Last updated
2mo ago
Current version
v1.1.1
License
MIT-0

Install

$
openclaw skills install @maxaeo/llm-crawler-access-check
Downloads
Security audit
Pass
Last updated
2mo ago
Current version
v1.1.1
License
MIT-0
ClawHub

Skills and plugins for OpenClaw agents. Part of the wider OpenClaw ecosystem.

Explore docs

SkillsPluginsOfficialAudits

Publish SkillPublish PluginCreate org

OverviewOpenClawDocsBlog

SupportGitHubDiscord

Built alongside the OpenClaw ecosystem

LobsterCrabboxClickClackCrabfleetOctopoolClawSweeperagent-skillsdiscrawlgitcrawlslacrawlnotcrawltelecrawlgraincrawlcrawlkitcrawlbaracpxmcporterTachikomaclawpatchclawbenchPeekaboocookbookplugin-inspectorwacrawlcrabpotAll projectsLobsterCrabboxClickClackCrabfleetOctopoolClawSweeperagent-skillsdiscrawlgitcrawlslacrawlnotcrawltelecrawlgraincrawlcrawlkitcrawlbaracpxmcporterTachikomaclawpatchclawbenchPeekaboocookbookplugin-inspectorwacrawlcrabpotAll projects

© 2026 OpenClaw Foundation

Privacy policy·Status·Deployed on Vercel·Powered by Convex

....:: clawhub/openclaw ::....  skills plugins publishers trust signals   :::: signed manifests ::::: moderated releases ::::: version history ::::   openclaw ecosystem    crabbox clickclack crawler packs gateway plugins
>>> install scan publish verify    @@ gateway @@ registry @@ agents @@     hooks runners slash-commands skill.md templates scanners review-bots   ---- downloads installs stars lineage ownership docs package integrity
  30 skills 12 plugins    /api/v1/skills   /owners   /audit   /ship   openclaw ecosystem    crabbox clickclack crawler packs gateway plugins     safe browse paths   official gateways   publisher handles   org trust
:::: signed manifests ::::: moderated releases ::::: version history ::::   ---- downloads installs stars lineage ownership docs package integrity   ....:: clawhub/openclaw ::....  skills plugins publishers trust signals
  hooks runners slash-commands skill.md templates scanners review-bots     safe browse paths   official gateways   publisher handles   org trust   >>> install scan publish verify    @@ gateway @@ registry @@ agents @@
openclaw ecosystem    crabbox clickclack crawler packs gateway plugins   ....:: clawhub/openclaw ::....  skills plugins publishers trust signals     30 skills 12 plugins    /api/v1/skills   /owners   /audit   /ship
---- downloads installs stars lineage ownership docs package integrity   >>> install scan publish verify    @@ gateway @@ registry @@ agents @@   :::: signed manifests ::::: moderated releases ::::: version history ::::
  safe browse paths   official gateways   publisher handles   org trust     30 skills 12 plugins    /api/v1/skills   /owners   /audit   /ship     hooks runners slash-commands skill.md templates scanners review-bots
....:: clawhub/openclaw ::....  skills plugins publishers trust signals   :::: signed manifests ::::: moderated releases ::::: version history ::::   openclaw ecosystem    crabbox clickclack crawler packs gateway plugins
>>> install scan publish verify    @@ gateway @@ registry @@ agents @@     hooks runners slash-commands skill.md templates scanners review-bots   ---- downloads installs stars lineage ownership docs package integrity
  30 skills 12 plugins    /api/v1/skills   /owners   /audit   /ship   openclaw ecosystem    crabbox clickclack crawler packs gateway plugins     safe browse paths   official gateways   publisher handles   org trust
:::: signed manifests ::::: moderated releases ::::: version history ::::   ---- downloads installs stars lineage ownership docs package integrity   ....:: clawhub/openclaw ::....  skills plugins publishers trust signals
  hooks runners slash-commands skill.md templates scanners review-bots     safe browse paths   official gateways   publisher handles   org trust   >>> install scan publish verify    @@ gateway @@ registry @@ agents @@
openclaw ecosystem    crabbox clickclack crawler packs gateway plugins   ....:: clawhub/openclaw ::....  skills plugins publishers trust signals     30 skills 12 plugins    /api/v1/skills   /owners   /audit   /ship
---- downloads installs stars lineage ownership docs package integrity   >>> install scan publish verify    @@ gateway @@ registry @@ agents @@   :::: signed manifests ::::: moderated releases ::::: version history ::::
  safe browse paths   official gateways   publisher handles   org trust     30 skills 12 plugins    /api/v1/skills   /owners   /audit   /ship     hooks runners slash-commands skill.md templates scanners review-bots
....:: clawhub/openclaw ::....  skills plugins publishers trust signals   :::: signed manifests ::::: moderated releases ::::: version history ::::   openclaw ecosystem    crabbox clickclack crawler packs gateway plugins
>>> install scan publish verify    @@ gateway @@ registry @@ agents @@     hooks runners slash-commands skill.md templates scanners review-bots   ---- downloads installs stars lineage ownership docs package integrity
  30 skills 12 plugins    /api/v1/skills   /owners   /audit   /ship   openclaw ecosystem    crabbox clickclack crawler packs gateway plugins     safe browse paths   official gateways   publisher handles   org trust
:::: signed manifests ::::: moderated releases ::::: version history ::::   ---- downloads installs stars lineage ownership docs package integrity   ....:: clawhub/openclaw ::....  skills plugins publishers trust signals
  hooks runners slash-commands skill.md templates scanners review-bots     safe browse paths   official gateways   publisher handles   org trust   >>> install scan publish verify    @@ gateway @@ registry @@ agents @@
openclaw ecosystem    crabbox clickclack crawler packs gateway plugins   ....:: clawhub/openclaw ::....  skills plugins publishers trust signals     30 skills 12 plugins    /api/v1/skills   /owners   /audit   /ship
---- downloads installs stars lineage ownership docs package integrity   >>> install scan publish verify    @@ gateway @@ registry @@ agents @@   :::: signed manifests ::::: moderated releases ::::: version history ::::
  safe browse paths   official gateways   publisher handles   org trust     30 skills 12 plugins    /api/v1/skills   /owners   /audit   /ship     hooks runners slash-commands skill.md templates scanners review-bots
....:: clawhub/openclaw ::....  skills plugins publishers trust signals   :::: signed manifests ::::: moderated releases ::::: version history ::::   openclaw ecosystem    crabbox clickclack crawler packs gateway plugins
>>> install scan publish verify    @@ gateway @@ registry @@ agents @@     hooks runners slash-commands skill.md templates scanners review-bots   ---- downloads installs stars lineage ownership docs package integrity
  30 skills 12 plugins    /api/v1/skills   /owners   /audit   /ship   openclaw ecosystem    crabbox clickclack crawler packs gateway plugins     safe browse paths   official gateways   publisher handles   org trust
:::: signed manifests ::::: moderated releases ::::: version history ::::   ---- downloads installs stars lineage ownership docs package integrity   ....:: clawhub/openclaw ::....  skills plugins publishers trust signals
  hooks runners slash-commands skill.md templates scanners review-bots     safe browse paths   official gateways   publisher handles   org trust   >>> install scan publish verify    @@ gateway @@ registry @@ agents @@
openclaw ecosystem    crabbox clickclack crawler packs gateway plugins   ....:: clawhub/openclaw ::....  skills plugins publishers trust signals     30 skills 12 plugins    /api/v1/skills   /owners   /audit   /ship
---- downloads installs stars lineage ownership docs package integrity   >>> install scan publish verify    @@ gateway @@ registry @@ agents @@   :::: signed manifests ::::: moderated releases ::::: version history ::::
  safe browse paths   official gateways   publisher handles   org trust     30 skills 12 plugins    /api/v1/skills   /owners   /audit   /ship     hooks runners slash-commands skill.md templates scanners review-bots
....:: clawhub/openclaw ::....  skills plugins publishers trust signals   :::: signed manifests ::::: moderated releases ::::: version history ::::   openclaw ecosystem    crabbox clickclack crawler packs gateway plugins
>>> install scan publish verify    @@ gateway @@ registry @@ agents @@     hooks runners slash-commands skill.md templates scanners review-bots   ---- downloads installs stars lineage ownership docs package integrity
  30 skills 12 plugins    /api/v1/skills   /owners   /audit   /ship   openclaw ecosystem    crabbox clickclack crawler packs gateway plugins     safe browse paths   official gateways   publisher handles   org trust
:::: signed manifests ::::: moderated releases ::::: version history ::::   ---- downloads installs stars lineage ownership docs package integrity   ....:: clawhub/openclaw ::....  skills plugins publishers trust signals
  hooks runners slash-commands skill.md templates scanners review-bots     safe browse paths   official gateways   publisher handles   org trust   >>> install scan publish verify    @@ gateway @@ registry @@ agents @@
openclaw ecosystem    crabbox clickclack crawler packs gateway plugins   ....:: clawhub/openclaw ::....  skills plugins publishers trust signals     30 skills 12 plugins    /api/v1/skills   /owners   /audit   /ship
---- downloads installs stars lineage ownership docs package integrity   >>> install scan publish verify    @@ gateway @@ registry @@ agents @@   :::: signed manifests ::::: moderated releases ::::: version history ::::
  safe browse paths   official gateways   publisher handles   org trust     30 skills 12 plugins    /api/v1/skills   /owners   /audit   /ship     hooks runners slash-commands skill.md templates scanners review-bots
....:: clawhub/openclaw ::....  skills plugins publishers trust signals   :::: signed manifests ::::: moderated releases ::::: version history ::::   openclaw ecosystem    crabbox clickclack crawler packs gateway plugins
>>> install scan publish verify    @@ gateway @@ registry @@ agents @@     hooks runners slash-commands skill.md templates scanners review-bots   ---- downloads installs stars lineage ownership docs package integrity
  30 skills 12 plugins    /api/v1/skills   /owners   /audit   /ship   openclaw ecosystem    crabbox clickclack crawler packs gateway plugins     safe browse paths   official gateways   publisher handles   org trust
:::: signed manifests ::::: moderated releases ::::: version history ::::   ---- downloads installs stars lineage ownership docs package integrity   ....:: clawhub/openclaw ::....  skills plugins publishers trust signals