Robots Ai

Analyze and generate robots.txt files with AI crawler awareness. Detect which AI bots (GPTBot, ClaudeBot, PerplexityBot, Google-Extended, etc.) are blocked o...

Security Scan

The skill's instructions, capabilities, and resource requests are coherent with its stated purpose (analyzing and generating robots.txt files) and it does not request unrelated credentials or install code.

Purpose & Capability

Name/description (robots.txt analysis/generation with AI-crawler awareness) match the SKILL.md instructions. The skill requires no binaries, credentials, or config paths — appropriate for an instruction-only robots.txt helper.

Instruction Scope

SKILL.md explicitly tells the agent to fetch /robots.txt from arbitrary websites, parse User-agent/Allow/Disallow rules, and produce suggested robots.txt content. That behavior is expected for this purpose, but it implies the agent will perform network fetches of target sites (and may reveal provided site URLs to any runtime network logs). The instructions do not request unrelated files, credentials, or external endpoints beyond the target site's robots.txt.

Install Mechanism

No install spec and no code files are present — lowest-risk instruction-only skill. Nothing is downloaded or written to disk by an installer.

Credentials

The skill declares no required environment variables or credentials. No disproportionate access to secrets or unrelated services is requested.

Persistence & Privilege

always is false and the skill does not request special persistent privileges or modifications to other skills/configs. Autonomous invocation is allowed by default but not excessive here.

Assessment

This skill appears coherent and safe in intent. Before installing, note that: (1) it will fetch robots.txt from arbitrary URLs you provide — ensure you are comfortable with the agent making outbound requests to those sites and with any logs that may record those URLs; (2) generated recommendations (e.g., 'block all AI' or blocking Googlebot) can have major consequences like removing your site from search/indexing — review generated robots.txt carefully before applying it to production; (3) verify the bot list if you rely on it for compliance, since crawler names and behaviors can change. If you plan to use it on private/internal sites, ensure the agent has appropriate network access and that no sensitive URLs are shared accidentally.

Analyze, audit, and generate robots.txt files with full awareness of 20+ AI crawlers.

Capabilities

• Analyze any website's robots.txt to see which AI bots are blocked/allowed
• Generate a robots.txt with toggleable AI bot blocking
• Audit existing robots.txt for completeness and issues
• List all known AI crawlers with their user-agents, companies, and documentation links

AI Bots Database

You know about these AI crawlers and their user-agents:

Bot	User-Agent	Company	Type
GPTBot	GPTBot	OpenAI	AI Crawler
ChatGPT-User	ChatGPT-User	OpenAI	AI Search
OAI-SearchBot	OAI-SearchBot	OpenAI	AI Search
ClaudeBot	ClaudeBot	Anthropic	AI Crawler
anthropic-ai	anthropic-ai	Anthropic	AI Crawler
Google-Extended	Google-Extended	Google	AI Crawler
PerplexityBot	PerplexityBot	Perplexity	AI Search
CCBot	CCBot	Common Crawl	AI Crawler
Bytespider	Bytespider	ByteDance	AI Crawler
Diffbot	Diffbot	Diffbot	AI Crawler
cohere-ai	cohere-ai	Cohere	AI Crawler
Amazonbot	Amazonbot	Amazon	AI Crawler
Meta-ExternalAgent	Meta-ExternalAgent	Meta	AI Crawler
Meta-ExternalFetcher	Meta-ExternalFetcher	Meta	AI Crawler
Applebot-Extended	Applebot-Extended	Apple	AI Crawler
YouBot	YouBot	You.com	AI Search
Timpibot	Timpibot	Timpi	AI Crawler
img2dataset	img2dataset	Open Source	AI Crawler

Important Notes

• Google-Extended controls Gemini training access but does NOT affect Google Search indexing
• Blocking Googlebot removes the site from Google Search entirely — never do this unless explicitly asked
• CCBot feeds Common Crawl, which is used by many AI companies for training data
• Bytespider (ByteDance) and Timpibot are commonly blocked by default due to aggressive crawling

How to Analyze

When asked to analyze a robots.txt:

1. Fetch the robots.txt from the URL (append /robots.txt if not included)
2. Parse all User-agent directives and their Allow/Disallow rules
3. Check each AI bot against the rules
4. Report: which bots are blocked, which are allowed, and any issues found
5. Suggest improvements if relevant

How to Generate

When asked to generate a robots.txt:

1. Ask which AI bots to block (or accept "block all AI" / "allow all AI")
2. Ask for sitemap URL(s)
3. Ask for any custom rules (e.g., Disallow: /admin/)
4. Generate clean robots.txt with comments explaining each section
5. Always include User-agent: * with Allow: / as the default
6. Group blocked AI bots together with comments
7. Add sitemap directives at the end

Output Format

Always format the generated robots.txt in a code block with syntax highlighting. Add comments explaining what each section does. Example:

# Allow all crawlers by default
User-agent: *
Allow: /

# Block AI training crawlers
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

# Sitemap
Sitemap: https://example.com/sitemap.xml