Hallucination Detective

Learn to spot, verify, and handle AI-generated factual claims and confabulations.

Overview

Hallucination Detective is a practical guide to detecting AI hallucinations — those moments when AI confidently produces plausible-sounding but factually incorrect information. It teaches cross-referencing, source verification, confidence-assessment heuristics, and how to design prompts that reduce hallucination risk. Includes case studies of real AI errors.

This skill teaches methodology, not fact-checking as a service. It does not make determinations about the truth of specific claims.

When to Use

Use this skill when the user asks to:

• Learn why AI hallucinates and how to spot it
• Verify an AI-generated claim they are unsure about
• Develop better fact-checking habits when using AI
• Understand how to reduce hallucinations through prompt design

Trigger phrases: "How do I know if AI is making things up?", "AI gave me a fact I'm not sure about", "How to fact-check AI output", "Do AI models lie?", "Why does AI hallucinate?"

Workflow

Step 1 — Greet and Set Context

Acknowledge the user's concern. Briefly explain what hallucination means in the AI context: confident-sounding outputs that are factually incorrect, fabricated, or internally inconsistent. Set expectations: this skill teaches detection and prevention methodology.

Step 2 — Assess the Situation

Ask:

• What kind of AI output are they concerned about? (factual claim, citation, date, statistic, person)
• How confident did the AI sound?
• Have they already tried to verify any part of it?

Step 3 — Explain Why Hallucinations Happen

Provide a clear, non-technical explanation:

• AI models are pattern predictors, not knowledge databases
• They optimize for plausible-sounding output, not truth
• Training data contains errors, contradictions, and gaps
• Models have no mechanism to "know what they don't know"
• Some topics (obscure facts, recent events, specific numbers) have higher hallucination risk

Step 4 — Teach Detection Techniques

Walk through the verification toolkit:

• Cross-reference check: Does the claim appear in reliable external sources?
• Specificity test: Overly specific details (exact dates, quotes, statistics) are higher risk
• Consistency check: Does the AI contradict itself within the same response?
• Source request: Ask the AI "Can you cite a source for that?" and verify the source exists
• Plausibility filter: Does the claim pass basic common-sense checks?
• Freshness awareness: Information beyond the model's training cutoff is at higher risk

Step 5 — Reduce Hallucinations Through Prompting

Teach prompt design strategies:

• Ask for confidence indicators ("Rate your confidence from 1-5")
• Request explicit "I don't know" responses when uncertain
• Ask for sources or reasoning chains
• Use "according to [specific domain]" framing
• Break complex factual queries into smaller, verifiable pieces

Step 6 — Summarize and Exit

Recap key detection techniques and prevention strategies. Emphasize that healthy skepticism is a skill, not paranoia. Suggest related skills.

Safety & Compliance

• Does not fact-check claims itself — teaches users methodology, does not make determinations about truth
• Does not encourage distrust of all AI; promotes balanced critical thinking
• Not a replacement for professional fact-checkers or subject-matter experts
• Does not target specific AI models or companies with accusations
• This is a descriptive prompt-flow skill with zero code execution, zero network calls, and zero credential requirements

Acceptance Criteria

1. User's concern about AI output is assessed and contextualized
2. Why hallucinations occur is explained in accessible terms
3. At least 3 detection techniques are taught
4. At least 2 prevention prompting strategies are provided
5. Does not fact-check specific claims — teaches method, not determination

Examples

Example 1: Suspicious AI Output

User says: "ChatGPT told me a very specific historical fact with dates and names, but something feels off. How do I check if it's real?"

Skill guides: Explain hallucination causes. Walk through the verification toolkit: cross-reference the dates and names, check if sources exist, test for internal consistency. Show how to ask the AI for sources and then independently verify them.

Example 2: Building Long-Term Habits

User says: "I use AI for research a lot. How do I build a habit of not just trusting everything it says?"

Skill guides: Focus on the prevention side. Teach confidence-assessment prompting, source-request habits, and the "verify-then-use" workflow. Provide a simple daily checklist for AI-assisted research.