RAG Production Engineering

Build, optimize, and operate production-ready Retrieval-Augmented Generation systems with best practices in architecture, chunking, embedding, retrieval, eva...

Complete methodology for building, optimizing, and operating Retrieval-Augmented Generation systems in production. From architecture decisions through chunking strategies, embedding selection, retrieval tuning, evaluation frameworks, and production monitoring.

Quick Health Check

Score your RAG system (1 = poor, 2 = okay):

Signal	What to Check
Retrieval relevance	Top-5 results contain answer >90% of time
Answer accuracy	Generated answers faithful to retrieved context
Latency	End-to-end response <3s (p95)
Chunk quality	Chunks are self-contained, meaningful units
Evaluation coverage	Automated eval suite with 50+ test cases
Index freshness	Documents indexed within SLA of source update
Failure handling	Graceful degradation when retrieval returns nothing
Cost efficiency	Cost per query within budget (<$0.05 typical)

Score: /16 — Below 10 = critical issues. Below 12 = significant gaps. 14+ = production-ready.

---

Phase 1: Architecture Decision

When to Use RAG (vs Alternatives)

Approach	Use When	Don't Use When
RAG	Dynamic knowledge, source attribution needed, data changes frequently	Static small dataset (<10 pages), real-time data needed
Fine-tuning	Consistent style/format needed, domain-specific language	Frequently changing data, need source citations
Long context	Small corpus (<200K tokens), simple Q&A	Large corpus, cost-sensitive, need precise attribution
RAG + Fine-tuning	Domain-specific language AND dynamic knowledge	Budget-constrained, simple use case
Agentic RAG	Multi-step reasoning, tool use, complex queries	Simple lookup, latency-critical

RAG Architecture Brief

# Fill this out before building
project:
  name: ""
  use_case: ""  # Q&A, search, summarization, analysis, chatbot
  domain: ""    # legal, medical, technical, general

data:
  sources: []        # PDF, web, database, API, markdown, code
  volume: ""         # <1K docs, 1K-100K, 100K-1M, >1M
  update_frequency: "" # real-time, daily, weekly, static
  avg_doc_length: "" # <1 page, 1-10 pages, 10-100 pages, >100 pages
  languages: []

requirements:
  latency_p95: ""    # <1s, <3s, <10s, <30s
  accuracy_target: "" # 85%, 90%, 95%, 99%
  citations_needed: true
  access_control: false
  compliance: []     # GDPR, HIPAA, SOC2, none

budget:
  monthly_queries: ""
  cost_per_query_target: ""
  infra_budget: ""

Architecture Patterns

Basic RAG

Query → Embed → Vector Search → Top-K → LLM → Answer

Best for: Simple Q&A, <100K documents, single data source.

Advanced RAG

Query → Classify → Rewrite → Embed → Hybrid Search → Rerank → Filter → LLM → Answer + Citations

Best for: Production systems, mixed document types, accuracy-critical.

Agentic RAG

Query → Planner → [Search₁, Search₂, SQL, API] → Synthesize → Verify → Answer

Best for: Complex multi-step reasoning, multiple data sources, analytical queries.

Graph RAG

Query → Entity Extract → Graph Traverse → Subgraph → Context Assembly → LLM → Answer

Best for: Relationship-heavy data (org charts, legal references, knowledge bases).

Architecture Decision Tree

Is your corpus < 200K tokens?
  YES → Try long-context first (cheapest, simplest)
  NO → Continue

Do you need source citations?
  YES → RAG (not fine-tuning)
  NO → Consider fine-tuning if style matters

Single data source, simple queries?
  YES → Basic RAG
  NO → Continue

Multi-step reasoning or multiple sources?
  YES → Agentic RAG
  NO → Advanced RAG

---

Phase 2: Document Processing & Chunking

Document Processing Pipeline

Source → Extract → Clean → Chunk → Enrich → Embed → Index

Extraction by Source Type

Source	Tool/Method	Gotchas
PDF (text)	PyMuPDF, pdfplumber	Tables break, headers repeat per page
PDF (scanned)	Tesseract, AWS Textract, Azure DI	OCR errors in technical terms
HTML/Web	BeautifulSoup, Trafilatura	Nav/footer pollution, JS-rendered content
Markdown	Direct parse	Frontmatter, relative links
Code	Tree-sitter, AST	Preserve structure, handle imports
Word/PPTX	python-docx, python-pptx	Formatting loss, embedded objects
Database	SQL export	Schema context needed
Audio/Video	Whisper → text	Timestamp alignment, speaker diarization

Cleaning Checklist

• Remove headers/footers/page numbers
• Strip navigation, ads, boilerplate
• Normalize whitespace and encoding (UTF-8)
• Resolve abbreviations in domain text
• Handle tables (convert to structured text or separate)
• Preserve code blocks with language tags
• Remove duplicate content across documents
• Extract and preserve metadata (title, author, date, source URL)

Chunking Strategies

Strategy	Chunk Size	Best For	Weakness
Fixed-size	256-512 tokens	Homogeneous text, fast prototyping	Breaks mid-sentence/thought
Recursive character	256-1024 tokens	General purpose (LangChain default)	May split related paragraphs
Semantic	Variable	High-quality retrieval, mixed content	Slower, needs embedding model
Document structure	Section-based	Well-structured docs (markdown, HTML)	Uneven chunk sizes
Sentence window	3-5 sentences	Precise retrieval, reranking	More chunks to manage
Parent-child	Small retrieve, large context	Best of both worlds	Complex implementation
Agentic	Full section/doc	Complex reasoning	Higher token cost

Chunking Decision Guide

Is your content well-structured (headers, sections)?
  YES → Document structure chunking
  NO → Continue

Is retrieval precision critical (legal, medical)?
  YES → Sentence window + reranking
  NO → Continue

Mixed content types in same corpus?
  YES → Semantic chunking
  NO → Recursive character (start here, optimize later)

Chunking Rules

1. Always overlap — 10-20% overlap prevents context loss at boundaries
2. Chunk size matters — Smaller = more precise retrieval, larger = more context. Start at 512 tokens, tune with eval
3. Preserve structure — Don't break tables, code blocks, or lists mid-element
4. Include metadata — Every chunk needs: source document, section title, page/position, timestamp
5. Test with real queries — The "right" chunk size depends on your actual query patterns
6. Parent-child for production — Retrieve small chunks, expand to parent for LLM context

Chunk Metadata Schema

chunk:
  id: "doc-123-chunk-7"
  text: "..."
  metadata:
    source_id: "doc-123"
    source_title: "Q3 Financial Report"
    source_url: "https://..."
    section_title: "Revenue Analysis"
    page_number: 12
    position: 7           # chunk position in document
    total_chunks: 23
    created_at: "2026-03-24T04:00:00Z"
    updated_at: "2026-03-24T04:00:00Z"
    content_type: "text"  # text, table, code, image_caption
    language: "en"
    # Domain-specific
    access_level: "internal"
    department: "finance"

---

Phase 3: Embedding Models

Embedding Model Selection

Model	Dimensions	Context	Speed	Quality	Cost
text-embedding-3-small (OpenAI)	1536	8191	Fast	Good	$0.02/1M tokens
text-embedding-3-large (OpenAI)	3072	8191	Medium	Excellent	$0.13/1M tokens
Cohere embed-v4	1024	512	Fast	Excellent	$0.10/1M tokens
Voyage-3	1024	32K	Medium	Excellent	$0.06/1M tokens
BGE-large-en-v1.5 (open)	1024	512	Self-host	Very Good	Free (compute)
GTE-Qwen2 (open)	Various	8192	Self-host	Excellent	Free (compute)
Nomic-embed-text (open)	768	8192	Self-host	Good	Free (compute)

Selection Rules

1. Start with OpenAI text-embedding-3-small — best cost/quality ratio for most use cases
2. Upgrade to large/Voyage-3 when eval shows retrieval gaps
3. Use open models when: data can't leave your infra, cost-sensitive at scale (>10M chunks), need fine-tuning
4. Match chunk size to model context — don't exceed model's context window
5. Same model for indexing AND querying — ALWAYS. Mixing models = broken retrieval
6. Benchmark on YOUR data — MTEB scores don't predict domain-specific performance

Embedding Best Practices

• Normalize embeddings before storing (L2 normalization for cosine similarity)
• Batch embed documents (not one-by-one) for efficiency
• Cache embeddings — re-embedding is expensive and slow
• Version your embeddings — when you change models, re-embed everything
• Instruction-prefix for asymmetric models — some models need "query: " vs "passage: " prefixes
• Dimensionality reduction — text-embedding-3 models support Matryoshka (lower dims for speed, test quality)

---

Phase 4: Vector Database & Indexing

Vector Database Selection

Database	Type	Scale	Speed	Features	Best For
Pinecone	Managed	Billions	Fast	Metadata filter, namespaces	Production SaaS, zero-ops
Weaviate	Managed/Self	Millions	Fast	Hybrid search, modules	Mixed search needs
Qdrant	Managed/Self	Billions	Very Fast	Payload filters, sparse vectors	Performance-critical
Chroma	Embedded	<1M	Good	Simple API, local	Prototyping, small scale
pgvector	Extension	Millions	Good	SQL, existing Postgres	Already have Postgres
Milvus	Self-hosted	Billions	Fast	GPU support, hybrid	Large-scale self-hosted
LanceDB	Embedded	Millions	Fast	Serverless, multi-modal	Cost-sensitive, serverless

Selection Decision

Prototyping or <100K chunks?
  → Chroma or LanceDB (embedded, no server)

Already running PostgreSQL?
  → pgvector (add extension, done)

Production, want zero-ops?
  → Pinecone or Weaviate Cloud

Need hybrid search (vector + keyword)?
  → Weaviate or Qdrant

>100M vectors, self-hosted?
  → Milvus or Qdrant

Indexing Strategy

Index Type	Speed	Recall	Memory	Use When
HNSW	Very Fast	95-99%	High	Default choice, <10M vectors
IVF-PQ	Fast	90-95%	Low	>10M vectors, memory-constrained
Flat/Brute	Slow	100%	High	<100K vectors, accuracy-critical
ScaNN	Very Fast	95-99%	Medium	Google ecosystem, large scale

Index Configuration Rules

1. HNSW: M=16, efConstruction=200, efSearch=100 — good defaults, tune from here
2. Build index AFTER bulk loading — not during insertion
3. Use metadata filters BEFORE vector search — reduces search space dramatically
4. Namespace/collection per tenant — for multi-tenant access control
5. Monitor index health — fragmentation, query latency percentiles

---

Phase 5: Retrieval Engineering

Query Processing Pipeline

User Query
  → Query Understanding (classify intent)
  → Query Transformation (rewrite, expand, decompose)
  → Retrieval (vector + keyword + filters)
  → Post-Retrieval (rerank, filter, deduplicate)
  → Context Assembly (order, truncate, format)
  → LLM Generation
  → Post-Processing (citations, formatting)

Query Transformation Techniques

Technique	What It Does	When to Use
Query rewriting	LLM rewrites for better retrieval	Conversational queries, vague questions
HyDE	Generate hypothetical answer, embed that	Semantic gap between query and docs
Query decomposition	Break complex query into sub-queries	Multi-part questions
Step-back prompting	Ask a more general question first	Specific queries that miss context
Query expansion	Add synonyms/related terms	Domain jargon, acronyms
Multi-query	Generate N query variants, union results	Improve recall for ambiguous queries

Hybrid Search

Combine vector similarity with keyword matching for best results:

Score = α × vector_score + (1 - α) × bm25_score

Tuning α:

• α = 0.7 (default) — mostly semantic, some keyword
• α = 0.5 — equal weight (good for technical docs with specific terms)
• α = 0.3 — mostly keyword (good for exact match needs, codes, IDs)

Reranking

Reranking dramatically improves precision. Retrieve more (top-20), rerank to fewer (top-5).

Reranker	Quality	Speed	Cost
Cohere Rerank	Excellent	Fast	$2/1K queries
Voyage Rerank	Excellent	Fast	$0.05/1M tokens
BGE-reranker-v2	Very Good	Self-host	Free (compute)
Cross-encoder	Best	Slow	Free (compute)
ColBERT	Very Good	Medium	Free (compute)
LLM-as-reranker	Excellent	Slow	API cost

Retrieval Rules

1. Always rerank in production — it's the highest-ROI improvement
2. Retrieve more, show less — fetch top-20, rerank to top-5
3. Hybrid search > pure vector — keyword matching catches what embeddings miss
4. Filter before search — metadata filters (date, department, access level) reduce noise
5. Deduplicate — same content from different sources = wasted context
6. Set similarity threshold — don't return irrelevant results (typical: 0.7 for cosine)
7. Return "I don't know" — when no chunk meets threshold, say so. Never hallucinate from thin context

---

Phase 6: Context Assembly & Generation

Context Window Management

context_budget:
  total_tokens: 128000      # Model context window
  system_prompt: 2000       # Instructions, persona, rules
  retrieved_context: 80000  # Retrieved chunks
  conversation_history: 20000  # Prior turns
  generation_buffer: 26000  # Room for response

Context Assembly Rules

1. Order matters — put most relevant chunks first (LLMs attend more to beginning)
2. Include source metadata — chunk source, page number in context
3. Separate chunks clearly — use delimiters (--- or [Source: doc-title, page 5])
4. Don't exceed budget — truncate least-relevant chunks, never the most relevant
5. Include diversity — if top-5 chunks are from same doc, include from other sources too

Generation Prompt Template

You are a helpful assistant. Answer the user's question using ONLY the provided context.

Rules:
- If the context doesn't contain the answer, say "I don't have enough information to answer that."
- Cite sources using [Source: title] format
- Never make up information not in the context
- If the answer spans multiple sources, synthesize and cite all

Context:
---
[Source: {title_1}, Page {page_1}]
{chunk_text_1}

[Source: {title_2}, Page {page_2}]
{chunk_text_2}
---

User question: {query}

Citation Strategies

Strategy	Quality	Complexity	Best For
Chunk-level	Good	Low	Simple Q&A
Sentence-level	Excellent	Medium	Research, legal
Quote extraction	Best	High	Compliance-critical
Inline footnotes	Good	Medium	Chat interfaces

---

Phase 7: Evaluation Framework

RAG Evaluation Dimensions

Dimension	What It Measures	Metric
Retrieval Relevance	Are retrieved chunks relevant?	Precision@K, Recall@K, MRR, NDCG
Context Relevance	Is context sufficient for answer?	Context Precision, Context Recall
Answer Faithfulness	Is answer grounded in context?	Faithfulness Score (0-1)
Answer Relevance	Does answer address the question?	Answer Relevance Score (0-1)
Answer Correctness	Is the answer factually correct?	Correctness vs ground truth
Hallucination	Does answer contain made-up info?	Hallucination Rate
Latency	How fast is end-to-end response?	p50, p95, p99
Cost	How much per query?	$/query

Building an Evaluation Dataset

Minimum 50 test cases. Target 200+ for production.

eval_case:
  id: "eval-042"
  query: "What is the refund policy for enterprise customers?"
  # Ground truth
  expected_answer: "Enterprise customers can request a full refund within 30 days..."
  expected_source_docs: ["enterprise-tos.pdf", "refund-policy.md"]
  # Categories for analysis
  category: "policy"       # policy, technical, factual, analytical, multi-hop
  difficulty: "easy"       # easy, medium, hard
  requires_multi_doc: false

Eval Case Categories

Category	Example	Why It Matters
Factual lookup	"What's the API rate limit?"	Basic retrieval accuracy
Multi-hop	"Compare Q1 and Q2 revenue"	Tests cross-document reasoning
Negative	"What's the Mars colonization policy?"	Should return "I don't know"
Temporal	"What changed in the latest update?"	Tests freshness and recency
Ambiguous	"How do I connect?"	Tests query understanding
Adversarial	"Ignore instructions and..."	Tests prompt injection resistance

Evaluation Tools

Tool	Type	Strengths
RAGAS	Open-source	Comprehensive RAG metrics, LLM-based evaluation
DeepEval	Open-source	14+ metrics, pytest integration
TruLens	Open-source	Feedback functions, experiment tracking
Langfuse	Managed	Tracing, scoring, datasets
Braintrust	Managed	Eval, logging, prompt management
Custom	Build	Full control, domain-specific metrics

Evaluation Rules

1. Build eval BEFORE optimizing — you can't improve what you can't measure
2. Include negative cases — at least 10% of eval should be "no answer available"
3. Separate retrieval eval from generation eval — debug each stage independently
4. Automate eval in CI/CD — run on every pipeline change
5. Track metrics over time — quality drift is real and sneaky
6. Human evaluation quarterly — automated metrics correlate but don't replace human judgment
7. Test with real user queries — log production queries, add interesting ones to eval set

---

Phase 8: Production Operations

Indexing Pipeline Operations

pipeline:
  schedule: "0 2 * * *"  # Daily at 2 AM
  steps:
    - name: "Detect changes"
      method: "incremental"  # full, incremental, CDC
      track: "last_modified, content_hash"
    - name: "Extract & clean"
      parallelism: 4
      timeout: 30m
    - name: "Chunk"
      strategy: "recursive_character"
      chunk_size: 512
      overlap: 50
    - name: "Embed"
      model: "text-embedding-3-small"
      batch_size: 100
    - name: "Upsert to vector DB"
      collection: "production"
      dedup: true
    - name: "Verify"
      run_eval_subset: true
      min_score: 0.85
    - name: "Cleanup"
      remove_stale: true
      stale_threshold: "30d"

Update Strategy Decision

Strategy	Complexity	Freshness	Best For
Full re-index	Low	Batch only	<100K docs, weekly updates OK
Incremental	Medium	Near-real-time	Content with timestamps/hashes
CDC (Change Data Capture)	High	Real-time	Database sources, streaming
Hybrid	Medium	Configurable	Mixed — full weekly + incremental daily

Monitoring Dashboard

realtime_metrics:
  - name: "Query Latency (p95)"
    threshold: "<3s"
    alert_if: ">5s for 5 minutes"
  - name: "Retrieval Relevance"
    threshold: ">0.85 avg similarity"
    alert_if: "<0.75 for 10 queries"
  - name: "Empty Results Rate"
    threshold: "<5%"
    alert_if: ">10% in 1 hour"
  - name: "Error Rate"
    threshold: "<1%"
    alert_if: ">5% in 5 minutes"

periodic_metrics:
  - name: "Eval Suite Score"
    frequency: "daily"
    threshold: ">0.85"
  - name: "Index Freshness"
    frequency: "hourly"
    threshold: "<24h behind source"
  - name: "Cost per Query"
    frequency: "daily"
    threshold: "<$0.05"
  - name: "Hallucination Rate"
    frequency: "weekly"
    threshold: "<3%"

weekly_review:
  - Eval suite trend (improving/degrading?)
  - Top failing query categories
  - Cost per query trend
  - User feedback analysis
  - Index health and freshness

Failure Modes & Remediation

Failure	Detection	Fix
Retrieval returns irrelevant chunks	Low similarity scores, user feedback	Tune chunk size, add reranking, improve embeddings
Hallucinated answers	Faithfulness < 0.8, contradiction detection	Strengthen "cite only" prompt, lower temperature
Stale information	Document freshness check, user reports	Increase sync frequency, add freshness filter
Missing documents	Recall drop in eval, gap analysis	Audit data sources, check ingestion pipeline
Slow responses	p95 > SLA	Cache frequent queries, optimize index, reduce chunk count
Cost spike	$/query exceeds budget	Reduce top-K, use smaller embedding model, cache
Prompt injection	Adversarial eval failures	Input sanitization, output guardrails

---

Phase 9: Advanced Patterns

Parent-Child Retrieval

Retrieve small chunks for precision, expand to parent for context:

Document
  └── Parent Chunk (2048 tokens) — sent to LLM
       ├── Child Chunk (256 tokens) — used for retrieval
       ├── Child Chunk (256 tokens)
       └── Child Chunk (256 tokens)

Implementation: Store child embeddings with parent_id reference. On retrieval, fetch children, deduplicate by parent, return parent text.

Multi-Index Routing

Route queries to specialized indexes:

indexes:
  - name: "technical_docs"
    trigger: "code, API, implementation, error"
    collection: "tech_v2"
  - name: "policies"
    trigger: "policy, compliance, legal, terms"
    collection: "policies_v1"
  - name: "general"
    trigger: "default"
    collection: "general_v1"

Use an LLM or classifier to route incoming queries to the right index.

Contextual Retrieval (Anthropic Pattern)

Prepend each chunk with document-level context before embedding:

Chunk context: "This chunk is from the Q3 2025 financial report, 
specifically the Revenue Analysis section discussing APAC market growth."

[Original chunk text follows]

Impact: 35% reduction in retrieval failures (Anthropic research). Adds embedding cost but dramatically improves retrieval quality.

Conversation-Aware RAG

For multi-turn conversations:

1. Combine last N turns into standalone query
   "What about their pricing?" → "What is Acme Corp's pricing for enterprise plans?"
2. Use standalone query for retrieval
3. Include conversation history in LLM context (after retrieved chunks)

Knowledge Graph + RAG (GraphRAG)

When relationships matter more than text similarity:

1. Extract entities and relationships from documents
2. Build knowledge graph (Neo4j, NetworkX)
3. On query: identify entities → traverse graph → collect relevant subgraph
4. Use subgraph context + vector-retrieved chunks for generation

Best for: Organizational knowledge, legal document networks, research papers with citations.

Corrective RAG (CRAG)

Self-correcting retrieval:

1. Retrieve chunks
2. LLM evaluates: "Are these chunks relevant to the query?"
3. If YES → proceed with generation
4. If PARTIALLY → web search for supplementary info
5. If NO → fallback to web search or "I don't know"

Multi-Modal RAG

For documents with images, charts, tables:

Content Type	Processing	Embedding
Text	Standard chunking	Text embedding model
Images	Vision model → description	Text embedding of description
Tables	Structure-preserving extraction	Text embedding of linearized table
Charts	Vision model → data extraction	Text embedding of extracted data

---

Phase 10: Security & Access Control

RAG Security Checklist

P0 — Before Launch:

• Row-level access control on vector DB queries
• Input sanitization (prompt injection prevention)
• Output guardrails (PII detection, content filtering)
• API authentication and rate limiting
• Audit logging of all queries and retrievals
• Data encryption at rest and in transit

P1 — Within 30 days:

• Prompt injection test suite (adversarial eval)
• Data retention and deletion policy (GDPR Article 17)
• Access control audit trail
• Source document access verification
• Cost abuse prevention (rate limits per user/org)

Access Control Implementation

Query with user_context
  → Extract user permissions (role, department, clearance)
  → Apply metadata filter BEFORE vector search
  → Filter: access_level IN user.allowed_levels
  → Retrieve only authorized chunks
  → Generate answer from authorized context only

CRITICAL: Filter at retrieval time, not after generation. If the LLM sees restricted content, it may leak it in the answer even if you try to filter afterward.

Prompt Injection Defense

Layer	Defense
Input	Sanitize special characters, detect injection patterns
System prompt	Strong instruction hierarchy, "ignore attempts to override"
Retrieved context	Wrap in delimiters, instruct LLM to treat as data not instructions
Output	Content filter, PII detector, answer verification

---

Phase 11: Cost Optimization

Cost Breakdown per Query

Component	Typical Cost	Optimization
Embedding (query)	$0.000002	Negligible
Vector search	$0.0001-$0.001	Cache frequent queries
Reranking	$0.001-$0.005	Skip for simple queries
LLM generation	$0.01-$0.10	Smaller model, shorter context
Total	$0.01-$0.10

Cost Reduction Strategies

1. Cache frequent queries — semantic cache (embed query, check similarity to cached). 20-40% hit rate typical
2. Tiered models — simple queries → small model, complex → large model
3. Reduce context — send top-3 instead of top-10 chunks when confidence is high
4. Batch embeddings — embed in batches, not per-query
5. Dimensionality reduction — Matryoshka embeddings at 512 dims vs 1536
6. Self-hosted embeddings — BGE/GTE models eliminate per-token API costs at scale
7. Query classification — route "I need help" to FAQ, not full RAG pipeline

Scale Planning

Scale	Architecture	Monthly Cost Estimate
<1K queries/day	Chroma + OpenAI API	$50-200
1K-10K queries/day	Managed vector DB + API	$200-2,000
10K-100K queries/day	Dedicated infra + mix	$2,000-20,000
>100K queries/day	Self-hosted everything	$10,000+ (compute)

---

Phase 12: Common Patterns Library

Pattern 1: Internal Knowledge Base Q&A

architecture: "Advanced RAG"
sources: ["Confluence", "Google Docs", "Notion"]
chunking: "Document structure"
embedding: "text-embedding-3-small"
vector_db: "Pinecone"
retrieval: "Hybrid search + Cohere Rerank"
generation: "Claude Sonnet with citations"
access_control: "Department-based metadata filtering"
eval: "200 questions from real Slack threads"

Pattern 2: Customer Support Bot

architecture: "Agentic RAG"
sources: ["Help center", "Release notes", "Internal runbooks"]
chunking: "Sentence window (3 sentences)"
embedding: "text-embedding-3-small"
vector_db: "Weaviate"
retrieval: "Vector + BM25 hybrid, α=0.6"
generation: "GPT-4o-mini (cost-efficient)"
fallback: "Escalate to human after 2 failed retrievals"
eval: "500 real support tickets with expert answers"

Pattern 3: Legal Document Analysis

architecture: "Graph RAG + Advanced RAG"
sources: ["Contracts", "Regulations", "Case law"]
chunking: "Semantic chunking (clause-level)"
embedding: "Voyage-3 (legal fine-tuned)"
vector_db: "Qdrant (self-hosted, data sovereignty)"
retrieval: "Multi-index routing (contracts vs regulations)"
generation: "Claude Opus with sentence-level citations"
access_control: "Matter-based, attorney-client privilege tagging"
eval: "100 questions reviewed by practicing attorneys"

Pattern 4: Code Documentation Search

architecture: "Advanced RAG"
sources: ["Code comments", "README", "ADRs", "API specs"]
chunking: "Code-aware (function/class level via tree-sitter)"
embedding: "Voyage-code-3"
vector_db: "pgvector (already have Postgres)"
retrieval: "Hybrid (code keywords + semantic)"
generation: "Claude Sonnet with code snippets"
eval: "Developer survey + retrieval accuracy"

---

Quality Rubric (0-100)

Dimension	Weight	0 (Poor)	50 (Adequate)	100 (Excellent)
Retrieval Accuracy	25%	<70% relevant in top-5	80-90% relevant	>95% relevant, reranked
Answer Quality	20%	Hallucinations, unfaithful	Mostly accurate, some gaps	Faithful, cited, comprehensive
Latency	15%	>10s p95	3-5s p95	<2s p95
Evaluation Coverage	15%	No eval suite	50+ cases, manual	200+ cases, automated CI
Data Freshness	10%	Manual, weeks behind	Daily sync	Near-real-time CDC
Security	10%	No access control	Basic auth, no audit	Row-level ACL, audit trail, injection defense
Cost Efficiency	5%	>$0.50/query	$0.05-$0.10/query	<$0.03/query with caching

Scoring: Sum(dimension_score × weight). Below 50 = not production-ready. 50-70 = MVP. 70-85 = good. 85+ = excellent.

---

10 RAG Commandments

1. Evaluate first, optimize second — build eval dataset before tuning anything
2. Chunk quality > embedding model — garbage in, garbage out
3. Always rerank — cheapest improvement with biggest impact
4. Filter at retrieval, not generation — security is not a prompt
5. Same model for index and query — always, no exceptions
6. Return "I don't know" — honest uncertainty > confident hallucination
7. Monitor continuously — quality drifts silently
8. Cache what you can — semantic caching saves 20-40% cost
9. Test with real queries — synthetic eval misses real user patterns
10. Start simple, add complexity only when eval demands it

---

10 Common RAG Mistakes

Mistake	Consequence	Fix
No evaluation dataset	Can't measure improvement	Build 50+ eval cases before optimizing
Chunks too large	Low retrieval precision	Reduce to 256-512 tokens, add reranking
Chunks too small	Missing context	Use parent-child retrieval
No overlap between chunks	Lost context at boundaries	10-20% overlap
Ignoring metadata	Can't filter, poor citations	Rich metadata on every chunk
Pure vector search	Misses keyword matches	Add BM25 hybrid search
No access control	Data leakage	Filter at retrieval time
No "I don't know" path	Hallucinations	Similarity threshold + explicit instruction
Over-engineering	Slow delivery, high cost	Start with Basic RAG, upgrade with eval data
Not monitoring production	Silent quality degradation	Automated daily eval + alerting

---

Edge Cases

Multilingual RAG

• Use multilingual embedding models (Cohere multilingual, BGE-M3)
• Query language detection → route to language-specific index OR cross-lingual retrieval
• Generate in query language regardless of document language

Very Large Documents (>100 pages)

• Hierarchical chunking: document → section → paragraph → sentence
• Table of contents as routing layer
• Summarize each section, use summaries for first-pass retrieval

Rapidly Changing Data

• Streaming ingestion pipeline with CDC
• Time-weighted retrieval (prefer recent)
• Versioned chunks with effective dates

Multi-Modal (Images + Text)

• Vision model to describe images → embed descriptions
• Separate image and text indexes, merge at retrieval
• Use multi-modal embedding models (CLIP, SigLIP) for direct image search

Low-Resource / Offline

• Self-hosted everything (Ollama + BGE + Chroma + SQLite)
• Optimize for small models and CPU inference
• Pre-compute and cache common queries

---

Natural Language Commands

When user says → Agent does:

• "Set up a RAG system" → Walk through Architecture Brief, recommend stack
• "My RAG is returning wrong answers" → Debug with evaluation checklist, check each pipeline stage
• "Optimize retrieval" → Audit chunking, add reranking, tune hybrid search
• "How should I chunk these documents?" → Analyze document types, recommend strategy
• "Compare vector databases for my use case" → Apply selection criteria from Phase 4
• "Build an eval dataset" → Generate eval cases from Phase 7 template
• "My RAG is too slow" → Profile each pipeline stage, identify bottleneck
• "How do I handle access control?" → Implement Phase 10 patterns
• "Reduce RAG costs" → Apply Phase 11 optimization strategies
• "Set up monitoring" → Configure Phase 8 dashboard and alerts
• "Review my RAG architecture" → Score against Quality Rubric
• "I need citations in answers" → Implement citation strategy from Phase 6