Eidolon Search

AI Agent memory search using SQLite FTS5. 90%+ token reduction (10x+) compared to reading full files. Use when the agent needs to search through markdown memory files, daily notes, or any text corpus efficiently. Triggers on memory search, file search, knowledge retrieval, or when context window is limited and full-file reading is too expensive.

FTS5-based memory search for AI Agents. Index markdown files once, search with 90%+ token savings.

Quick Start

1. Index memory files (once)

python3 scripts/build-index.py <memory_dir> <db_path>

Example:

python3 scripts/build-index.py ./memory ./memory.db

This creates a SQLite database with FTS5 full-text index of all .md files in the directory (recursive).

2. Search

python3 scripts/search.py <query> [limit] [db_path]

Example:

python3 scripts/search.py "Physical AI roadmap" 5
python3 scripts/search.py "project timeline" 10 ./memory.db

Default limit: 10. Default db_path: ./memory.db

Output: matching snippets with file paths and relevance scores.

3. Re-index when files change

Run build-index.py again. It rebuilds the index from scratch (fast, <1 second for typical workspaces).

When to Use

• Memory search: Find specific information across many daily notes or memory files
• Token-limited contexts: When reading all files would exceed context limits
• Repeated searches: Index once, search many times
• Large workspaces: 10+ markdown files with cumulative size >50KB

When NOT to Use

• Single small file (<5KB): just read it directly
• Need semantic/meaning-based search: FTS5 is keyword-based only
• See references/QUALITY.md for known limitations

Search Tips for Agents

FTS5 is keyword-based. Improve results by:

• Use specific terms: "Jetson Orin" not "hardware plans"
• Use OR for synonyms: "car OR vehicle OR automobile"
• Use quotes for phrases: "Physical AI"
• Try multiple queries if first attempt returns nothing
• Check file paths in results to read full context when needed

Benchmarks

• Token savings: 90%+ (measured 93-98.9%)
• Speed: 15x faster (measured 10-20x)
• Details: references/PERFORMANCE.md

Run benchmarks yourself:

python3 scripts/benchmark-recall.py    # Recall@5, Recall@10
python3 scripts/benchmark-cache.py     # Warm vs cold cache

DB Schema

CREATE VIRTUAL TABLE memory_fts USING fts5(path, content);

Direct SQL access:

sqlite3 memory.db "SELECT path, snippet(memory_fts, 1, '<b>', '</b>', '...', 32) FROM memory_fts WHERE memory_fts MATCH 'query' ORDER BY rank LIMIT 5;"