Zouroboros Bench

Benchmark harness for AI memory systems. Evaluates LongMemEval, LoCoMo, and ConvoMem datasets against any memory backend via the zouroboros-memory CLI. Inclu...

Security Scan

The package mostly matches its stated benchmarking purpose, but the runtime code will read and sometimes write a local 'memory' database (defaulting to ~/.zouroboros/memory.db) and will send benchmark/memory content to external services (OpenAI and a configurable 'Mimir' gate), behavior that is not fully declared in the SKILL metadata or README.

Purpose & Capability

The skill is a Node-based benchmark harness that calls an external memory CLI and cloud/local LLM endpoints — this is consistent with a memory benchmarking tool. However, the code defaults to reading/writing a persistent local memory DB at ~/.zouroboros/memory.db (via better-sqlite3 or the memory CLI) and will attempt to store 'mimir-correction' facts back into the memory system. The SKILL metadata did not declare any required config paths or warn that it may modify a user's persistent memory DB, which is surprising for some users.

Instruction Scope

SKILL.md instructs installation and running npx commands and documents env vars like ZOUROBOROS_MEMORY_CLI, ZOUROBOROS_MEMORY_DB, OPENAI_API_KEY, and OLLAMA_URL. The runtime code, however, will: (1) read local dataset files and the user's home memory DB by default, (2) invoke external binaries (zouroboros-memory) via execSync, (3) send data to api.openai.com and to a GATE_URL endpoint (MIMIR_GATE_URL) and to an Ollama URL — potentially transmitting benchmark contents and memory content to remote services. The README does not explicitly warn that dataset or memory contents may be transmitted to third-party services or that the tool will write corrections back into a persistent memory DB.

Install Mechanism

Install is a normal npm package (zouroboros-bench). No downloadable archives or obscure URLs were used. Code is included in the package (dist/). This is an expected install mechanism for a Node.js tool.

Credentials

The registry metadata declares only OPENAI_API_KEY as required (which the code uses). The code also respects other env vars (ZOUROBOROS_MEMORY_CLI, ZOUROBOROS_MEMORY_DB, OLLAMA_URL, MIMIR_GATE_URL, ZO_ANSWER_MODEL). The main concern: the tool accesses a default memory DB path in the user's home directory (ZO_MEMORY_DB default -> ~/.zouroboros/memory.db) and may write entries into that DB via the memory CLI. That means the skill can access and mutate persistent, possibly sensitive local data without the SKILL manifest declaring a required config path or explicitly warning the user.

Persistence & Privilege

The skill does not request 'always: true' and doesn't modify other skills, but it does persist corrections into a memory backend (via execSync of the memory CLI) and reads a local SQLite DB by default. That gives it persistent side effects on the user's memory store, which the SKILL.md/metadata do not prominently warn about. Users should treat this as a write-capable tool for local persistent state.

What to consider before installing

What to consider before installing/running: - The tool is intended for benchmarking memory backends and mostly does what it claims, but the runtime will by default open and may write to a persistent memory DB at ~/.zouroboros/memory.db (or whatever ZOUROBOROS_MEMORY_DB points to). If you have an existing Zouroboros memory DB you care about, the benchmark may modify it (it can store 'mimir-correction-*' facts). - The code calls external services: OpenAI (api.openai.com) if you set OPENAI_API_KEY, a configurable 'Mimir' gate endpoint (MIMIR_GATE_URL, default http://localhost:7820), and an Ollama local model URL. Running with OPENAI_API_KEY will send prompts and model responses (potentially dataset or memory contents) to OpenAI's API. Ensure you are comfortable with that data leaving your machine. - Mitigations: run the benchmark in an isolated environment (container, VM, or throwaway account), set ZOUROBOROS_MEMORY_DB to a temporary path so the tool does not touch your real home DB, and avoid providing OPENAI_API_KEY if you do not want cloud uploads — use an offline Ollama instance instead. Inspect or run the package source locally before npm installing globally. Review and, if necessary, modify dist/adapters/mimir-judge.js and other adapters if you want to disable the code path that writes to the memory system. - If you need a simple read-only benchmark run: ensure ZOUROBOROS_MEMORY_CLI points to a no-op or stub binary or use ZOUROBOROS_MEMORY_DB pointing to an isolated SQLite file; do not provide OPENAI_API_KEY and set MIMIR_GATE_URL to localhost or an internal endpoint you control. - If you are not comfortable with these behaviors or cannot run in an isolated environment, treat this skill as potentially risky and avoid giving it access to your real memory DB or OpenAI credentials.

Usage

Install: npm install zouroboros-bench zouroboros-memory

Run all benchmarks

npx zouroboros-bench --limit 50

Run specific benchmark

npx zouroboros-bench --benchmarks longmemeval --limit 100 --judge

Generate report

npx zouroboros-bench-report --runs ./data/runs/

Environment Variables

• ZOUROBOROS_MEMORY_CLI — Path to memory CLI binary (default: zouroboros-memory)
• ZOUROBOROS_MEMORY_DB — SQLite DB path for benchmarks
• OPENAI_API_KEY — Required for GPT-4o judge
• OLLAMA_URL — Ollama URL for local LLM (default: http://localhost:11434)