Back to skill

Security audit

Zouroboros Bench

Security checks for vulnerabilities and agentic risk

Overview

The skill appears to be a real benchmark tool, but it needs review because crafted inputs can reach shell commands and conversation data can be sent to external model services.

Install only after reviewing the command-execution and data-sharing behavior. Use trusted, public datasets or a restricted account, avoid exposing sensitive environment variables, prefer pinned package versions, and run local-only unless you explicitly intend to send benchmark conversation context to OpenAI. Treat Mimir writeback as persistent application state and avoid using it against a personal or production memory database without isolation and rollback.

Vulnerability Patterns
  • Agent Memory PoisoningWrites attacker-controlled rules into memory that affect later sessions
  • Insecure Skill Coding PracticesFinds exploitable flaws such as hardcoded secrets or command injection
  • Skill Instruction HijackingAlters the agent's session goals or safety constraints when the skill loads
  • Remote Payload Retrieval and ExecutionFetches external code whose behavior can change after review
  • Embedded Malicious CodeShips malicious scripts inside the skill and executes them locally
Findings (4)

T09 · Insecure Skill Coding Practices

Error
Location
src/adapters/convomem-adapter.ts:134
Finding

Shell Command Injection Through Dataset-Controlled Search Queries

Content
View full analysis
Remediation
View remediation

T09 · Insecure Skill Coding Practices

Error
Location
src/scripts/run-all.ts:101
Finding

Shell Command Injection Through the Run-All Limit Argument

Content
View full analysis
s.trim()); const limit = values.limit!; const judge = values.judge!; // ... for (const name of runnable) { const bench = BENCHMARKS[name]; const cmd = `node ${join(ADAPTERS_DIR, bench.adapter)} ${bench.datasetFlag} --output ${RUNS_DIR} --limit ${limit}${judge ? " --judge" : ""}`; console.log(`\n${"━".repeat(60)}`); console.log(`Running ${name.toUpperCase()}`); console.log(` ${cmd}`); console.log(`${"━".repeat(60)}`); const t0 = Date.now(); try { execSync(cmd, { stdio: "inherit", timeout: 1_800_000, cwd: PROJECT_ROOT, }); ``` ### Technical Analysis The `--limit` option is parsed as an unrestricted string. It is then concatenated directly into a command string passed to `execSync`. Because no integer validation or shell-safe argument separation is applied, shell metacharacters in the argument are interpreted by the command shell. The selected benchmark name is restricted through the `BENCHMARKS` mapping, but that restriction does not protect the unvalidated `limit` value. ### Attack Path 1. An attacker, wrapper script, CI parameter, or automation system controls the `--limit` argument. 2. The attacker supplies a value containing shell operators or command substitution. 3. `run-all.ts` inserts the value directly after `--limit`. 4. `execSync` executes the resulting command through the platform shell. 5. The injected comman ...[truncated 508 chars]
Remediation
View remediation
1_000_000) { throw new Error("--limit must be an integer in the permitted range"); } ``` 2. Invoke Node without a shell: ```ts import { execFileSync } from "child_process"; const argv = [ join(ADAPTERS_DIR, bench.adapter), "--dataset", bench.datasetCheck, "--output", RUNS_DIR, "--limit", String(limit), ]; if (judge) argv.push("--judge"); execFileSync(process.execPath, argv, { shell: false, stdio: "inherit", timeout: 1_800_000, cwd: PROJECT_ROOT, }); ``` 3. Represent dataset flags as argument arrays rather than preformatted strings. 4. Use the same argument-separated approach for report generation. 5. Add tests proving that non-numeric and shell-metacharacter-containing limit values are rejected. ]]>

T02 · Agent Memory Poisoning

Error
Location
src/adapters/mimir-judge.ts:112
Finding

Mimir Memory Persistence Permits Command Injection and Persistent Memory Poisoning

Content
View full analysis
; const seen = new Set(); const patterns: DeprecatedPattern[] = []; for (const row of rows) { const parts = row.entity.split("."); if (parts.length < 2) continue; const term = parts[parts.length - 1]; if (term.length < 4) continue; if (seen.has(term.toLowerCase())) continue; seen.add(term.toLowerCase()); const valueLower = row.value.toLowerCase(); let matchedSignal = "deprecated"; for (const sig of STATUS_SIGNALS) { if (valueLower.includes(sig)) { matchedSignal = sig; break; } } patterns.push({ pattern: term.toLowerCase(), signal: matchedSignal, entity: row.entity }); } ``` These values are subsequently interpolated into a shell command and stored with stable decay: ```ts async function storeContradiction( entity: string, pattern: string, signal: string, question: string, hypothesis: string, ): Promise { try { const key = `mimir-correction-${pattern}`; const db = new Database(MEMORY_DB_PATH, { readonly: true }); const existing = db .prepare("SELECT id FROM facts WHERE entity = ? AND key = ? LIMIT 1") .get(entity, key) as { id: string } | null; db.close(); if (existing) return; const value = `[[${entity}]] flagged as ${signal} by Mimir judge. Hypothesis "${hypothesis.slice(0, 120)}" incorrectly referenced "${patt ...[truncated 3263 chars]
Remediation
View remediation

other

Warning
Location
src/adapters/convomem-adapter.ts:199
Finding

Conversation Data Is Sent to OpenAI by Default When an API Key Is Present

Content
View full analysis
{ if (ANSWER_MODEL.startsWith("gpt-") && OPENAI_API_KEY) { return openaiGenerate(prompt, ANSWER_MODEL); } return ollamaGenerate(prompt, ANSWER_MODEL); } async function openaiGenerate(prompt: string, model = "gpt-4o-mini"): Promise { const resp = await fetch("https://api.openai.com/v1/chat/completions", { method: "POST", headers: { "Content-Type": "application/json", Authorization: `Bearer ${OPENAI_API_KEY}`, }, body: JSON.stringify({ model, messages: [{ role: "user", content: prompt }], temperature: 0.1, max_tokens: 200, }), }); if (!resp.ok) { throw new Error(`OpenAI error: ${resp.status} ${await resp.text()}`); } const data = (await resp.json()) as any; return (data.choices?.[0]?.message?.content ?? "").trim(); } ``` The prompt includes retrieved conversation memories and the benchmark question: ```ts const contextStr = context.join("\n---\n").slice(0, CONTEXT_CHAR_LIMIT); hypothesis = await generateAnswer( `Given these conversation memories, answer the question with ONLY the specific fact requested. Give a short, direct answer — just ...[truncated 3074 chars]
Remediation
View remediation
Vulnerability Patterns
  • Data ExfiltrationExternal Transmission, Env Variable Harvesting, File System Enumeration
  • Supply ChainUnpinned Dependencies, External Script Fetching, Obfuscated Code
  • Excessive AgencyUnrestricted Tool Access, Autonomous Decision Making, Scope Creep
  • MCP Least PrivilegeUnderdeclared Capability, Wildcard Permission, Missing Permission Declaration
  • MCP Tool PoisoningHidden Instructions, Unicode Deception, Parameter Description Injection
Findings (87)

Tp4

High
Category
MCP Tool Poisoning
Confidence
95% confidence
Finding

Advertising LongMemEval, LoCoMo, ConvoMem, and Mimir support while only implementing a subset and using a different judge creates a misleading security and functionality profile. Because this is a benchmark harness, such mismatches can lead to invalid evaluation decisions, accidental external data transmission, and misplaced trust in results.

Content

No source excerpt is available for this finding.

Tp4

High
Category
MCP Tool Poisoning
Confidence
97% confidence
Finding

Advertising LongMemEval, LoCoMo, ConvoMem, and Mimir support while only implementing a subset and using a different judge creates a misleading security and functionality profile. Because this is a benchmark harness, such mismatches can lead to invalid evaluation decisions, accidental external data transmission, and misplaced trust in results.

Content

No source excerpt is available for this finding.

Tp4

High
Category
MCP Tool Poisoning
Confidence
94% confidence
Finding

Advertising LongMemEval, LoCoMo, ConvoMem, and Mimir support while only implementing a subset and using a different judge creates a misleading security and functionality profile. Because this is a benchmark harness, such mismatches can lead to invalid evaluation decisions, accidental external data transmission, and misplaced trust in results.

Content

No source excerpt is available for this finding.

Tp4

High
Category
MCP Tool Poisoning
Confidence
92% confidence
Finding

Advertising LongMemEval, LoCoMo, ConvoMem, and Mimir support while only implementing a subset and using a different judge creates a misleading security and functionality profile. Because this is a benchmark harness, such mismatches can lead to invalid evaluation decisions, accidental external data transmission, and misplaced trust in results.

Content

No source excerpt is available for this finding.

Intent-Code Divergence

High
Category
Not specified by scanner
Confidence
97% confidence
Finding

The comments state the adapter uses only the memory CLI and no direct external mechanisms, but the implementation also performs multiple HTTP calls to OpenAI and optionally Ollama. This mismatch is dangerous because operators may trust the documentation and run the tool in environments where conversation data must not leave the system.

Content

No source excerpt is available for this finding.

Rp1

Medium
Category
MCP Rug Pull
Confidence
84% confidence
Finding

The README instructs users to execute an unpinned package via npx, which can fetch and run the latest published version at execution time. If the package or one of its publish paths is compromised, users could run attacker-controlled code directly on their machine.

Content

No source excerpt is available for this finding.

Rp1

Medium
Category
MCP Rug Pull
Confidence
84% confidence
Finding

This command uses npx without an explicit version, so it may download and execute whatever package version is current when the user runs it. That creates supply-chain exposure because a malicious or hijacked release could be executed immediately from the documentation example.

Content

No source excerpt is available for this finding.

Rp1

Medium
Category
MCP Rug Pull
Confidence
82% confidence
Finding

The report-generation example invokes an unpinned package through npx, which can result in execution of an unexpected future release. In a benchmark tool context this is still dangerous because users are being encouraged to run code locally, and documentation-driven one-liners are commonly copied verbatim.

Content

No source excerpt is available for this finding.

Undeclared Tool Scope

Medium
Category
MCP Least Privilege
Confidence
94% confidence
Finding

The skill declares environment and network-related behavior but does not define an explicit tool scope such as permissions or allowed-tools. In a benchmark skill that can contact external LLM services and consume API keys, this creates an under-specified trust boundary and can lead users or hosting platforms to authorize broader access than intended.

Content

No source excerpt is available for this finding.

Rp1

Medium
Category
MCP Rug Pull
Confidence
88% confidence
Finding

Using npx zouroboros-bench without pinning a version makes execution depend on whatever package version is current at runtime. This introduces a supply-chain risk where a compromised, malicious, or incompatible future release could be fetched and executed automatically.

Content

No source excerpt is available for this finding.

Rp1

Medium
Category
MCP Rug Pull
Confidence
88% confidence
Finding

This unpinned npx zouroboros-bench invocation has the same supply-chain exposure as the other example: the executed code may change over time without review. For a tool with network and API-key usage, silently pulling a new release materially increases risk.

Content

No source excerpt is available for this finding.

Rp1

Medium
Category
MCP Rug Pull
Confidence
87% confidence
Finding

Running npx zouroboros-bench-report without a fixed version allows arbitrary future package contents to be executed. Even a reporting tool can access local benchmark artifacts and potentially exfiltrate data or alter outputs if the package is compromised.

Content

No source excerpt is available for this finding.

Context-Inappropriate Capability

Medium
Category
Not specified by scanner
Confidence
97% confidence
Finding

The adapter transmits benchmark questions and retrieved conversation context to external LLM services for query expansion, answer generation, and judging. Because the retrieved context is built from conversation memories and may contain sensitive user data, this creates a real data-exfiltration/privacy risk, especially since transmission happens automatically when API keys are present.

Content

No source excerpt is available for this finding.

External Transmission

Medium
Category
Data Exfiltration
Confidence
90% confidence
Finding

The hardcoded OpenAI endpoint confirms that data is sent to an external service rather than remaining local. In this adapter, the surrounding code shows that transmitted payloads include conversation-derived context, making the endpoint usage materially relevant to privacy and data-governance risk.

Content

Scanner excerpt · dist/adapters/convomem-adapter.js (reported line 42)May include surrounding context.

js
Do NOT repeat the question or rephrase it. Focus on finding orthogonal constraints.
Output only 3 queries, one per line, no numbering, no explanation.`;
  try {
    const resp = await fetch("https://api.openai.com/v1/chat/completions", {
      method: "POST",
      headers: { "Content-Type": "application/json", Authorization: `Bearer ${OPENAI_API_KEY}` },
      body: JSON.stringify({

External Transmission

Medium
Category
Data Exfiltration
Confidence
90% confidence
Finding

The hardcoded OpenAI endpoint confirms that data is sent to an external service rather than remaining local. In this adapter, the surrounding code shows that transmitted payloads include conversation-derived context, making the endpoint usage materially relevant to privacy and data-governance risk.

Content

Scanner excerpt · dist/adapters/convomem-adapter.js (reported line 42)May include surrounding context.

js
Do NOT repeat the question or rephrase it. Focus on finding orthogonal constraints.
Output only 3 queries, one per line, no numbering, no explanation.`;
  try {
    const resp = await fetch("https://api.openai.com/v1/chat/completions", {
      method: "POST",
      headers: { "Content-Type": "application/json", Authorization: `Bearer ${OPENAI_API_KEY}` },
      body: JSON.stringify({

Missing User Warnings

Medium
Category
Not specified by scanner
Confidence
95% confidence
Finding

Conversation content is embedded into prompts and sent to OpenAI without any user-facing warning, consent gate, or disclosure. In a memory-benchmarking skill, the context is especially likely to contain personal facts and conversational history, so silent transmission materially increases privacy and compliance risk.

Content

No source excerpt is available for this finding.

Missing User Warnings

Medium
Category
Not specified by scanner
Confidence
93% confidence
Finding

The adapter constructs a shell command string and executes it with execSync using a configurable executable path and argument list. If any component such as MEMORY_CLI or future argument sources is attacker-controlled, this can become command injection or unintended command execution with the privileges of the process.

Content

No source excerpt is available for this finding.

External Transmission

Medium
Category
Data Exfiltration
Confidence
90% confidence
Finding

The OpenAI endpoint at this location is used for answer generation, which means conversation-backed prompts leave the local environment. In a tool designed to evaluate memory systems, that context can be rich in sensitive personal details, so the transmission is security-relevant rather than merely functional.

Content

Scanner excerpt · dist/adapters/convomem-adapter.js (reported line 141)May include surrounding context.

js
return ollamaGenerate(prompt, ANSWER_MODEL);
}
async function openaiGenerate(prompt, model = "gpt-4o-mini") {
  const resp = await fetch("https://api.openai.com/v1/chat/completions", {
    method: "POST",
    headers: { "Content-Type": "application/json", Authorization: `Bearer ${OPENAI_API_KEY}` },
    body: JSON.stringify({

External Transmission

Medium
Category
Data Exfiltration
Confidence
90% confidence
Finding

The OpenAI endpoint at this location is used for answer generation, which means conversation-backed prompts leave the local environment. In a tool designed to evaluate memory systems, that context can be rich in sensitive personal details, so the transmission is security-relevant rather than merely functional.

Content

Scanner excerpt · dist/adapters/convomem-adapter.js (reported line 141)May include surrounding context.

js
return ollamaGenerate(prompt, ANSWER_MODEL);
}
async function openaiGenerate(prompt, model = "gpt-4o-mini") {
  const resp = await fetch("https://api.openai.com/v1/chat/completions", {
    method: "POST",
    headers: { "Content-Type": "application/json", Authorization: `Bearer ${OPENAI_API_KEY}` },
    body: JSON.stringify({

External Transmission

Medium
Category
Data Exfiltration
Confidence
89% confidence
Finding

The OpenAI endpoint here is used for judge decisions, exporting evaluation content to a third party. While less dangerous than arbitrary code execution, it is still a true confidentiality issue because benchmark questions, answers, and potentially sensitive hypotheses are disclosed externally.

Content

Scanner excerpt · dist/adapters/convomem-adapter.js (reported line 190)May include surrounding context.

js
${criteria}

Respond with exactly one word: CORRECT or INCORRECT`;
  const resp = await fetch("https://api.openai.com/v1/chat/completions", {
    method: "POST",
    headers: { "Content-Type": "application/json", Authorization: `Bearer ${apiKey}` },
    body: JSON.stringify({

External Transmission

Medium
Category
Data Exfiltration
Confidence
89% confidence
Finding

The OpenAI endpoint here is used for judge decisions, exporting evaluation content to a third party. While less dangerous than arbitrary code execution, it is still a true confidentiality issue because benchmark questions, answers, and potentially sensitive hypotheses are disclosed externally.

Content

Scanner excerpt · dist/adapters/convomem-adapter.js (reported line 190)May include surrounding context.

js
${criteria}

Respond with exactly one word: CORRECT or INCORRECT`;
  const resp = await fetch("https://api.openai.com/v1/chat/completions", {
    method: "POST",
    headers: { "Content-Type": "application/json", Authorization: `Bearer ${apiKey}` },
    body: JSON.stringify({

Intent-Code Divergence

Medium
Category
Not specified by scanner
Confidence
91% confidence
Finding

The code advertises 'PRODUCTION CLI ONLY' while also making direct network calls to OpenAI and Ollama elsewhere in the file. This mismatch can mislead operators into assuming the adapter is confined to local CLI behavior, reducing scrutiny over outbound data flows and increasing the chance of unintended disclosure.

Content

No source excerpt is available for this finding.

External Transmission

Medium
Category
Data Exfiltration
Confidence
90% confidence
Finding

The hardcoded OpenAI API endpoint confirms that answer generation can leave the local environment and reach a third-party service. In this skill's context, that matters because the transmitted prompt is derived from stored conversation memory, increasing privacy exposure beyond ordinary telemetry.

Content

Scanner excerpt · dist/adapters/locomo-adapter.js (reported line 82)May include surrounding context.

js
return ollamaGenerate(prompt, ANSWER_MODEL);
}
async function openaiGenerate(prompt, model = "gpt-4o-mini") {
  const resp = await fetch("https://api.openai.com/v1/chat/completions", {
    method: "POST",
    headers: { "Content-Type": "application/json", Authorization: `Bearer ${OPENAI_API_KEY}` },
    body: JSON.stringify({

External Transmission

Medium
Category
Data Exfiltration
Confidence
90% confidence
Finding

The hardcoded OpenAI API endpoint confirms that answer generation can leave the local environment and reach a third-party service. In this skill's context, that matters because the transmitted prompt is derived from stored conversation memory, increasing privacy exposure beyond ordinary telemetry.

Content

Scanner excerpt · dist/adapters/locomo-adapter.js (reported line 82)May include surrounding context.

js
return ollamaGenerate(prompt, ANSWER_MODEL);
}
async function openaiGenerate(prompt, model = "gpt-4o-mini") {
  const resp = await fetch("https://api.openai.com/v1/chat/completions", {
    method: "POST",
    headers: { "Content-Type": "application/json", Authorization: `Bearer ${OPENAI_API_KEY}` },
    body: JSON.stringify({

Missing User Warnings

Medium
Category
Not specified by scanner
Confidence
94% confidence
Finding

The adapter sends retrieved conversation context and user questions to external model APIs when generating answers, but there is no explicit disclosure, consent check, or data-minimization control at the call site. In a memory benchmarking tool, the dataset may contain personal or sensitive conversation content, so transmitting it to third-party services can create an unannounced privacy and compliance risk.

Content

No source excerpt is available for this finding.