Security audit

Zouroboros Bench

Security checks across malware telemetry and agentic risk

Overview

This is a real benchmark tool, but it can send conversation-memory benchmark data to model APIs and includes memory writeback behavior that is not clearly scoped for users.

Review this before installing if your benchmark datasets or memory store contain private conversations, customer data, proprietary evaluations, or institutional knowledge. Use sanitized data, avoid pointing Mimir at a real persistent memory database unless writeback is intended, and treat datasets, CLI paths, and endpoint environment variables as trusted inputs.

SkillSpector

By NVIDIA

Vulnerability Patterns

Data ExfiltrationExternal Transmission, Env Variable Harvesting, File System Enumeration
Excessive AgencyUnrestricted Tool Access, Autonomous Decision Making, Scope Creep
Behavioral ASTexec() Call, eval() Call, Dynamic Import
MCP Least PrivilegeUnderdeclared Capability, Wildcard Permission, Missing Permission Declaration
MCP Tool PoisoningHidden Instructions, Unicode Deception, Parameter Description Injection

Findings (44)

Lp3

Medium

Category: MCP Least Privilege
Confidence: 90% confidence
Finding: The skill declares no permissions while clearly requiring environment-variable access and outbound network access. This undermines informed consent and sandboxing expectations, making it easier for the skill to access sensitive secrets like OPENAI_API_KEY and communicate externally without an explicit permission model.

Tp4

High

Category: MCP Tool Poisoning
Confidence: 95% confidence
Finding: The documented purpose understates the actual behavior: beyond benchmarking, the skill reportedly accesses the SQLite memory database directly, writes contradiction facts back into the memory system, and sends data to external services. In a memory-system benchmark context, this is dangerous because evaluation tooling should not silently mutate the tested system or exfiltrate benchmark or memory contents to third parties without clear disclosure and controls.

Description-Behavior Mismatch

Medium

Confidence: 96% confidence
Finding: This code sends benchmark question text and retrieved conversation snippets to OpenAI to generate implicit follow-up queries. That is a real data-exfiltration risk because dataset contents and memory context leave the local benchmark environment and are disclosed to a third party, which is more dangerous in a memory-benchmark adapter because the inputs can contain realistic conversation data and latent sensitive facts.

Context-Inappropriate Capability

Medium

Confidence: 95% confidence
Finding: The adapter builds a shell command string with a CLI path taken from an environment variable and executes it with execSync. Because the command is shell-interpreted, a malicious or compromised environment value for ZOUROBOROS_MEMORY_CLI can inject arbitrary shell metacharacters and achieve command execution.

Description-Behavior Mismatch

Medium

Confidence: 92% confidence
Finding: The adapter performs answer generation by sending prompts and retrieved context to either OpenAI or a configurable local Ollama endpoint. This extends the benchmark beyond local retrieval and creates a confidentiality boundary crossing, especially if the Ollama URL is remote or the OpenAI path is enabled with real conversation data.

Description-Behavior Mismatch

Medium

Confidence: 97% confidence
Finding: The judge function uploads the benchmark question, ground truth answer, category, and model output to OpenAI for scoring. This is a true external disclosure issue because evaluation artifacts may contain sensitive dataset content, and the benchmark context increases exposure by sending both source truth and generated outputs together.

Context-Inappropriate Capability

Medium

Confidence: 95% confidence
Finding: The adapter sends benchmark conversation context, questions, ground truth answers, and model hypotheses to external model services for answer generation and optional judging. Even if this is expected benchmark behavior, it creates a real data exposure path because dataset contents are transmitted off-process/off-host without enforcement of redaction, consent, or a strict local-only mode.

Description-Behavior Mismatch

Medium

Confidence: 89% confidence
Finding: The judge component is not purely evaluating outputs: when it detects a contradiction, it writes a new fact into the memory backend via storeContradiction. In a benchmark harness, this creates hidden state mutation during evaluation, which can contaminate future runs, alter benchmark behavior, and create an integrity boundary violation between evaluator and system-under-test.

Context-Inappropriate Capability

Medium

Confidence: 91% confidence
Finding: The code spawns an external CLI subprocess to store contradiction records in the memory system, giving the judge an unnecessary state-modification capability. This expands the attack surface and trust boundary: evaluation input can indirectly trigger local command execution and persistent data changes outside the normal benchmark role.

Context-Inappropriate Capability

Medium

Confidence: 93% confidence
Finding: The judge sends Mimir memory context, benchmark questions, ground truth, and hypothesis content to OpenAI for adjudication. That is a real third-party data disclosure path beyond local evaluation, and could expose proprietary, sensitive, or customer-derived memory content depending on what the backend stores.

Description-Behavior Mismatch

Medium

Confidence: 91% confidence
Finding: The adapter sends benchmark questions and retrieved conversation context to OpenAI for implicit query expansion, which is an external data transfer outside a strictly local benchmark harness. In a memory-evaluation setting, those conversations can contain sensitive benchmark or user-like content, so exporting them to a third-party API creates confidentiality and scope-creep risk.

Intent-Code Divergence

High

Confidence: 95% confidence
Finding: The file header states the adapter uses 'ONLY the memory CLI', but the implementation also calls OpenAI and Ollama APIs directly. This mismatch can mislead operators into running the tool under assumptions of local-only processing, increasing the chance of unintended data egress.

Context-Inappropriate Capability

Medium

Confidence: 92% confidence
Finding: The adapter reads OPENAI_API_KEY and later uses it to transmit benchmark prompts, questions, ground truth answers, and retrieved context to external providers. In this benchmark context, that can leak sensitive conversational data and evaluation content to third parties without strong in-file consent or minimization controls.

Description-Behavior Mismatch

Medium

Confidence: 91% confidence
Finding: The judge is not purely evaluative: it writes new contradiction records back into the shared memory system, creating side effects during benchmark execution. In a benchmarking context this is risky because evaluation can poison or drift the corpus, contaminate future runs, and persist snippets of benchmark inputs into long-lived storage.

Context-Inappropriate Capability

Medium

Confidence: 95% confidence
Finding: This code builds a shell command with interpolated entity, key, and value and executes it via execSync. Although encodeURIComponent is applied to the value, entity/key are still shell-interpreted, and any attacker-controlled content reaching them could trigger command injection or unintended CLI behavior; additionally it gives judge logic the ability to mutate external state via subprocess.

Context-Inappropriate Capability

Medium

Confidence: 93% confidence
Finding: The judge sends benchmark question, ground truth, hypothesis, and retrieved institutional context to OpenAI. That is an external data egress path which may expose proprietary benchmark data, architectural context, or user-derived content outside the local environment.

Intent-Code Divergence

Medium

Confidence: 94% confidence
Finding: The script hard-codes methodology claims such as GPT-4o judging being used conditionally and retrieval having no cloud dependencies, without deriving those statements from the actual benchmark run metadata. This can produce materially misleading reports that misrepresent evaluation conditions, which undermines integrity of benchmark results and can be abused to make unsupported security/privacy or performance claims.

Missing User Warnings

Medium

Confidence: 90% confidence
Finding: Conversation-derived context is transmitted to OpenAI without any runtime disclosure, consent checkpoint, or clear indication in the adapter flow. Even if intended for benchmarking, silent third-party transmission is dangerous because users may reasonably expect a local harness and may unknowingly expose sensitive conversational material.

Missing User Warnings

Medium

Confidence: 89% confidence
Finding: Conversation-derived context is included in prompts sent to OpenAI or to an Ollama server without any user-facing disclosure in this code path. In a memory benchmark harness, the data may still contain sensitive conversational content, so silent transmission increases privacy and compliance risk.

Missing User Warnings

Medium

Confidence: 92% confidence
Finding: The GPT-4 judge prompt sends the benchmark question, ground truth, and system answer to OpenAI whenever --judge is enabled and an API key is present. That is a real external disclosure channel, and the absence of an explicit warning or consent gate makes accidental data sharing more likely.

Missing User Warnings

Medium

Confidence: 93% confidence
Finding: The adapter sends full prompts containing retrieved conversation context to an external model endpoint via Ollama, and may route to OpenAI when the selected model starts with 'gpt-' and an API key is present. Even in a benchmark harness, this can expose sensitive conversation data to services outside the local trust boundary without explicit consent, minimization, or disclosure.

Missing User Warnings

Medium

Confidence: 97% confidence
Finding: This code transmits prompt data to the OpenAI Chat Completions API, and the prompt can contain conversation history assembled in cotAnswer. That creates a real data-exposure risk if datasets contain private or regulated content, especially since there is no visible user warning, consent gate, or sanitization in this file.

Missing User Warnings

Low

Confidence: 99% confidence
Finding: The command string is built by concatenating the executable, subcommand, and joined arguments, then passed to execSync, which invokes a shell. Because MEMORY_CLI comes from an environment variable and args may incorporate externally influenced strings, this creates a command-injection surface that can lead to arbitrary command execution on the host.

Missing User Warnings

Medium

Confidence: 84% confidence
Finding: The judge transmits benchmark-derived content to an HTTP gate service without any visible disclosure or consent mechanism in this file. Even if the default is localhost, the URL is environment-configurable, so questions and related content may be sent to a remote service unexpectedly.

Missing User Warnings

Medium

Confidence: 88% confidence
Finding: The subprocess invocation persists contradiction records to the memory backend without any user-facing notice, making benchmark execution silently mutate state. Hidden writes are dangerous in evaluation tooling because they undermine reproducibility and may pollute the memory system with judge-generated content.

VirusTotal

66/66 vendors flagged this skill as clean.

View on VirusTotal

Static analysis

Detected: suspicious.dangerous_exec, suspicious.env_credential_access

Shell command execution detected (child_process).

Critical

Code: suspicious.dangerous_exec
Location: dist/adapters/convomem-adapter.js:82

Shell command execution detected (child_process).

Critical

Code: suspicious.dangerous_exec
Location: dist/adapters/locomo-adapter.js:26

Shell command execution detected (child_process).

Critical

Code: suspicious.dangerous_exec
Location: dist/adapters/longmemeval-adapter.js:20

Shell command execution detected (child_process).

Critical

Code: suspicious.dangerous_exec
Location: dist/adapters/mimir-judge.js:62

Shell command execution detected (child_process).

Critical

Code: suspicious.dangerous_exec
Location: dist/scripts/run-all.js:90

Shell command execution detected (child_process).

Critical

Code: suspicious.dangerous_exec
Location: src/adapters/convomem-adapter.ts:139

Shell command execution detected (child_process).

Critical

Code: suspicious.dangerous_exec
Location: src/adapters/locomo-adapter.ts:90

Shell command execution detected (child_process).

Critical

Code: suspicious.dangerous_exec
Location: src/adapters/longmemeval-adapter.ts:107

Shell command execution detected (child_process).

Critical

Code: suspicious.dangerous_exec
Location: src/adapters/mimir-judge.ts:130

Shell command execution detected (child_process).

Critical

Code: suspicious.dangerous_exec
Location: src/scripts/run-all.ts:109

Environment variable access combined with network send.

Critical

Code: suspicious.env_credential_access
Location: dist/adapters/convomem-adapter.js:9

Environment variable access combined with network send.

Critical

Code: suspicious.env_credential_access
Location: dist/adapters/locomo-adapter.js:9

Environment variable access combined with network send.

Critical

Code: suspicious.env_credential_access
Location: dist/adapters/longmemeval-adapter.js:9

Environment variable access combined with network send.

Critical

Code: suspicious.env_credential_access
Location: dist/adapters/mimir-judge.js:6

Environment variable access combined with network send.

Critical

Code: suspicious.env_credential_access
Location: src/adapters/convomem-adapter.ts:27

Environment variable access combined with network send.

Critical

Code: suspicious.env_credential_access
Location: src/adapters/locomo-adapter.ts:28

Environment variable access combined with network send.

Critical

Code: suspicious.env_credential_access
Location: src/adapters/longmemeval-adapter.ts:33

Environment variable access combined with network send.

Critical

Code: suspicious.env_credential_access
Location: src/adapters/mimir-judge.ts:25