T09 · Insecure Skill Coding Practices
- Location
src/adapters/convomem-adapter.ts:134- Finding
Shell Command Injection Through Dataset-Controlled Search Queries
- Content
View full analysis
- Remediation
View remediation
Security audit
Security checks for vulnerabilities and agentic risk
The skill appears to be a real benchmark tool, but it needs review because crafted inputs can reach shell commands and conversation data can be sent to external model services.
Install only after reviewing the command-execution and data-sharing behavior. Use trusted, public datasets or a restricted account, avoid exposing sensitive environment variables, prefer pinned package versions, and run local-only unless you explicitly intend to send benchmark conversation context to OpenAI. Treat Mimir writeback as persistent application state and avoid using it against a personal or production memory database without isolation and rollback.
src/adapters/convomem-adapter.ts:134Shell Command Injection Through Dataset-Controlled Search Queries
src/scripts/run-all.ts:101Shell Command Injection Through the Run-All Limit Argument
src/adapters/mimir-judge.ts:112Mimir Memory Persistence Permits Command Injection and Persistent Memory Poisoning
src/adapters/convomem-adapter.ts:199Conversation Data Is Sent to OpenAI by Default When an API Key Is Present
Advertising LongMemEval, LoCoMo, ConvoMem, and Mimir support while only implementing a subset and using a different judge creates a misleading security and functionality profile. Because this is a benchmark harness, such mismatches can lead to invalid evaluation decisions, accidental external data transmission, and misplaced trust in results.
Advertising LongMemEval, LoCoMo, ConvoMem, and Mimir support while only implementing a subset and using a different judge creates a misleading security and functionality profile. Because this is a benchmark harness, such mismatches can lead to invalid evaluation decisions, accidental external data transmission, and misplaced trust in results.
Advertising LongMemEval, LoCoMo, ConvoMem, and Mimir support while only implementing a subset and using a different judge creates a misleading security and functionality profile. Because this is a benchmark harness, such mismatches can lead to invalid evaluation decisions, accidental external data transmission, and misplaced trust in results.
Advertising LongMemEval, LoCoMo, ConvoMem, and Mimir support while only implementing a subset and using a different judge creates a misleading security and functionality profile. Because this is a benchmark harness, such mismatches can lead to invalid evaluation decisions, accidental external data transmission, and misplaced trust in results.
The comments state the adapter uses only the memory CLI and no direct external mechanisms, but the implementation also performs multiple HTTP calls to OpenAI and optionally Ollama. This mismatch is dangerous because operators may trust the documentation and run the tool in environments where conversation data must not leave the system.
The README instructs users to execute an unpinned package via npx, which can fetch and run the latest published version at execution time. If the package or one of its publish paths is compromised, users could run attacker-controlled code directly on their machine.
This command uses npx without an explicit version, so it may download and execute whatever package version is current when the user runs it. That creates supply-chain exposure because a malicious or hijacked release could be executed immediately from the documentation example.
The report-generation example invokes an unpinned package through npx, which can result in execution of an unexpected future release. In a benchmark tool context this is still dangerous because users are being encouraged to run code locally, and documentation-driven one-liners are commonly copied verbatim.
The skill declares environment and network-related behavior but does not define an explicit tool scope such as permissions or allowed-tools. In a benchmark skill that can contact external LLM services and consume API keys, this creates an under-specified trust boundary and can lead users or hosting platforms to authorize broader access than intended.
Using npx zouroboros-bench without pinning a version makes execution depend on whatever package version is current at runtime. This introduces a supply-chain risk where a compromised, malicious, or incompatible future release could be fetched and executed automatically.
This unpinned npx zouroboros-bench invocation has the same supply-chain exposure as the other example: the executed code may change over time without review. For a tool with network and API-key usage, silently pulling a new release materially increases risk.
Running npx zouroboros-bench-report without a fixed version allows arbitrary future package contents to be executed. Even a reporting tool can access local benchmark artifacts and potentially exfiltrate data or alter outputs if the package is compromised.
The adapter transmits benchmark questions and retrieved conversation context to external LLM services for query expansion, answer generation, and judging. Because the retrieved context is built from conversation memories and may contain sensitive user data, this creates a real data-exfiltration/privacy risk, especially since transmission happens automatically when API keys are present.
The hardcoded OpenAI endpoint confirms that data is sent to an external service rather than remaining local. In this adapter, the surrounding code shows that transmitted payloads include conversation-derived context, making the endpoint usage materially relevant to privacy and data-governance risk.
Do NOT repeat the question or rephrase it. Focus on finding orthogonal constraints.
Output only 3 queries, one per line, no numbering, no explanation.`;
try {
const resp = await fetch("https://api.openai.com/v1/chat/completions", {
method: "POST",
headers: { "Content-Type": "application/json", Authorization: `Bearer ${OPENAI_API_KEY}` },
body: JSON.stringify({
The hardcoded OpenAI endpoint confirms that data is sent to an external service rather than remaining local. In this adapter, the surrounding code shows that transmitted payloads include conversation-derived context, making the endpoint usage materially relevant to privacy and data-governance risk.
Do NOT repeat the question or rephrase it. Focus on finding orthogonal constraints.
Output only 3 queries, one per line, no numbering, no explanation.`;
try {
const resp = await fetch("https://api.openai.com/v1/chat/completions", {
method: "POST",
headers: { "Content-Type": "application/json", Authorization: `Bearer ${OPENAI_API_KEY}` },
body: JSON.stringify({
Conversation content is embedded into prompts and sent to OpenAI without any user-facing warning, consent gate, or disclosure. In a memory-benchmarking skill, the context is especially likely to contain personal facts and conversational history, so silent transmission materially increases privacy and compliance risk.
The adapter constructs a shell command string and executes it with execSync using a configurable executable path and argument list. If any component such as MEMORY_CLI or future argument sources is attacker-controlled, this can become command injection or unintended command execution with the privileges of the process.
The OpenAI endpoint at this location is used for answer generation, which means conversation-backed prompts leave the local environment. In a tool designed to evaluate memory systems, that context can be rich in sensitive personal details, so the transmission is security-relevant rather than merely functional.
return ollamaGenerate(prompt, ANSWER_MODEL);
}
async function openaiGenerate(prompt, model = "gpt-4o-mini") {
const resp = await fetch("https://api.openai.com/v1/chat/completions", {
method: "POST",
headers: { "Content-Type": "application/json", Authorization: `Bearer ${OPENAI_API_KEY}` },
body: JSON.stringify({
The OpenAI endpoint at this location is used for answer generation, which means conversation-backed prompts leave the local environment. In a tool designed to evaluate memory systems, that context can be rich in sensitive personal details, so the transmission is security-relevant rather than merely functional.
return ollamaGenerate(prompt, ANSWER_MODEL);
}
async function openaiGenerate(prompt, model = "gpt-4o-mini") {
const resp = await fetch("https://api.openai.com/v1/chat/completions", {
method: "POST",
headers: { "Content-Type": "application/json", Authorization: `Bearer ${OPENAI_API_KEY}` },
body: JSON.stringify({
The OpenAI endpoint here is used for judge decisions, exporting evaluation content to a third party. While less dangerous than arbitrary code execution, it is still a true confidentiality issue because benchmark questions, answers, and potentially sensitive hypotheses are disclosed externally.
${criteria}
Respond with exactly one word: CORRECT or INCORRECT`;
const resp = await fetch("https://api.openai.com/v1/chat/completions", {
method: "POST",
headers: { "Content-Type": "application/json", Authorization: `Bearer ${apiKey}` },
body: JSON.stringify({
The OpenAI endpoint here is used for judge decisions, exporting evaluation content to a third party. While less dangerous than arbitrary code execution, it is still a true confidentiality issue because benchmark questions, answers, and potentially sensitive hypotheses are disclosed externally.
${criteria}
Respond with exactly one word: CORRECT or INCORRECT`;
const resp = await fetch("https://api.openai.com/v1/chat/completions", {
method: "POST",
headers: { "Content-Type": "application/json", Authorization: `Bearer ${apiKey}` },
body: JSON.stringify({
The code advertises 'PRODUCTION CLI ONLY' while also making direct network calls to OpenAI and Ollama elsewhere in the file. This mismatch can mislead operators into assuming the adapter is confined to local CLI behavior, reducing scrutiny over outbound data flows and increasing the chance of unintended disclosure.
The hardcoded OpenAI API endpoint confirms that answer generation can leave the local environment and reach a third-party service. In this skill's context, that matters because the transmitted prompt is derived from stored conversation memory, increasing privacy exposure beyond ordinary telemetry.
return ollamaGenerate(prompt, ANSWER_MODEL);
}
async function openaiGenerate(prompt, model = "gpt-4o-mini") {
const resp = await fetch("https://api.openai.com/v1/chat/completions", {
method: "POST",
headers: { "Content-Type": "application/json", Authorization: `Bearer ${OPENAI_API_KEY}` },
body: JSON.stringify({
The hardcoded OpenAI API endpoint confirms that answer generation can leave the local environment and reach a third-party service. In this skill's context, that matters because the transmitted prompt is derived from stored conversation memory, increasing privacy exposure beyond ordinary telemetry.
return ollamaGenerate(prompt, ANSWER_MODEL);
}
async function openaiGenerate(prompt, model = "gpt-4o-mini") {
const resp = await fetch("https://api.openai.com/v1/chat/completions", {
method: "POST",
headers: { "Content-Type": "application/json", Authorization: `Bearer ${OPENAI_API_KEY}` },
body: JSON.stringify({
The adapter sends retrieved conversation context and user questions to external model APIs when generating answers, but there is no explicit disclosure, consent check, or data-minimization control at the call site. In a memory benchmarking tool, the dataset may contain personal or sensitive conversation content, so transmitting it to third-party services can create an unannounced privacy and compliance risk.