Back to skill

Security audit

Memory Bench Designer

Security checks for vulnerabilities and agentic risk

Overview

This skill is mostly purpose-aligned, but it needs review because it tells the agent to run a shell command using a generated name without safe validation.

Install only if you are comfortable with an agent creating benchmark files and result directories, running the memory-bench CLI, and potentially downloading an embedding model. Use a simple alphanumeric scenario name, run it from a scratch directory, inspect the exact command before execution, and avoid existing output paths unless you are willing to overwrite them.

Vulnerability Patterns
  • Insecure DependenciesIntroduces malicious components through unsafe dependency sources
  • Insecure Skill Coding PracticesFinds exploitable flaws such as hardcoded secrets or command injection
  • Skill Instruction HijackingAlters the agent's session goals or safety constraints when the skill loads
  • Agent Memory PoisoningWrites attacker-controlled rules into memory that affect later sessions
  • Remote Payload Retrieval and ExecutionFetches external code whose behavior can change after review
Findings (2)

T09 · Insecure Skill Coding Practices

Error
Location
SKILL.md:51
Finding

Shell Command Injection Through an Unvalidated Scenario Name

Content
View full analysis
.yaml` — the scenario config - `weights-.yaml` — family weights for Judgment (optional) Use templates/scenario.yaml.tmpl and templates/weights.yaml.tmpl as starting points. Substitute the values from Stage 1. Show the user the generated scenario.yaml and ask: *"Look right, or tweak anything before we run?"* Keep this confirmation to one round — don't re-litigate Stage 1. ## Stage 3 — Rollout Invoke the runner via Bash: \``` memory-bench run --scenario scenario-.yaml --out results// --embedding --composite \``` ``` ### Technical Analysis The Skill directs the agent to use a scenario name in filenames and then interpolate that value directly into a command executed through Bash. It does not require validation, normalization, shell escaping, or an argument-array process API. If the generated name contains shell metacharacters such as semicolons, command substitutions, redirection operators, or whitespace, Bash can interpret part of the value as command syntax rather than as a filename. The value may be influenced by the user's requested use-case or scenario name. For example, if a scenario name equivalent to the following were used verbatim: ```text demo; attacker-command ``` the resulting Bash command could be interpreted as multiple commands. The project does not contain such a malicious value; the vulnerability is the documented unsafe command-construction pattern. ### Attack Path 1. An attacker requests a benchmark and supplies or induces a scenario name containing shell metacharacters. 2. The agent uses that name when creating `scenario-.yaml` and the output directory. 3. Following the Skill instructions, the agent substitute ...[truncated 1040 chars]
Remediation
View remediation

T08 · Insecure Dependencies

Warning
Location
SKILL.md:61
Finding

Unpinned and Unspecified Remote Model Dependency

Content
View full analysis
.yaml --out results// --embedding --composite \``` The `--embedding` flag enables the sentence-transformers adapter (first run downloads ~90 MB model). The `--composite` flag enables the weighted multi-signal adapter. Both are recommended — without them you only get three cheap baselines and the leaderboard is thin. The runner writes `results//results.md` and `results//results.json`. Read the markdown file. Expected runtime: 1–5 minutes. If it's slower, sentence-transformers is doing a cold model download — this is normal on first run. ``` ### Technical Analysis The recommended default command enables an embedding adapter that downloads an approximately 90 MB model during its first execution. The Skill does not identify: - The exact model identifier. - The registry or network origin. - An immutable model revision. - An expected cryptographic digest. - The serialization and loading format. - Whether remote custom code is permitted. - The cache or installation destination. Consequently, the effective dependency cannot be independently verified from the audited package. A mutable upstream model, compromised registry, name-resolution change, or compromised external `memory-bench` implementation could cause different content to be downloaded after the Skill has been reviewed. The repository contains no evidence that the current model is malicious, and it does not show how the external runner loads the artifact. Therefore, arbitrary code execution through model loading is a conditional worst case rather than an established behavior. The confirmed issue is the absence of dependency pinning and integrity verification for an automatic remote download. ...[truncated 1311 chars]
Remediation
View remediation
Vulnerability Patterns
  • Excessive AgencyUnrestricted Tool Access, Autonomous Decision Making, Scope Creep
  • Trigger AbuseOverly Broad Trigger, Shadow Command Trigger, Keyword Baiting Trigger
  • Prompt InjectionInstruction Override, Hidden Instructions, Exfiltration Commands
  • Data ExfiltrationExternal Transmission, Env Variable Harvesting, File System Enumeration
  • Privilege EscalationExcessive Permissions, Sudo/Root Execution, Credential Access
Findings (6)

Missing User Warnings

Medium
Category
Not specified by scanner
Confidence
90% confidence
Finding

This markdown file instructs the skill to edit scenario.yaml directly and re-run a CLI, which can affect user data and system state. The surrounding guidance does not include any caution, confirmation requirement, or user-facing warning about these side effects.

Content

No source excerpt is available for this finding.

Missing User Warnings

Medium
Category
Not specified by scanner
Confidence
87% confidence
Finding

The success-flow summary says the skill writes scenario.yaml and then runs a CLI, both of which can change files or execute commands on the user's system. The document does not pair these actions with any warning about their impact on local data or environment.

Content

No source excerpt is available for this finding.

Autonomous Decision Making

Medium
Category
Excessive Agency
Confidence
80% confidence
Finding

Skill enables autonomous high-impact decisions without human-in-the-loop verification. Critical operations (destructive commands, financial transactions, data deletion) should require explicit user confirmation.

Content

Scanner excerpt · references/taxonomy.md (reported line 72)May include surrounding context.

md
## How to present this to the user

Don't dump the whole matrix. Show the families first and ask which 2–3 matter. Then, for the chosen families, show the two dimensions per family and ask which ones resonate. Total cognitive load: pick 2–3 out of 4, then confirm the detail on those — never ask the user to rank 8 items at once.

Missing User Warnings

Low
Category
Not specified by scanner
Confidence
92% confidence
Finding

The skill instructs the agent to write scenario and weights files into the user's current working directory without any explicit up-front warning in the skill metadata or an explicit consent gate immediately before filesystem modification. This is risky because it creates or overwrites local artifacts in whatever directory the agent is running from, which can surprise users, pollute repositories, or interfere with existing files if names collide.

Content

No source excerpt is available for this finding.

Missing User Warnings

Low
Category
Not specified by scanner
Confidence
95% confidence
Finding

The skill directs the agent to invoke a CLI that writes output artifacts to disk and may trigger a model download, but it does not present these side effects as an explicit safety warning in the skill description or require clear user consent at run time. This is dangerous because command execution changes the local environment, consumes bandwidth and storage, and may surprise users who did not expect downloads or result directories to be created.

Content

No source excerpt is available for this finding.

Missing User Warnings

Low
Category
Not specified by scanner
Confidence
78% confidence
Finding

This markdown file applies to SQP-2, and it explicitly states that scenario-npc-cognition.yaml is written and later that the skill reads results and writes output. Those are user-data and filesystem-affecting behaviors, but the example does not include any warning or disclosure about file creation/output locations or that existing files in those paths may be overwritten.

Content

No source excerpt is available for this finding.