T02 · Agent Memory Poisoning
Error
- Location
tasks/memory/memory-organization/task.yaml:12- Finding
Synthetic benchmark prompt writes preferences into persistent agent memory
- Content
View full analysis
- Remediation
View remediation
Security audit
AgentBench
Security checks for vulnerabilities and agentic risk
Overview
This benchmark skill is mostly coherent, but one bundled task intentionally writes package-authored preferences into the agent’s real persistent memory and its result-signing claims overstate verification.
Install only if you are comfortable running a benchmark that executes bundled local setup scripts and may alter agent memory. Run it in an isolated environment, avoid or manually reset the persistent-memory task, and do not treat the generated signature or 'externally verified' label as independent proof of result integrity.
Vulnerability Patterns
- Agent Memory PoisoningWrites attacker-controlled rules into memory that affect later sessions
- Insecure Skill Coding PracticesFinds exploitable flaws such as hardcoded secrets or command injection
- Skill Instruction HijackingAlters the agent's session goals or safety constraints when the skill loads
- Remote Payload Retrieval and ExecutionFetches external code whose behavior can change after review
- Embedded Malicious CodeShips malicious scripts inside the skill and executes them locally
Findings (2)
T09 · Insecure Skill Coding Practices
Warning
- Location
SKILL.md:199- Finding
Forgeable integrity signature and unsupported externally verified result label
- Content
View full analysis
- Remediation
View remediation
Vulnerability Patterns
- Prompt InjectionInstruction Override, Hidden Instructions, Exfiltration Commands
- Data ExfiltrationExternal Transmission, Env Variable Harvesting, File System Enumeration
- Excessive AgencyUnrestricted Tool Access, Autonomous Decision Making, Scope Creep
- Memory PoisoningPersistent Context Injection, Context Window Stuffing, Memory Manipulation
- Rogue AgentSelf-Modification, Session Persistence
Findings (48)
Tp4
High
- Category
- MCP Tool Poisoning
- Confidence
- 97% confidence
- Finding
The declared description presents this as a broad benchmark suite for evaluating an agent across many task categories. The supplied code chunk does not run or define such a benchmark; it performs setup for a single scenario by creating markdown files representing CI/CD skills and committing them into a new git repository. While file creation could be part of a benchmark, the actual behavior is narrowly focused on repository/task fixture initialization and includes undeclared git operations. That is a materially different primary purpose from the declared benchmark description.
- Content
Tp4
High
- Category
- MCP Tool Poisoning
- Confidence
- 96% confidence
- Finding
The declared description presents the skill as a broad benchmark suite for evaluating an OpenClaw agent across 40 tasks. The supplied code does not implement such a benchmark runner or evaluator. Instead, it sets up a single task environment: a local repository with source files, changelog, checklist, commits, and a version tag to support a release-preparation exercise. While this may be part of a benchmark dataset, the actual code chunk’s direct behavior is task-fixture creation for one scenario, which is materially narrower and different from the declared primary purpose.
- Content
Tp4
High
- Category
- MCP Tool Poisoning
- Confidence
- 96% confidence
- Finding
The declared description presents the skill as a broad benchmark for evaluating an agent across 40 real-world tasks and categories. The supplied code does not perform benchmarking, orchestration, scoring, or broad task evaluation. Instead, it creates a single repository scenario under tasks/research/codebase-archaeology by writing files and making a sequence of git commits to simulate project evolution, including a deliberate regression and skipped test. That behavior is materially narrower and different in primary purpose: it prepares one specific benchmark artifact rather than executing or managing the described benchmark suite.
- Content
Anti-Refusal Statement
High
- Category
- Anti-Refusal
- Confidence
- 80% confidence
- Finding
Skill instructs the agent to omit warnings, disclaimers, or ethical commentary. Stripping safety caveats hides risk from the user and is a common jailbreak preamble.
- Content
yaml Tests debugging silent failures. A 4-stage data pipeline runs without errors but loses ~10% of records. The validate stage crashes on null/empty amount values, the transform stage silently catches and drops those rows, and the export stage writes fewer records with no warning. user_message: | The data pipeline (python pipeline.py) processes data/input.csv into
Missing User Warnings
High
- Category
- Not specified by scanner
- Confidence
- 98% confidence
- Finding
The
cat > fileredirections will replace any existing files at those paths, but the script provides no visible disclosure, backup step, or confirmation prompt. Because this can destroy prior workspace content, it is a safety-critical file write that should be explicitly communicated to the user.- Content
Instruction Override
High
- Category
- Prompt Injection
- Confidence
- 70% confidence
- Finding
This pattern attempts to override system instructions or ignore safety constraints. Without LLM analysis, manual review is recommended.
- Content
sh | `DB_PORT` | 5432 | Database port | | `DB_NAME` | webapp_db | Database name | | `SECRET_KEY` | dev-secret | Application secret key | | `DEBUG` | false | Enable debug mode | MDEOF # =============================================================================
Undeclared Tool Scope
Medium
- Category
- MCP Least Privilege
- Confidence
- 92% confidence
- Finding
The skill orchestrates file reads/writes, temporary workspace creation, result persistence, optional web activity, and shell execution behavior, but it declares no explicit tool scope or permissions. That leaves users and enforcement layers without clear boundaries on what the skill may do, increasing the chance of over-broad execution in environments that rely on manifest metadata for trust and policy decisions.
- Content
Missing User Warnings
Medium
- Category
- Not specified by scanner
- Confidence
- 97% confidence
- Finding
The skill instructs the agent to execute task-local
setup.shscripts directly withbashand laterteardown.sh, treating benchmark content as executable code. Because task definitions are effectively untrusted input, this creates a direct code-execution path that can modify files, exfiltrate data, or abuse available credentials/tools if the benchmark bundle is tampered with or maliciously authored.- Content
Missing User Warnings
Medium
- Category
- Not specified by scanner
- Confidence
- 87% confidence
- Finding
The script runs git init, sets git identity, stages all files, and creates a commit, which changes repository state and may affect the user's workspace history. There is no explicit prompt or warning in the script indicating that version-control state will be created or modified.
- Content
Missing User Warnings
Medium
- Category
- Not specified by scanner
- Confidence
- 90% confidence
- Finding
This code file creates and overwrites multiple files via here-doc redirection and printf, including import.py and several JSON files under the provided workspace. Although the actions are visible in code, there is no confirmation prompt or explicit warning comment/docstring in the shell script that it will modify repository contents and create a git commit.
- Content
Missing User Warnings
Medium
- Category
- Not specified by scanner
- Confidence
- 96% confidence
- Finding
This code immediately changes into a user-supplied workspace, initializes a Git repository, creates directories, and begins writing files without any prompt, print statement, or inline disclosure to the user. These are state-changing filesystem and repository operations, so the lack of visible warning makes the behavior easy to trigger without awareness.
- Content
File System Enumeration
Medium
- Category
- Data Exfiltration
- Confidence
- 60% confidence
- Finding
Code scans file system directories looking for sensitive files. This could be reconnaissance for credential theft.
- Content
sh python -m filesync scan <directory> [-o OUTPUT] ``` - Walk the directory tree using `pathlib.Path.rglob`. - For each file, record: - `path` (relative to the scanned directory) - `size` (bytes)
Session Persistence
Medium
- Category
- Rogue Agent
- Confidence
- 60% confidence
- Finding
Skill establishes unauthorized persistence across sessions via cron jobs, startup scripts, or state files. Session persistence allows an attacker to maintain access beyond the current interaction.
- Content
sh - `created_at`: ISO 8601 timestamp of when the scan was run - `algorithm`: hash algorithm used - `files`: list of file entry objects (sorted by path) - If `-o` is given, write to that file; otherwise print to stdout. - Respect ignore patterns from `.filesyncrc` (see Configuration below). ### 2. `diff`
Missing User Warnings
Medium
- Category
- Not specified by scanner
- Confidence
- 89% confidence
- Finding
This markdown file defines skills that provision infrastructure, configure services, deploy applications, monitor production health, and perform rollback, all of which can materially affect system integrity and availability. The description lists these behaviors but provides no warning or disclosure to users about their operational impact, potential disruption, or reversibility constraints.
- Content
Description-Behavior Mismatch
Medium
- Category
- Not specified by scanner
- Confidence
- 97% confidence
- Finding
The manifest describes a benchmark for testing an OpenClaw agent across real-world tasks, but this script actually populates a git repository with markdown skill files describing a full CI/CD pipeline, including artifact publishing and production deployment. Creating benchmark fixtures is plausible, but the concrete content here is about software delivery workflows rather than benchmarking logic itself, which creates a semantic mismatch between the claimed purpose and the implemented setup behavior.
- Content
Missing User Warnings
Medium
- Category
- Not specified by scanner
- Confidence
- 93% confidence
- Finding
The requirements direct collection of personal and potentially sensitive information, including dietary restrictions, accessibility needs, and emergency contact details, but do not require any privacy notice, consent disclosure, or handling expectations. In a real registration workflow, this can lead to overcollection, uninformed submission of sensitive data, and noncompliant processing of personal data.
- Content
Persistent Context Injection
Medium
- Category
- Memory Poisoning
- Confidence
- 80% confidence
- Finding
Skill injects content designed to persist in agent memory or context across interactions. Persistent injection can alter agent behavior long after the initial interaction.
- Content
yaml - role: user message: | New rule: from now on, always wrap numeric values in backtick formatting (like `42` or `$1,250.00`) for readability in all reports. expect: "acknowledgment"
Natural-Language Policy Violations
Medium
- Category
- Not specified by scanner
- Confidence
- 94% confidence
- Finding
The description states that all outputs must use British English spelling throughout, and the turns/validators reinforce this as a mandatory constraint. This is a language/locale requirement applied unconditionally, with no user choice or region-specific justification, which fits the policy-violation criteria.
- Content
Missing User Warnings
Medium
- Category
- Not specified by scanner
- Confidence
- 88% confidence
- Finding
This code file creates directories, writes multiple project files via heredocs, runs
git init, stages all files, and creates a commit, which are safety-relevant modifications to the user's workspace. Aside from internal comments, there is no user-facing prompt, warning, or disclosure in the script itself about these actions before they occur.- Content
Persistent Context Injection
Medium
- Category
- Memory Poisoning
- Confidence
- 85% confidence
- Finding
The task explicitly directs the agent to persist user instructions for future sessions in memory files, which creates a cross-session prompt injection surface. If an attacker can cause unsafe or overly broad preferences to be stored, those instructions may influence later tasks outside their original context and lead to policy drift or unsafe tool behavior.
- Content
yaml turns: - role: user message: | Remember this for future sessions: I always want you to use TypeScript over JavaScript, prefer pnpm over npm, and never auto-commit without asking me first. expect: "memory-write"
Autonomous Decision Making
Medium
- Category
- Excessive Agency
- Confidence
- 75% confidence
- Finding
Skill enables autonomous high-impact decisions without human-in-the-loop verification. Critical operations (destructive commands, financial transactions, data deletion) should require explicit user confirmation.
- Content
yaml - role: user message: | Remember this for future sessions: I always want you to use TypeScript over JavaScript, prefer pnpm over npm, and never auto-commit without asking me first. expect: "memory-write" validators:
Missing User Warnings
Medium
- Category
- Not specified by scanner
- Confidence
- 85% confidence
- Finding
The script runs
git initand configures commit identity, then later stages and commits all generated files. Repository initialization and commit creation are state-changing operations that can affect user workflows, but the script provides no visible warning or confirmation to the user.- Content
Missing User Warnings
Medium
- Category
- Not specified by scanner
- Confidence
- 89% confidence
- Finding
This code file performs extensive file creation and modification across the workspace, including writing source files, tests, markdown, and Git metadata. Although the operations are commented for developer readability, there is no user-facing disclosure, prompt, or runtime notice before these writes occur.
- Content
Missing User Warnings
Medium
- Category
- Not specified by scanner
- Confidence
- 90% confidence
- Finding
The script immediately changes into a user-supplied path, initializes a Git repository there, and repeatedly overwrites files and creates commits without any confirmation, dry-run mode, or guardrails. In benchmark context this behavior is likely intentional setup logic, but it is still dangerous because running it against the wrong directory can silently destroy or replace user files and mutate repository state.
- Content
Intent-Code Divergence
Medium
- Category
- Not specified by scanner
- Confidence
- 98% confidence
- Finding
The docstring says even-length inputs should return the average of the two middle elements, but the code uses sorted_data[mid] and sorted_data[mid + 1], which shifts one position to the right. This directly contradicts the documented intent for median calculation.
- Content
Static analysis
No suspicious patterns detected.