Security audit

Karpathy Autoresearch

Security checks for vulnerabilities and agentic risk

Overview

The skill has a coherent optimization purpose, but it gives broad authority to edit files, run arbitrary eval commands, and change git history without enough scoping or warnings.

Install only if you are comfortable with an agent editing the target project, running evaluation code, and creating or reverting git commits. Use a disposable clone or branch, approve the exact mutable file and eval command, avoid untrusted eval strings, and check that no secrets or unrelated files can be staged before running it.

A.I.G

Vulnerability Patterns

Insecure Skill Coding PracticesFinds exploitable flaws such as hardcoded secrets or command injection
Skill Instruction HijackingAlters the agent's session goals or safety constraints when the skill loads
Agent Memory PoisoningWrites attacker-controlled rules into memory that affect later sessions
Remote Payload Retrieval and ExecutionFetches external code whose behavior can change after review
Embedded Malicious CodeShips malicious scripts inside the skill and executes them locally

Findings (2)

T09 · Insecure Skill Coding Practices

Error

Location: scripts/loop.py:40
Finding: Unrestricted Shell Command Execution Through the Evaluation Argument
Content: View full analysis
Vulnerability Details

File Location: scripts/loop.py, lines 40-50 and line 229
Vulnerability Type: OS command injection through an unrestricted shell command
Risk Level: High

Vulnerable Code

python
def run_eval(eval_cmd: str) -> float: """Run the evaluation command and parse SCORE from output.""" try: result = subprocess.run( eval_cmd, shell=True, capture_output=True, text=True, timeout=300, # 5 min timeout per eval )

The command is obtained directly from a command-line argument:

python
parser.add_argument("--eval", required=True, help="Eval command (must print 'SCORE: <float>')")

Technical Analysis

The --eval value is passed as a string to subprocess.run() with shell=True. The operating system shell therefore interprets shell operators, command substitutions, redirections, pipelines, and environment-variable expansions contained in that value.

No validation, executable allowlist, argument separation, confirmation, or sandbox is applied before execution. Consequently, an evaluation command obtained from an untrusted project, copied from untrusted instructions, or supplied by a malicious operator can append arbitrary commands to an otherwise legitimate evaluator invocation.

The timeout limits execution duration but does not prevent a payload from performing immediate actions or spawning detached processes.

Attack Path

An attacker supplies or recommends a project-specific evaluation command containing shell syntax and a malicious payload.

A user or Agent invokes loop.py with the command through --eval.

main() passes the unmodified string into run_loop() and then run_eval().

run_eval() invokes the command with shell=True.

The shell interprets the injected operators and executes the payload during the baseline evaluation.

If t ...[truncated 1003 chars]
Remediation: View remediation
Remediation Suggestions

Remove shell=True and execute an argument list with shell=False.

Replace the free-form --eval string with a repeatable argument interface or a configuration containing a validated executable and separate arguments.

Restrict evaluator executables to an explicit allowlist or require confirmation before running an unrecognized executable.

Resolve and validate evaluator paths, rejecting unexpected executable locations.

Run evaluators in a sandbox or container with:

A read-only project mount where possible.

Access only to required files.

No inherited secrets or unnecessary environment variables.

Disabled or restricted networking.

CPU, memory, process, and execution-time limits.

Clearly warn users that evaluation code is executable and must be reviewed before use.

If shell syntax is an unavoidable product requirement, require explicit opt-in and display the exact command for confirmation before every execution.

T09 · Insecure Skill Coding Practices

Warning

Location: scripts/loop.py:78
Finding: Overbroad Git Staging Can Record Unrelated Sensitive Files
Content: View full analysis
Vulnerability Details

File Location: scripts/loop.py, lines 78-92; SKILL.md, lines 152-159
Vulnerability Type: Excessive file staging and unintended sensitive-data retention
Risk Level: Medium

Vulnerable Code

python
def git_init(workdir: str): """Initialize git repo if not already initialized.""" subprocess.run(["git", "init"], cwd=workdir, capture_output=True) subprocess.run(["git", "add", "-A"], cwd=workdir, capture_output=True) subprocess.run( ["git", "commit", "-m", "autoresearch: initial state", "--allow-empty"], cwd=workdir, capture_output=True, ) def git_commit(workdir: str, message: str): """Stage all changes and commit.""" subprocess.run(["git", "add", "-A"], cwd=workdir, capture_output=True) subprocess.run(["git", "commit", "-m", message], cwd=workdir, capture_output=True)

The Skill instructions also direct the Agent to stage all repository content:

bash
# Before starting git init git add -A git commit -m "baseline: score {X}" # After each successful mutation git add -A git commit -m "exp-{N}: {what changed} | {old_score} → {new_score}"

Technical Analysis

The optimization process only needs to track the selected mutable file and, optionally, dedicated experiment metadata. Instead, both initialization and each accepted experiment execute git add -A from the supplied working directory.

This stages all unignored additions, modifications, and deletions within the repository scope. If the working directory contains credentials, local configuration, private datasets, generated artifacts, or unrelated work, those files can be included in the baseline or experiment commits.

Removing a sensitive file in a later commit does not remove it from earlier Git history. The project contains no automatic remote upload, so exposure is not immediate; however, committed data may later be disclosed wh ...[truncated 1624 chars]
Remediation: View remediation
Remediation Suggestions

Stage only the selected mutable file:
python
subprocess.run( ["git", "add", "--", validated_mutable_path], cwd=workdir, check=True, )

Add only explicitly required experiment logs rather than the entire working tree.

Resolve both the repository root and mutable path, then verify that the mutable file is inside the intended repository.

Require explicit confirmation before initializing Git in a directory that is not already a repository.

Before committing, inspect the staged file list with git diff --cached --name-only and reject unexpected paths.

Warn or stop when common sensitive filenames are staged, including .env, credential files, private keys, token stores, and cloud configuration.

Encourage a restrictive .gitignore, but do not treat it as a substitute for explicit staging.

Use check=True and surface Git errors rather than suppressing all results.

Update SKILL.md to replace every git add -A instruction with path-limited staging.

If sensitive material has already been committed, rotate affected credentials and rewrite repository history before sharing or pushing it.

SkillSpector

Vulnerability Patterns

Data ExfiltrationExternal Transmission, Env Variable Harvesting, File System Enumeration
Excessive AgencyUnrestricted Tool Access, Autonomous Decision Making, Scope Creep
System Prompt LeakageDirect Leakage, Indirect Extraction, Tool-Based Exfiltration
Tool MisuseTool Parameter Abuse, Chaining Abuse, Unsafe Defaults
Trigger AbuseOverly Broad Trigger, Shadow Command Trigger, Keyword Baiting Trigger

Findings (22)

Tp4

High

Category: MCP Tool Poisoning
Confidence: 97% confidence
Finding: The declared description promises a complete autonomous autoresearch optimizer for OpenClaw skills: repeatedly run the skill, score outputs, mutate the prompt, and keep improvements. The supplied code chunk only implements an evaluation scaffold. Its core scoring function is a stub that raises NotImplementedError, and the script merely reads a file and test cases, iterates through cases, aggregates scores, and prints the result. There is no optimization loop, no mutation strategy, no comparison across variants, no persistence of better versions, and nothing specific to OpenClaw skills beyond example usage strings. This is a materially different and much narrower behavior than the declared purpose, so it should be flagged as a mismatch.
Content: No source excerpt is available for this finding.

Tp4

High

Category: MCP Tool Poisoning
Confidence: 94% confidence
Finding: The description overstates autonomy and specificity. The code does implement part of an autoresearch-style loop: baseline evals, repeated experiments, score comparison, keeping improvements via git commits, reverting regressions, and logging. However, the crucial claimed behavior—autonomously mutating the skill/prompt—is absent. The script explicitly says it is a reference implementation and pauses for manual external mutation, breaking in non-interactive mode. It also works on any file path and any eval command, not specifically OpenClaw skills. Git use and eval execution are consistent supporting details, but the primary declared purpose is materially broader and more autonomous than the actual code.
Content: No source excerpt is available for this finding.

Tool Parameter Abuse

High

Category: Tool Misuse
Confidence: 99% confidence
Finding: This is a classic tool-parameter abuse issue: untrusted input is passed directly into a powerful execution primitive (subprocess.run(..., shell=True)). Because the skill is specifically designed to run evaluation loops repeatedly, abuse of this parameter can yield repeated arbitrary command execution, persistence via filesystem changes, credential access, and broader compromise of the host environment.
Content: Scanner excerpt · scripts/loop.py (reported line 44)May include surrounding context.
python
def run_eval(eval_cmd: str) -> float: """Run the evaluation command and parse SCORE from output.""" try: result = subprocess.run( eval_cmd, shell=True, capture_output=True,

Missing User Warnings

Medium

Category: Not specified by scanner
Confidence: 95% confidence
Finding: The README explicitly states that each experiment results in a git commit and that failed experiments are reverted, but it does not warn users that running the skill will autonomously modify repository state. In a self-improving loop, repeated automated commits/reverts can alter tracked files, disrupt branches, overwrite in-progress work, or create confusing history if run in the wrong repository or without isolation.
Content: No source excerpt is available for this finding.

Undeclared Tool Scope

Medium

Category: MCP Least Privilege
Confidence: 95% confidence
Finding: The skill instructs the agent to read and write project files and execute shell commands such as git init, git commit, git checkout, and running eval scripts, but it declares no explicit tool scope or permission boundaries. This creates a capability mismatch where a broadly triggered skill can modify repository state and execute project logic without clear user-visible constraints or least-privilege controls.
Content: No source excerpt is available for this finding.

Vague Triggers

Medium

Category: Not specified by scanner
Confidence: 96% confidence
Finding: The trigger phrases are broad enough to match common requests like 'improve this skill' or 'run evals on,' which can cause accidental activation in contexts where the user did not intend autonomous file modification or shell execution. In combination with write and shell behaviors, overbroad routing materially increases the chance of unsafe or surprising actions.
Content: No source excerpt is available for this finding.

Vague Triggers

Medium

Category: Not specified by scanner
Confidence: 95% confidence
Finding: The Triggers section lacks activation constraints, so the skill can be invoked without sufficient context about target files, tool permissions, or acceptable side effects. Because this skill is designed to mutate files and run commands in a loop, ambiguity at activation time makes unintended repository changes more likely.
Content: No source excerpt is available for this finding.

Missing User Warnings

Medium

Category: Not specified by scanner
Confidence: 94% confidence
Finding: The skill encourages running eval scripts, backtests, and LLM-as-judge workflows without warning that these may execute arbitrary project code or send proprietary data to external services. In practice, eval harnesses are code execution surfaces, so this omission can lead to data leakage, credential exposure, or execution of untrusted repository logic.
Content: No source excerpt is available for this finding.

Missing User Warnings

Medium

Category: Not specified by scanner
Confidence: 96% confidence
Finding: The skill directs automatic git initialization, commits, checkouts, and repeated file mutations, but it does not prominently warn users that repository state and working files will be changed. This is dangerous because accidental execution can alter source code, overwrite work, create commits, or revert files in ways that are hard to notice until after damage is done.
Content: No source excerpt is available for this finding.

Indirect Prompt Extraction

Medium

Category: System Prompt Leakage
Confidence: 75% confidence
Finding: Skill contains patterns that could indirectly extract system prompts through rephrasing, translation, summarization, or side-channel techniques.
Content: Scanner excerpt · SKILL.md (reported line 142)May include surrounding context.
md
Good mutations change ONE thing at a time: - **Numeric parameters**: Adjust thresholds, weights, window sizes - **Prompt wording**: Rephrase instructions, add/remove constraints - **Structure**: Reorder sections, add examples, remove redundancy - **Rules**: Add a new rule, tighten an existing one, relax a constraint

Missing User Warnings

Medium

Category: Not specified by scanner
Confidence: 89% confidence
Finding: This is a markdown file, so SQP-2 applies to omissions in user-facing warnings. The document presents changes to a live-trading strategy, including removal of a safety threshold and dynamic position sizing, but provides no caution about financial loss, backtest overfitting, or that results are historical and may not generalize to real trading.
Content: No source excerpt is available for this finding.

Missing User Warnings

Medium

Category: Not specified by scanner
Confidence: 84% confidence
Finding: The script modifies repository state by running git init, git add -A, git commit, and git checkout -- <file>, which can alter version-control history and revert file contents. While these actions are visible in the algorithm description, there is no explicit warning that running the script will automatically stage all changes in the working directory and create commits.
Content: No source excerpt is available for this finding.

subprocess module call

Medium

Category: Dangerous Code Execution
Confidence: 99% confidence
Finding: This code executes a user-provided eval_cmd with shell=True, which allows arbitrary shell command execution. In this skill's context, the eval command is a primary input surface and the tool is designed to repeatedly invoke it, so an attacker can run arbitrary OS commands, exfiltrate data, or modify the environment under the guise of evaluation.
Content: Scanner excerpt · scripts/loop.py (reported line 44)May include surrounding context.
python
def run_eval(eval_cmd: str) -> float: """Run the evaluation command and parse SCORE from output.""" try: result = subprocess.run( eval_cmd, shell=True, capture_output=True,

subprocess module call

Medium

Category: Dangerous Code Execution
Confidence: 70% confidence
Finding: subprocess module calls execute external commands. Without careful input validation, this enables command injection.
Content: Scanner excerpt · scripts/loop.py (reported line 80)May include surrounding context.
python
def git_init(workdir: str): """Initialize git repo if not already initialized.""" subprocess.run(["git", "init"], cwd=workdir, capture_output=True) subprocess.run(["git", "add", "-A"], cwd=workdir, capture_output=True) subprocess.run( ["git", "commit", "-m", "autoresearch: initial state", "--allow-empty"],

subprocess module call

Medium

Category: Dangerous Code Execution
Confidence: 70% confidence
Finding: subprocess module calls execute external commands. Without careful input validation, this enables command injection.
Content: Scanner excerpt · scripts/loop.py (reported line 81)May include surrounding context.
python
def git_init(workdir: str): """Initialize git repo if not already initialized.""" subprocess.run(["git", "init"], cwd=workdir, capture_output=True) subprocess.run(["git", "add", "-A"], cwd=workdir, capture_output=True) subprocess.run( ["git", "commit", "-m", "autoresearch: initial state", "--allow-empty"], cwd=workdir,

subprocess module call

Medium

Category: Dangerous Code Execution
Confidence: 70% confidence
Finding: subprocess module calls execute external commands. Without careful input validation, this enables command injection.
Content: Scanner excerpt · scripts/loop.py (reported line 91)May include surrounding context.
python
def git_init(workdir: str): """Initialize git repo if not already initialized.""" subprocess.run(["git", "init"], cwd=workdir, capture_output=True) subprocess.run(["git", "add", "-A"], cwd=workdir, capture_output=True) subprocess.run( ["git", "commit", "-m", "autoresearch: initial state", "--allow-empty"], cwd=workdir,

subprocess module call

Medium

Category: Dangerous Code Execution
Confidence: 70% confidence
Finding: subprocess module calls execute external commands. Without careful input validation, this enables command injection.
Content: Scanner excerpt · scripts/loop.py (reported line 82)May include surrounding context.
python
"""Initialize git repo if not already initialized.""" subprocess.run(["git", "init"], cwd=workdir, capture_output=True) subprocess.run(["git", "add", "-A"], cwd=workdir, capture_output=True) subprocess.run( ["git", "commit", "-m", "autoresearch: initial state", "--allow-empty"], cwd=workdir, capture_output=True,

subprocess module call

Medium

Category: Dangerous Code Execution
Confidence: 70% confidence
Finding: subprocess module calls execute external commands. Without careful input validation, this enables command injection.
Content: Scanner excerpt · scripts/loop.py (reported line 92)May include surrounding context.
python
def git_commit(workdir: str, message: str): """Stage all changes and commit.""" subprocess.run(["git", "add", "-A"], cwd=workdir, capture_output=True) subprocess.run(["git", "commit", "-m", message], cwd=workdir, capture_output=True) def git_revert_file(workdir: str, filepath: str):

subprocess module call

Medium

Category: Dangerous Code Execution
Confidence: 70% confidence
Finding: subprocess module calls execute external commands. Without careful input validation, this enables command injection.
Content: Scanner excerpt · scripts/loop.py (reported line 97)May include surrounding context.
python
def git_revert_file(workdir: str, filepath: str): """Revert a single file to last committed version.""" subprocess.run( ["git", "checkout", "--", filepath], cwd=workdir, capture_output=True )

Description-Behavior Mismatch

Medium

Category: Not specified by scanner
Confidence: 97% confidence
Finding: The manifest describes a skill that autonomously runs repeated optimization by mutating prompts and keeping improvements. In this file, the loop pauses for a human to manually modify the target file and explicitly errors in non-interactive mode, so the actual behavior does not provide the claimed autonomous mutation capability.
Content: No source excerpt is available for this finding.

Missing User Warnings

Medium

Category: Not specified by scanner
Confidence: 87% confidence
Finding: The CLI accepts an eval command that will later be executed through the shell, yet the user-facing interface does not clearly warn that arbitrary system commands will run. In a self-optimization skill, this missing disclosure increases the likelihood that unsafe commands are supplied or socially engineered into use, compounding the underlying command-execution risk.
Content: No source excerpt is available for this finding.

Description-Behavior Mismatch

Low

Category: Not specified by scanner
Confidence: 84% confidence
Finding: The manifest suggests optimizing an entire skill, but this implementation operates on one '--mutable' file and reverts only that file on failure. That is a narrower behavior than the manifest's broad description of autonomously optimizing a skill.
Content: No source excerpt is available for this finding.

Static analysis

No suspicious patterns detected.

Overview

A.I.G

T09 · Insecure Skill Coding Practices

Vulnerability Details

Vulnerable Code

Technical Analysis

Attack Path

Remediation Suggestions

T09 · Insecure Skill Coding Practices

Vulnerability Details

Vulnerable Code

Technical Analysis

Remediation Suggestions

SkillSpector

Tp4

Tp4

Tool Parameter Abuse

Missing User Warnings

Undeclared Tool Scope

Vague Triggers

Vague Triggers

Missing User Warnings

Missing User Warnings

Indirect Prompt Extraction

Missing User Warnings

Missing User Warnings

subprocess module call

subprocess module call

subprocess module call

subprocess module call

subprocess module call

subprocess module call

subprocess module call

Description-Behavior Mismatch

Missing User Warnings

Description-Behavior Mismatch

Static analysis