Autoimprove

• Outcome: Review
• Latest audit: Sep 12, 2026 · 2:06 AM UTC
• Version: 1.0.1

Overview

This autonomous optimization skill is open about running commands and changing repositories, but its headless mode, credential exposure, and destructive rollback behavior need careful review before use.

Install only if you are comfortable with an agent that can edit project files, run project-supplied shell commands, create commits, and reset repository history. Use it first in an isolated clone, container, or disposable worktree with no secrets in the environment, no production kubeconfig or database credentials, and a reviewed improve.md. Avoid headless overnight runs until you have verified the exact scope, commands, backup branch, and recovery process.

A.I.G

Vulnerability Patterns

• Insecure Dependencies: Introduces malicious components through unsafe dependency sources
• Insecure Skill Coding Practices: Finds exploitable flaws such as hardcoded secrets or command injection
• Skill Instruction Hijacking: Alters the agent's session goals or safety constraints when the skill loads
• Agent Memory Poisoning: Writes attacker-controlled rules into memory that affect later sessions
• Remote Payload Retrieval and Execution: Fetches external code whose behavior can change after review

Findings (3)

T09 · Insecure Skill Coding Practices

Error

• Location: SKILL.md:518
• Finding: Unrestricted Execution of Repository-Controlled Shell Commands<![CDATA[ ## Vulnerability Details **File Location**: `SKILL.md:518-526` **Vulnerability Type**: Arbitrary shell-command execution without technical isolation **Risk Level**: High ### Vulnerable Code ```markdown ## Prerequisites and security **Runtime requirements**: git is required. The check commands in your improve.md determine what else is needed (go, python, npm, docker, kubectl, psql, etc.). Verify these are installed before starting. **Credentials**: The agent runs arbitrary shell commands from your improve.md. It inherits whatever credentials are available to the process (AWS keys, DB creds, kubeconfigs, API tokens). Run autoimprove with least-privilege credentials. Strip environment variables you don't want the agent to access. **First run**: Always interactive. The readiness check (Step 1) confirms scope, reviews generated tests, and establishes a baseline before the loop starts. Don't run headless until you've verified one interactive run works correctly. **Backup**: Before headless runs, the readiness check creates a backup branch automatically. The loop uses git commits and resets for rollback, but the backup branch protects against edge cases. **Scope enforcement**: The rules below (NEVER modify files outside scope) are policy constraints, not technical enforcement. The agent follows them in practice, but there is no sandbox preventing out-of-scope edits. For sensitive repos, run in a cloned fork or container where damage is reversible. ``` The corresponding exported protocol also directs the agent to execute repository-supplied commands in `references/protocol.md:43-45`: ```markdown 4. **Test**: If `Check.test` is specified, run it. If tests fail: `git reset --hard HEAD~1`, log as "test_failed", continue to next iteration. 5. **Evaluate**: Run the score command (`Check.run`). If it times out, kill it and treat as failure. ``` ### Technical Analysis The skill treats `Check.test` and `Check.run` values from `improve.md` as executable she ...[truncated 2380 chars]
• Remediation: <![CDATA[ ## Remediation Suggestions 1. Display the exact resolved test and scoring commands and require explicit user approval before their first execution. 2. Disable shell interpretation by default. Parse commands into executable and argument arrays and invoke them without `sh -c`, `bash -c`, or equivalent shell expansion. 3. Reject command substitution, pipelines, redirections, control operators, and multiline commands unless the user explicitly enables an advanced unsafe mode. 4. Execute all repository-controlled commands inside an ephemeral container or virtual machine with: - A minimal environment containing no inherited secrets. - A writable mount limited to an isolated repository worktree. - Read-only system and dependency mounts. - Network access disabled by default. - CPU, memory, process, and execution-time limits. - No Docker socket, SSH agent, cloud metadata endpoint, or host credential mounts. 5. Add a configurable executable allowlist and require separate approval for sensitive tools such as `kubectl`, `psql`, `docker`, cloud CLIs, and package managers. 6. Treat repositories and their `improve.md` files as untrusted input in both interactive and headless modes. 7. Refuse unattended operation until command validation and isolation have been successfully verified. ]]>

T09 · Insecure Skill Coding Practices

Warning

• Location: references/protocol.md:42
• Finding: Unsafe Destructive Git Rollback Using Unconditional Hard Reset<![CDATA[ ## Vulnerability Details **File Location**: `references/protocol.md:42-54` **Vulnerability Type**: Destructive repository rollback and potential loss of unrelated changes **Risk Level**: Medium ### Vulnerable Code ```markdown 4. **Test**: If `Check.test` is specified, run it. If tests fail: `git reset --hard HEAD~1`, log as "test_failed", continue to next iteration. 5. **Evaluate**: Run the score command (`Check.run`). If it times out, kill it and treat as failure. 6. **Score**: Extract the score from stdout using the extraction method in `Check.score`. 7. **Guard**: If `Check.guard` is specified, extract the guard metric and check the threshold. If violated: `git reset --hard HEAD~1`, log as "guard_failed", continue. 8. **Decide**: - If the score command failed (non-zero exit or timeout): `git reset --hard HEAD~1`, log as "error" - If the score improved: keep the commit, update the baseline, log as "kept" - If the score is equal AND `keep_if_equal` is true: keep the commit, log as "kept_equal" - If the score did not improve: `git reset --hard HEAD~1`, log as "discarded" ``` Equivalent hard-reset operations appear throughout `SKILL.md:382-423`. ### Technical Analysis The rollback procedure assumes that `HEAD~1` is always the exact repository state immediately preceding the experiment and that no unrelated tracked changes exist when rollback occurs. It then applies `git reset --hard`, which changes the branch reference and forcibly overwrites tracked files in the index and working tree. The skill initially checks for a clean working tree and creates a backup branch, but these are policy-level mitigations rather than transactional enforcement. Test and scoring commands can modify unrelated tracked files, invoke Git themselves, create commits, switch branches, or run concurrently with another process. Any such state change invalidates the assumption that `HEAD~1` is the safe rollback target. Using a relative revision also makes roll ...[truncated 1624 chars]
• Remediation: <![CDATA[ ## Remediation Suggestions 1. Record the exact pre-experiment commit SHA and experiment commit SHA rather than relying on `HEAD~1`. 2. Immediately before rollback, verify all of the following: - The repository is the expected repository. - The current branch is the expected branch. - `HEAD` exactly matches the recorded experiment commit. - No unexpected staged, unstaged, or untracked changes are present. 3. If any verification fails, stop and request manual recovery instead of performing a hard reset. 4. Run every experiment in a separate disposable Git worktree or temporary clone. Delete the isolated worktree when the experiment is rejected rather than resetting the user's primary worktree. 5. Keep user worktrees read-only to test and scoring commands whenever possible. 6. Restore only the isolated experiment state using the recorded SHA, not a relative revision. 7. Preserve a recovery reference for every experiment before destructive operations. 8. Serialize repository operations and use locking to prevent concurrent agent or tool activity in the same worktree. ]]>

T08 · Insecure Dependencies

Warning

• Location: docs/index.html:634
• Finding: Remote JavaScript Dependencies Loaded Without Subresource Integrity<![CDATA[ ## Vulnerability Details **File Location**: `docs/index.html:634-635` **Vulnerability Type**: Unverified third-party browser dependency execution **Risk Level**: Medium ### Vulnerable Code ```html <script src="https://cdn.jsdelivr.net/npm/reveal.js@5.1.0/dist/reveal.js"></script> <script src="https://cdn.jsdelivr.net/npm/reveal.js@5.1.0/plugin/highlight/highlight.js"></script> ``` The scripts are subsequently trusted by local initialization code in `docs/index.html:636-648`: ```html <script> Reveal.initialize({ hash: true, slideNumber: true, progress: true, controls: true, center: false, transition: 'none', backgroundTransition: 'none', plugins: [RevealHighlight], width: 1280, height: 720, }); </script> ``` ### Technical Analysis The documentation page loads executable JavaScript from a third-party CDN. The package version is pinned, which limits ordinary version drift, but the script elements do not specify Subresource Integrity hashes. The page also lacks an observed Content Security Policy that restricts script execution. Consequently, the browser trusts whatever content the CDN returns for those URLs at viewing time. The effective JavaScript payload can therefore differ from the content reviewed in this project if the CDN, package publication, account, delivery path, or upstream asset is compromised. This differs from a static hyperlink: the remote responses execute automatically in the browser when the documentation page is opened. ### Attack Path 1. An attacker compromises the relevant CDN delivery path, package publication account, upstream release asset, or another component capable of changing the response. 2. Malicious JavaScript is served from one of the referenced URLs. 3. A user opens `docs/index.html` while network access is available. 4. The browser downloads and executes the modified script because no integrity hash verifies its expected contents. 5. The script executes in the docume ...[truncated 766 chars]
• Remediation: <![CDATA[ ## Remediation Suggestions 1. Vendor the required Reveal.js files into the project and serve them locally from reviewed, immutable assets. 2. If CDN hosting is retained, calculate trusted cryptographic hashes and add `integrity` and `crossorigin="anonymous"` attributes to every external script and stylesheet. 3. Add a restrictive Content Security Policy, preferably through HTTP headers when hosted. Restrict `script-src` to approved local files and hash or nonce the inline initialization script. 4. Pin dependencies through a lockfile or reproducible asset-generation process and verify checksums during builds. 5. Periodically review and update Reveal.js and its plugins for known vulnerabilities. 6. Consider producing a self-contained static documentation bundle that requires no network access. ]]>

SkillSpector

Vulnerability Patterns

• Prompt Injection: Instruction Override, Hidden Instructions, Exfiltration Commands
• Data Exfiltration: External Transmission, Env Variable Harvesting, File System Enumeration
• Privilege Escalation: Excessive Permissions, Sudo/Root Execution, Credential Access
• Excessive Agency: Unrestricted Tool Access, Autonomous Decision Making, Scope Creep
• Memory Poisoning: Persistent Context Injection, Context Window Stuffing, Memory Manipulation

Findings (25)

External Model or Provider Selection

High

• Category: Excessive Agency

Content

/autoimprove

# Headless (overnight)
claude -p "run /autoimprove on improve.md" --allowedTools bash,read,write,edit
```

## The `improve.md` format

• Confidence: 94% confidence
• Finding: The README explicitly recommends headless execution with powerful tool permissions: bash, read, write, and edit. In the context of an autonomous optimization loop that can modify files, run commands, and iterate unattended, this creates a high-risk pathway for uncontrolled code execution, repository tampering, secrets exposure, or persistence of harmful changes if the agent behaves unexpectedly or the project contents are adversarial.

Self-Modification

High

• Category: Rogue Agent

Content

2. **Goal-aware threat modeling**: The optimization goal predicts what the agent will try, which predicts what it will break. Generate tests that guard against the failure modes of THAT specific goal:

   **When goal = faster (lower latency, fewer allocations):**
   The agent will skip work, take shortcuts, and remove safety checks.
   - Unicode/multibyte input still works (fast paths assume ASCII)
   - Empty, nil, zero-length inputs don't crash (nil checks removed for speed)
   - Error messages are still correct and informative (error formatting skipped)

• Confidence: 90% confidence
• Finding: Skill modifies its own code, configuration, or behavior at runtime. Self-modification enables an agent to escalate privileges, disable safety constraints, or install persistent backdoors.

Tool Parameter Abuse

High

• Category: Tool Misuse

Content

4. Run the test command (if Check.test is specified)
  5. IF tests fail:
       → git reset --hard HEAD~1
       → Log as "test_failed"
       → consecutive_failures += 1
       → Print: "✗ Experiment {id}: {title} — tests failed"

• Confidence: 93% confidence
• Finding: The workflow performs 'git reset --hard HEAD~1' automatically after failed experiments. Although intended for rollback, hard reset is a destructive operation; combined with a skill that auto-commits and runs in loops, a mistake in repository state handling, wrong working directory, or weak guardrails can irreversibly discard local work or amplify operational damage.

Tool Parameter Abuse

High

• Category: Tool Misuse

Content

7.5. Check guard metrics (if Check.guard is specified):
       → Extract guard metric value from stdout
       → IF guard threshold violated (e.g., error_rate > 0.05):
         → git reset --hard HEAD~1
         → Log as "guard_failed"
         → consecutive_failures += 1
         → Print: "✗ Experiment {id}: {title} — guard violated: {guard_name} = {value}"

• Confidence: 93% confidence
• Finding: This is another automatic 'git reset --hard HEAD~1' path triggered on guard failures. The repetition across multiple branches increases the chance of destructive rollback being exercised frequently during unattended runs, making accidental data loss or incorrect repository rollback more likely.

Tool Parameter Abuse

High

• Category: Tool Misuse

Content

8. DECIDE:
     IF score command failed (non-zero exit, timeout):
       → git reset --hard HEAD~1
       → Log as "error"
       → consecutive_failures += 1

• Confidence: 93% confidence
• Finding: This rollback path hard-resets the repository when the score command errors or times out. Since the score command itself is arbitrary and domain-specific, errors may happen often; coupling arbitrary command execution with destructive history rewriting creates a substantial safety risk in autonomous mode.

Tool Parameter Abuse

High

• Category: Tool Misuse

Content

→ Print: "≈ Experiment {id}: {title} — {score} (kept, equal score)"

     ELSE:
       → git reset --hard HEAD~1
       → Log as "discarded"
       → consecutive_failures += 1
       → Print: "✗ Experiment {id}: {title} — {score} (no improvement)"

• Confidence: 93% confidence
• Finding: The default non-improvement path also executes 'git reset --hard HEAD~1', making destructive rollback part of normal operation rather than an exceptional path. In the broader skill context, where scope enforcement is only policy and commands are arbitrary, this elevates the danger because unattended experimentation can repeatedly mutate and then forcibly rewrite state.

Credential Access

High

• Category: Privilege Escalation

Content

**Runtime requirements**: git is required. The check commands in your improve.md determine what else is needed (go, python, npm, docker, kubectl, psql, etc.). Verify these are installed before starting.

**Credentials**: The agent runs arbitrary shell commands from your improve.md. It inherits whatever credentials are available to the process (AWS keys, DB creds, kubeconfigs, API tokens). Run autoimprove with least-privilege credentials. Strip environment variables you don't want the agent to access.

**First run**: Always interactive. The readiness check (Step 1) confirms scope, reviews generated tests, and establishes a baseline before the loop starts. Don't run headless until you've verified one interactive run works correctly.

• Confidence: 97% confidence
• Finding: The skill explicitly states it runs arbitrary shell commands from improve.md and inherits ambient credentials such as AWS keys, DB credentials, kubeconfigs, and API tokens. Even though this is framed as a warning, it confirms the design permits command execution in a credential-rich environment, which can expose secrets or allow privileged actions if improve.md is malicious, mistaken, or compromised.

Hidden Instructions

High

• Category: Prompt Injection

Content

<div class="reveal">
    <div class="slides">

      <!-- TITLE -->
      <section class="title-slide" data-background="#000">
        <h1>From <span class="red">autoresearch</span><br>to <span class="red">autoimprove</span></h1>
        <p class="subtitle">Generalizing the agentic experiment loop</p>

• Confidence: 70% confidence
• Finding: Hidden instructions were detected in comments or invisible text. These could contain malicious directives. Manual review is recommended.

Hidden Instructions

High

• Category: Prompt Injection

Content

<p><span class="inline-code">exclude</span> prevents the agent from grading its own homework.</p>
      </section>

      <!-- THREE-LAYER CHECK -->
      <section data-background="#000">
        <h2>Three-layer check</h2>
        <div class="red-bar"></div>

• Confidence: 70% confidence
• Finding: Hidden instructions were detected in comments or invisible text. These could contain malicious directives. Manual review is recommended.

Self-Modification

High

• Category: Rogue Agent

Content

<tr><th>Goal</th><th>Agent will try to...</th><th>Tests guard against...</th></tr>
          </thead>
          <tbody>
            <tr><td>Faster</td><td>Skip work, remove checks</td><td>Edge cases, unicode, nil, concurrency</td></tr>
            <tr><td>Smaller</td><td>Remove things, swap deps</td><td>Features still work, runtime deps present</td></tr>
            <tr><td>More accurate</td><td>Overfit, leak data</td><td>Data leakage, reproducibility, valid outputs</td></tr>
            <tr><td>Better RAG</td><td>Game retrieval, stuff context</td><td>Format consistency, hallucination, empty results</td></tr>

• Confidence: 70% confidence
• Finding: Skill modifies its own code, configuration, or behavior at runtime. Self-modification enables an agent to escalate privileges, disable safety constraints, or install persistent backdoors.

Hidden Instructions

High

• Category: Prompt Injection

Content

query expansion, chain-of-thought generation.</code></pre>
      </section>

      <!-- EXAMPLE: AUTOML -->
      <section data-background="#000">
        <h2>Example: tabular ML</h2>
        <div class="red-bar"></div>

• Confidence: 70% confidence
• Finding: Hidden instructions were detected in comments or invisible text. These could contain malicious directives. Manual review is recommended.

Hidden Instructions

High

• Category: Prompt Injection

Content

</table>
      </section>

      <!-- REAL WORLD: ROUND 2 -->
      <section data-background="#000">
        <h2>Real-world test: round 2</h2>
        <div class="red-bar"></div>

• Confidence: 70% confidence
• Finding: Hidden instructions were detected in comments or invisible text. These could contain malicious directives. Manual review is recommended.

External Model or Provider Selection

High

• Category: Excessive Agency

Content

<div class="red-bar"></div>
        <p>The skill runs in Claude Code. The protocol runs anywhere.</p>
        <pre><code class="language-bash">/autoimprove                                  # Claude Code (interactive)
claude -p "run /autoimprove on improve.md"    # headless overnight
/autoimprove --export                         # generates program.md
codex -p "follow program.md"                  # any agent can follow it
gemini -p "follow program.md"</code></pre>

• Confidence: 90% confidence
• Finding: Skill selects an external model or provider that may use a different account or billing plan than the operator expects. Undisclosed model switches can cause unexpected cost or quota consumption.

Missing User Warnings

High

• Confidence: 98% confidence
• Finding: This section authorizes destructive repository operations such as repeated commits and hard resets during an autonomous loop, but provides no mandatory user warning, backup, or checkpointing beyond git history assumptions. That is risky because repositories may contain untracked work, nested repos, or unexpected states, and `git reset --hard` can irreversibly discard local changes.

Tool Parameter Abuse

High

• Category: Tool Misuse

Content

3. **Commit**: `git add <changed files> && git commit -m "autoimprove: <short description>"`. Verify HEAD changed by comparing `git rev-parse HEAD` before and after. If HEAD didn't change, the commit failed — stop and report.

4. **Test**: If `Check.test` is specified, run it. If tests fail: `git reset --hard HEAD~1`, log as "test_failed", continue to next iteration.

5. **Evaluate**: Run the score command (`Check.run`). If it times out, kill it and treat as failure.

• Confidence: 99% confidence
• Finding: The protocol directly instructs use of `git reset --hard HEAD~1` after test failures, which is a destructive command that can wipe local modifications in the working tree. In an autonomous agent context, this is especially dangerous because the command is triggered programmatically and repeatedly, increasing the chance of unintended data loss.

Tool Parameter Abuse

High

• Category: Tool Misuse

Content

6. **Score**: Extract the score from stdout using the extraction method in `Check.score`.

7. **Guard**: If `Check.guard` is specified, extract the guard metric and check the threshold. If violated: `git reset --hard HEAD~1`, log as "guard_failed", continue.

8. **Decide**:
   - If the score command failed (non-zero exit or timeout): `git reset --hard HEAD~1`, log as "error"

• Confidence: 99% confidence
• Finding: This instance again mandates `git reset --hard HEAD~1` on guard failures, coupling an evaluation condition directly to a destructive repository reset. If guard extraction or evaluation logic is wrong, the agent could discard valid work automatically, causing silent loss of changes.

Tool Parameter Abuse

High

• Category: Tool Misuse

Content

7. **Guard**: If `Check.guard` is specified, extract the guard metric and check the threshold. If violated: `git reset --hard HEAD~1`, log as "guard_failed", continue.

8. **Decide**:
   - If the score command failed (non-zero exit or timeout): `git reset --hard HEAD~1`, log as "error"
   - If the score improved: keep the commit, update the baseline, log as "kept"
   - If the score is equal AND `keep_if_equal` is true: keep the commit, log as "kept_equal"
   - If the score did not improve: `git reset --hard HEAD~1`, log as "discarded"

• Confidence: 99% confidence
• Finding: The protocol uses `git reset --hard HEAD~1` when score execution fails or times out, meaning transient command failures can trigger destructive rollback. Because `Check.run` is derived from repository-controlled configuration, malformed or adversarial commands could induce repeated hard resets and data loss.

Tool Parameter Abuse

High

• Category: Tool Misuse

Content

- If the score command failed (non-zero exit or timeout): `git reset --hard HEAD~1`, log as "error"
   - If the score improved: keep the commit, update the baseline, log as "kept"
   - If the score is equal AND `keep_if_equal` is true: keep the commit, log as "kept_equal"
   - If the score did not improve: `git reset --hard HEAD~1`, log as "discarded"

9. **Log**: Save experiment JSON to `.autoimprove/experiments/NNN-slug.json` with this schema:
   ```json

• Confidence: 99% confidence
• Finding: This rollback on non-improving scores again relies on `git reset --hard HEAD~1`, causing destructive cleanup for normal optimization outcomes. In the context of an autonomous optimizer that iterates many times, repeated hard resets materially increase the risk of losing unrelated local work or leaving the repository in an unexpected state.

Vague Triggers

Medium

• Confidence: 88% confidence
• Finding: The README says 'One command' and 'just run `/autoimprove` in your project,' while also stating users 'don't need to know which sub-commands exist.' For a markdown skill description, this does not clearly bound when the skill should activate versus when it should not, and it lacks negative examples or narrower trigger scope.

Vague Triggers

Medium

• Confidence: 97% confidence
• Finding: The skill advertises activation on very broad phrases like 'optimize this', 'improve performance', and 'make this faster', which are common benign requests and can cause the autonomous loop to trigger in contexts the user did not intend. Because this skill executes arbitrary commands from improve.md and can make repeated code changes, accidental invocation materially increases the chance of unsafe repository modification or command execution.

Vague Triggers

Medium

• Confidence: 93% confidence
• Finding: The 'When to Use' guidance includes ambiguous conditions like 'User wants to optimize something measurable' and 'User says optimize/improve/make faster,' which are too general for a high-autonomy skill. In this context, vague activation criteria are dangerous because the skill can scaffold files, run shell commands, and enter a self-directed optimization loop with limited technical enforcement.

Context Window Stuffing

Medium

• Category: Memory Poisoning

Content

<tr><td>Faster</td><td>Skip work, remove checks</td><td>Edge cases, unicode, nil, concurrency</td></tr>
            <tr><td>Smaller</td><td>Remove things, swap deps</td><td>Features still work, runtime deps present</td></tr>
            <tr><td>More accurate</td><td>Overfit, leak data</td><td>Data leakage, reproducibility, valid outputs</td></tr>
            <tr><td>Better RAG</td><td>Game retrieval, stuff context</td><td>Format consistency, hallucination, empty results</td></tr>
            <tr><td>Lower cost</td><td>Downsize, cut redundancy</td><td>Load handling, failover, durability</td></tr>
          </tbody>
        </table>

• Confidence: 85% confidence
• Finding: Skill attempts to fill the context window with filler content, displacing legitimate instructions and safety constraints. This can degrade agent performance or bypass safety boundaries.

Missing User Warnings

Medium

• Confidence: 94% confidence
• Finding: The Kubernetes example explicitly runs `kubectl apply -f k8s/`, which can modify whatever cluster the current kubeconfig points to. In an autonomous optimization skill, that is more dangerous than a normal doc example because users may copy it into unattended loops and accidentally mutate a live cluster without confirmation, environment gating, or warnings.

Missing User Warnings

Medium

• Confidence: 95% confidence
• Finding: The protocol explicitly instructs an autonomous agent to modify files, run commands, and continue without asking for human input. In a skill designed to operate on arbitrary repositories, this is dangerous because it enables unattended changes and execution with no explicit confirmation, safety interlocks, or disclosure of repository impact.

Missing User Warnings

Low

• Confidence: 88% confidence
• Finding: The Docker example runs `docker build -t test .`, which executes instructions from the local Dockerfile and build context. That can run arbitrary build steps, consume resources, access networked package sources, or expose host-linked secrets available during builds, and the example provides no warning about that risk.

Static analysis

No suspicious patterns detected.