Auto Improvement Orchestrator Skill

• Outcome: Review
• Latest audit: May 28, 2026 · 10:07 AM UTC
• Version: 1.0.3

Overview

This skill is a coherent auto-improvement pipeline, but it can modify skills, run tests/LLM evaluations, persist state, and read session logs with several under-disclosed or weakly scoped controls.

Install only if you want an experimental, high-authority skill improvement system. Run it in a disposable repo or separate worktree, use mock/dry-run-style modes where available, set explicit state directories and cost/iteration limits, avoid pointing it at sensitive skills or private session logs unless you understand what will be read and sent to LLM backends, and review diffs before keeping generated changes.

SkillSpector

Vulnerability Patterns

• Data Exfiltration: External Transmission, Env Variable Harvesting, File System Enumeration
• Excessive Agency: Unrestricted Tool Access, Autonomous Decision Making, Scope Creep
• Trigger Abuse: Overly Broad Trigger, Shadow Command Trigger, Keyword Baiting Trigger
• Behavioral AST: exec() Call, eval() Call, Dynamic Import
• MCP Tool Poisoning: Hidden Instructions, Unicode Deception, Parameter Description Injection

Findings (57)

subprocess module call

Medium

• Category: Dangerous Code Execution

Content

output_file = Path(tmpdir) / "ai_output.txt"
            output_file.write_text(output, encoding="utf-8")

            result = subprocess.run(
                ["python3", "-m", "pytest", str(test_path), "-v",
                 "--tb=short", f"--rootdir={tmpdir}"],
                capture_output=True,

• Confidence: 89% confidence
• Finding: This code executes pytest against a task-controlled test file, which means arbitrary Python in fixture tests will run with the evaluator's privileges. Although the path checks reduce traversal risk, they do not prevent execution of malicious or overly powerful tests placed under the allowed fixtures directory, and the subprocess inherits the parent environment.

subprocess module call

Medium

• Category: Dangerous Code Execution

Content

f"Respond with ONLY a JSON object: {{\"score\": <float>, \"reasoning\": \"<str>\"}}"
        )
        try:
            result = subprocess.run(
                ["claude", "-p", "--output-format", "json"],
                input=prompt,
                capture_output=True,

• Confidence: 94% confidence
• Finding: The code sends rubric text and untrusted AI output to an external CLI-backed LLM process. This creates a prompt-injection and data-exfiltration surface: attacker-controlled output can influence the judge, and sensitive task or model output may be disclosed to an external model backend without explicit safeguards at the call site.

subprocess module call

Medium

• Category: Dangerous Code Execution

Content

)

    try:
        result = subprocess.run(
            ["claude", "-p", "--output-format", "json"],
            input=prompt,
            capture_output=True,

• Confidence: 93% confidence
• Finding: The code sends attacker-controlled SKILL.md content directly into an external LLM judge and then trusts the returned text enough to influence scoring and later file mutations. This creates an output-injection/prompt-injection boundary where malicious skill content can manipulate the judge result and drive unsafe automated changes, even though the subprocess invocation itself is not shell-injected.

subprocess module call

Medium

• Category: Dangerous Code Execution

Content

# Reliability: test results (pure-text skills without scripts/ default to 1.0)
    if has_tests:
        try:
            result = subprocess.run(
                [sys.executable, "-m", "pytest",
                 str(skill_path / "tests"), "-q", "--tb=no"],
                capture_output=True, text=True, timeout=30,

• Confidence: 90% confidence
• Finding: Running pytest against a target skill's tests executes arbitrary Python from that skill directory. In this auto-improvement context, skills are effectively untrusted input, so evaluation can become arbitrary code execution with the privileges of the orchestrator.

Intent-Code Divergence

Medium

• Confidence: 91% confidence
• Finding: The README presents a command as 'read-only, no changes' but points to `self_improve.py`, which elsewhere is described as applying changes with backup/revert behavior. That mismatch can cause users to run a mutating workflow under the false assumption that it is safe, leading to unintended edits to skills or state directories.

Intent-Code Divergence

Medium

• Confidence: 97% confidence
• Finding: The code and docstrings claim tamper detection via a 'digital signature', but the implementation only recomputes an unauthenticated SHA-256 hash over mutable suite contents. Anyone who can modify the benchmark data can also recompute the signature, so this does not provide authenticity and may cause users to trust benchmark integrity more than warranted.

Intent-Code Divergence

Medium

• Confidence: 96% confidence
• Finding: The code advertises hidden/encrypted tests and a secure execution environment, but the implementation uses reversible XOR with a raw SHA-256 password-derived key and ignores the per-test salt during encryption/decryption. It also exposes full HiddenTest objects, including ciphertext and salts, to visible roles, so anyone with access can recover or manipulate test material far more easily than the documentation suggests, undermining secrecy and fairness guarantees.

Context-Inappropriate Capability

High

• Confidence: 99% confidence
• Finding: The evaluator loads a Python file from a supplied skill path and executes it via importlib, which runs module top-level code immediately and then invokes a discovered entrypoint. In a security-sensitive evaluation pipeline, this means a malicious or untrusted skill can achieve arbitrary code execution in the evaluator's process, access filesystem and network resources, and tamper with benchmark results or the host environment.

Intent-Code Divergence

Medium

• Confidence: 98% confidence
• Finding: The code presents the `signature` field as a digital signature for integrity verification, but it is only a truncated SHA-256 hash over selected fields with no secret key or asymmetric signing key. Any attacker who can modify a receipt file can recompute this value and make `verify_signature()` pass, which can create false trust in tamper-resistance for human review decisions.

Intent-Code Divergence

Medium

• Confidence: 95% confidence
• Finding: The task description says Tier 1 AI patterns should be removed, but the judge requires the output to contain the word "is," creating a contradictory evaluation target. This can systematically reward incorrect behavior, poison benchmark results, and mislead downstream auto-improvement loops into making the deslop skill worse rather than better.

Description-Behavior Mismatch

Medium

• Confidence: 97% confidence
• Finding: The script advertises rollback for a narrow 'generic-skill lane' document scope, but both CLI inputs and receipt/execution metadata are used directly as filesystem paths with no validation that the backup and target stay under an approved root or are limited to expected file types. An attacker who can influence those inputs could overwrite arbitrary files accessible to the process, making this a path-trust/arbitrary file write issue rather than just a documentation mismatch.

Description-Behavior Mismatch

Medium

• Confidence: 88% confidence
• Finding: The script automatically rewrites SKILL.md, creates files, and commits changes despite being framed as an evaluation/improvement pipeline. In practice this creates a dangerous integrity risk: untrusted or manipulated evaluation inputs can cause silent repository modifications that users may not expect.

Description-Behavior Mismatch

Medium

• Confidence: 88% confidence
• Finding: The skill explicitly proposes and applies edits to agent definition files and CLAUDE.md, which goes beyond passive evaluation and turns reflective analysis into code/configuration mutation. Persistent self-modification is risky because conversation content can contain mistakes, prompt injection, or transient user preferences that become durable behavior changes across future sessions.

Context-Inappropriate Capability

Medium

• Confidence: 91% confidence
• Finding: Direct persistent modification of agent files and CLAUDE.md creates a lasting trust-boundary violation: untrusted conversation-derived signals are converted into durable instructions. In the context of an auto-improvement/orchestration ecosystem, this is more dangerous because changes can silently influence future agent behavior, accumulate over time, and be hard to audit if users do not expect local state mutation.

Missing User Warnings

Medium

• Confidence: 86% confidence
• Finding: The README encourages parsing `~/.claude/projects/` session logs, which may contain prompts, code, secrets, file paths, and other sensitive user data, without a prominent privacy warning or minimization guidance. Users may unknowingly ingest or persist sensitive material into feedback stores, increasing exposure and retention risk.

Vague Triggers

Medium

• Confidence: 83% confidence
• Finding: The trigger list contains broad phrases such as 'learner', 'orchestrator', 'skill quality', and similar generic terms that could activate in unrelated conversations. For a skill capable of iterative improvement and file modification, overly broad activation increases the chance of accidental invocation and unintended high-impact actions in the wrong context.

Missing User Warnings

Medium

• Confidence: 90% confidence
• Finding: The design proposes continuous parsing of local Claude session logs and storing user message snippets in a feedback store, but it does not require an explicit, upfront user notice or consent flow before collecting behavioral data derived from real sessions. Even with local-only storage and 200-character limits, snippets can still contain sensitive prompts, credentials, proprietary code fragments, or personal data, making this a real privacy/security weakness rather than just a documentation gap.

Missing User Warnings

Medium

• Confidence: 91% confidence
• Finding: This section proposes an autonomous self-improvement loop that performs code changes, evaluation, commit, and rollback on a skill path, but it does not describe any safety boundaries, user confirmation, sandboxing, or restrictions on which files may be modified. In the context of an auto-improvement orchestrator, such guidance can normalize or encourage unattended code mutation against real repositories, increasing the risk of unintended destructive edits, policy bypasses, or propagation of insecure changes.

Vague Triggers

Medium

• Confidence: 94% confidence
• Finding: The trigger phrase "keep improving" is broad natural language that can match ordinary user requests unrelated to this specific skill. In an autoloop controller, accidental activation is more dangerous than in a passive reference skill because it can launch repeated improvement cycles, consume budget, and persist state across sessions.

Vague Triggers

Medium

• Confidence: 92% confidence
• Finding: The Chinese trigger phrase "一直优化" is similarly broad and conversational, making unintended invocation plausible during normal discussion. Because this skill controls persistent automated improvement with iteration and cost behavior, mistaken routing could cause unnecessary runs, budget use, or unintended modifications to skill artifacts.

Missing User Warnings

Medium

• Confidence: 90% confidence
• Finding: The guide recommends installing a system cron job that repeatedly runs the skill but does not clearly warn that this creates unattended, recurring execution. For a self-improvement/orchestration skill, that omission materially increases the risk of unintended repeated actions, resource consumption, or repeated propagation of bad changes because users may treat the setup as routine infrastructure rather than autonomous execution.

Vague Triggers

Medium

• Confidence: 91% confidence
• Finding: The shell command test uses a broad natural-language prompt ('列出当前目录的文件') without constraints, negative cases, or an explicit safe execution policy. In an auto-improvement/evaluation pipeline, such benchmark prompts can normalize unrestricted shell usage and may cause downstream agents to invoke shell tools too broadly, increasing the risk of command execution beyond intended read-only listing behavior.

Vague Triggers

Medium

• Confidence: 92% confidence
• Finding: The trigger phrase "evaluate improvement" is broad enough to match ordinary user requests about reviewing or improving content, which can cause this skill to activate outside its intended pipeline context. In an auto-improvement orchestration environment, unintended activation could route general prompts into scoring workflows, increasing the chance of mis-execution, context confusion, or unsafe autonomous actions by downstream stages.

Vague Triggers

Medium

• Confidence: 95% confidence
• Finding: The trigger "盲审" is extremely short and semantically broad, so it may be invoked by many unrelated discussions mentioning blind review rather than this specific scoring skill. Because this skill sits in an orchestrated improvement pipeline, accidental invocation can misroute tasks, consume model/tooling resources, and introduce incorrect automated evaluations into later pipeline decisions.

Missing User Warnings

Medium

• Confidence: 87% confidence
• Finding: When --llm-judge is enabled, the code sends candidate data and possibly target SKILL.md content into an LLM backend via llm_judge.evaluate(...). In this auto-improvement/orchestration context, those artifacts can contain sensitive prompts, internal workflow logic, or security guidance; transmitting them to external providers without explicit disclosure, minimization, or policy checks creates a real data exfiltration and compliance risk.

Static analysis

Detected: suspicious.dynamic_code_execution, suspicious.exposed_secret_literal

Dynamic code execution detected.

Critical

• Code: suspicious.dynamic_code_execution
• Location: skills/improvement-discriminator/interfaces/critic_engine.py:379

Dynamic code execution detected.

Critical

• Code: suspicious.dynamic_code_execution
• Location: skills/improvement-discriminator/scripts/score.py:402

Dynamic code execution detected.

Critical

• Code: suspicious.dynamic_code_execution
• Location: skills/improvement-executor/tests/test_execute.py:21

Dynamic code execution detected.

Critical

• Code: suspicious.dynamic_code_execution
• Location: skills/improvement-executor/tests/test_rollback.py:22

Dynamic code execution detected.

Critical

• Code: suspicious.dynamic_code_execution
• Location: skills/improvement-gate/tests/test_gate.py:19

Dynamic code execution detected.

Critical

• Code: suspicious.dynamic_code_execution
• Location: skills/improvement-gate/tests/test_review.py:18

Dynamic code execution detected.

Critical

• Code: suspicious.dynamic_code_execution
• Location: skills/improvement-generator/tests/test_propose.py:25

Dynamic code execution detected.

Critical

• Code: suspicious.dynamic_code_execution
• Location: skills/improvement-orchestrator/tests/test_orchestrate.py:30

File appears to expose a hardcoded API secret or token.

Critical

• Code: suspicious.exposed_secret_literal
• Location: skills/improvement-discriminator/interfaces/critic_engine.py:685

File appears to expose a hardcoded API secret or token.

Critical

• Code: suspicious.exposed_secret_literal
• Location: skills/improvement-discriminator/tests/test_p1_integration.py:115

File appears to expose a hardcoded API secret or token.

Critical

• Code: suspicious.exposed_secret_literal
• Location: skills/improvement-discriminator/tests/test_p2a_integration.py:79