Improvement Learner

• Outcome: Review
• Latest audit: Sep 16, 2026 · 5:15 AM UTC
• Version: 1.0.0

Overview

This appears to be a real skill-improvement tool, but it can modify repositories, run target test code, send SKILL.md content to an external Claude CLI, and create git commits without clear opt-in.

Install or run this only on a disposable copy of skills you trust. Do not use it on private or third-party skill directories unless you run in a sandbox with no sensitive environment variables and no network. Use --mock to avoid the Claude CLI path, inspect diffs before keeping changes, and do not assume the documented evaluation command is read-only.

A.I.G

Vulnerability Patterns

• Insecure Skill Coding Practices: Finds exploitable flaws such as hardcoded secrets or command injection
• Skill Instruction Hijacking: Alters the agent's session goals or safety constraints when the skill loads
• Agent Memory Poisoning: Writes attacker-controlled rules into memory that affect later sessions
• Remote Payload Retrieval and Execution: Fetches external code whose behavior can change after review
• Embedded Malicious Code: Ships malicious scripts inside the skill and executes them locally

Findings (3)

T09 · Insecure Skill Coding Practices

Error

• Location: scripts/self_improve.py:552

Finding

Arbitrary Code Execution Through Automatic Execution of Target Test Suites

Content

View full analysis

Remediation

View remediation

T09 · Insecure Skill Coding Practices

Warning

• Location: scripts/self_improve.py:88

Finding

Target Skill Content Is Sent to an External LLM CLI Without Explicit Consent

Content

View full analysis

float: """Score a SKILL.md using LLM-as-judge. Returns 0.0–1.0. Calls `claude -p` with a rubric prompt. Falls back to regex heuristics if claude CLI is unavailable or mock=True. """ if mock or not shutil.which("claude"): return _regex_fallback_accuracy(skill_content) prompt = _ACCURACY_JUDGE_PROMPT.format( skill_content=skill_content[:8000] # cap to avoid huge prompts ) try: result = subprocess.run( ["claude", "-p", "--output-format", "json"], input=prompt, capture_output=True, text=True, timeout=120, ) ``` ### Technical Analysis When a program named `claude` is available on `PATH`, the evaluator sends up to 8,000 characters of the target `SKILL.md` to that CLI through standard input. Depending on the CLI's configuration, this content may be transmitted to an external LLM service. The default behavior is selected implicitly by tool availability rather than by affirmative user consent. The documented evaluation workflow does not clearly identify the content being shared, the possible external recipient, or the need to use `--mock` for local-only evaluation. A length limit reduces prompt size but does not provide confidentiality. The content can include proprietary instructions, internal architecture information, personal information, or accidentally committed secrets. The existing secret checks also occur after accuracy evaluation in `evaluate_skill_dimensions()`, so they do not prevent content from first being supplied to the LLM judge. ### Attack Path 1. A user selects a private or sensitive Skill for evaluation. 2. A `claude` executable is ...[truncated 880 chars]

Remediation

View remediation

T09 · Insecure Skill Coding Practices

Warning

• Location: scripts/self_improve.py:1126

Finding

Accepted Improvements Automatically Stage Files and Create Git Commits

Content

View full analysis

None: """Attempt a git commit (best-effort, non-fatal).""" try: subprocess.run( ["git", "add", "SKILL.md", "references/", "scripts/", "tests/"], cwd=str(skill_path), capture_output=True, timeout=10, ) subprocess.run( ["git", "commit", "-m", message], cwd=str(skill_path), capture_output=True, timeout=10, ) except Exception: pass # non-fatal ``` ### Technical Analysis Every accepted self-improvement calls `commit_change()`, which stages broad pathspecs and attempts to create a Git commit without a dedicated opt-in option or confirmation. The staged paths are not restricted to files created or modified by the current operation. Existing user changes under `SKILL.md`, `references/`, `scripts/`, or `tests/` can therefore be included in the generated commit. A Git commit can also execute repository-configured commit hooks under the user's account. Errors are silently ignored, leaving the caller without reliable information about whether files were staged, committed, or left in a partially modified repository state. ### Attack Path 1. A user has pre-existing uncommitted changes under one of the staged paths, or evaluates a repository containing configured Git commit hooks. 2. The user runs the self-improvement workflow. 3. An improvement is accepted by the Pareto checks. 4. `commit_change()` runs `git add` over all four broad pathspecs. 5. Pre-existing changes in those paths are staged along with generated modifications. 6. `git commit` records the staged changes and may execute configured commit hooks. 7. Failures or partial completion are hidden because exceptions and comman ...[truncated 565 chars]

Remediation

View remediation

SkillSpector

Vulnerability Patterns

• Data Exfiltration: External Transmission, Env Variable Harvesting, File System Enumeration
• Excessive Agency: Unrestricted Tool Access, Autonomous Decision Making, Scope Creep
• Rogue Agent: Self-Modification, Session Persistence
• Trigger Abuse: Overly Broad Trigger, Shadow Command Trigger, Keyword Baiting Trigger
• Behavioral AST: exec() Call, eval() Call, Dynamic Import

Findings (20)

Tp2

High

• Category: MCP Tool Poisoning
• Confidence: 85% confidence

Finding

Mixing characters from multiple Unicode scripts in a single identifier is a common technique to create visually ambiguous tool names.

Content

No source excerpt is available for this finding.

Tp4

High

• Category: MCP Tool Poisoning
• Confidence: 91% confidence

Finding

声明描述的是一个较全面的 skill 评估/自改进能力，包含质量评分、结构优化、扣分解释以及特定评估方法论。实际代码仅完成其中一小部分：从已有 JSON 报告读取历史数据，做均值对比式趋势判断，输出建议和可视化。因此，虽然“追踪评估分数变化趋势”这一点与代码相符，但其余核心宣称能力均未在该代码中体现，且代码的主要目的明显偏向历史趋势报告生成，而不是完整的质量评分与自动优化。未见越权资源访问，主要问题是声明能力显著大于实际行为。

Content

No source excerpt is available for this finding.

Context-Inappropriate Capability

High

• Category: Not specified by scanner
• Confidence: 99% confidence

Finding

The tool executes arbitrary pytest suites under the target skill directory, giving analyzed content a path to run code on the host. Given the skill's stated purpose of scoring and self-improvement, this is an unjustified and dangerous capability escalation that can be abused by malicious test files.

Content

No source excerpt is available for this finding.

Self-Modification

High

• Category: Rogue Agent
• Confidence: 85% confidence

Finding

Skill modifies its own code, configuration, or behavior at runtime. Self-modification enables an agent to escalate privileges, disable safety constraints, or install persistent backdoors.

Content

Scanner excerpt · tests/test_learner.py (reported line 531)May include surrounding context.

python

(skill / "SKILL.md").write_text("original\n", encoding="utf-8")

        backup = backup_skill(skill)
        # Modify skill
        (skill / "SKILL.md").write_text("modified\n", encoding="utf-8")
        # Revert
        revert_to_backup(skill, backup)

Undeclared Tool Scope

Medium

• Category: MCP Least Privilege
• Confidence: 91% confidence

Finding

The skill advertises CLI commands that read/write files and invoke shell-based Python scripts, but it declares no explicit tool scope such as permissions or allowed-tools. This creates an authorization gap: an agent may infer broad file and shell access from the instructions and execute them without a least-privilege boundary, increasing the risk of unintended filesystem modification or command execution.

Content

No source excerpt is available for this finding.

Vague Triggers

Medium

• Category: Not specified by scanner
• Confidence: 93% confidence

Finding

This manifest/markdown file defines activation triggers, and "self-improve" is a generic phrase that could naturally appear in many unrelated conversations. Although the surrounding description narrows the skill's purpose, the trigger itself lacks domain-specific constraints or negative examples, increasing the risk of unintended invocation.

Content

No source excerpt is available for this finding.

Vague Triggers

Medium

• Category: Not specified by scanner
• Confidence: 87% confidence

Finding

The phrase means "quality scoring" and is broad enough to apply to many contexts beyond evaluating AI skill files. Because the trigger list does not constrain it to skills or this tool's workflow, it may cause accidental activation.

Content

No source excerpt is available for this finding.

Vague Triggers

Medium

• Category: Not specified by scanner
• Confidence: 90% confidence

Finding

This phrase roughly means "where were points deducted," which is common conversational language and not uniquely tied to this skill. Without explicit scope limits or negative examples, it risks colliding with ordinary feedback or grading discussions.

Content

No source excerpt is available for this finding.

Vague Triggers

Medium

• Category: Not specified by scanner
• Confidence: 91% confidence

Finding

"Automatic optimization" is a general phrase that could refer to many tasks unrelated to improving SKILL.md quality. The trigger list would be safer if it tied optimization explicitly to skill evaluation or self-improvement of the skill file.

Content

No source excerpt is available for this finding.

Context-Inappropriate Capability

Medium

• Category: Not specified by scanner
• Confidence: 94% confidence

Finding

The implementation invokes an external LLM CLI to evaluate SKILL.md content, which introduces undisclosed third-party processing and possible data leakage beyond local self-improvement. Because the skill is advertised mainly as evaluation/optimization, users may not expect contents of their skills to be transmitted outside the machine.

Content

No source excerpt is available for this finding.

Missing User Warnings

Medium

• Category: Not specified by scanner
• Confidence: 95% confidence

Finding

The subprocess call sends skill content to an external LLM without explicit user disclosure, which can expose proprietary prompts, internal procedures, or embedded secrets. This is especially risky because the target skill content is treated as analysis input, and users may assume evaluation remains local.

Content

No source excerpt is available for this finding.

subprocess module call

Medium

• Category: Dangerous Code Execution
• Confidence: 92% confidence

Finding

This subprocess invocation calls an external claude CLI and sends up to 8000 characters of SKILL.md content to it. Even though shell injection is avoided by passing an argument list, it still expands the skill's trust boundary by exfiltrating repository content to an external model/process without explicit consent, which is risky for private or sensitive skills.

Content

Scanner excerpt · scripts/self_improve.py (reported line 101)May include surrounding context.

python

)

    try:
        result = subprocess.run(
            ["claude", "-p", "--output-format", "json"],
            input=prompt,
            capture_output=True,

subprocess module call

Medium

• Category: Dangerous Code Execution
• Confidence: 98% confidence

Finding

Running pytest against a target skill directory executes Python test code from that directory, which may be attacker-controlled if the evaluated skill is untrusted. In this context, a quality-evaluation tool is effectively granting arbitrary code execution during analysis, which can lead to local compromise, secret access, file tampering, or persistence.

Content

Scanner excerpt · scripts/self_improve.py (reported line 556)May include surrounding context.

python

# Base score from tests (0.0-1.0)
    if has_tests:
        try:
            result = subprocess.run(
                [sys.executable, "-m", "pytest",
                 str(skill_path / "tests"), "-q", "--tb=no"],
                capture_output=True, text=True, timeout=30,

Description-Behavior Mismatch

Medium

• Category: Not specified by scanner
• Confidence: 95% confidence

Finding

The implementation performs direct file mutation, backup/restore, reference extraction, autogenerated tests, and git operations, which goes beyond passive evaluation and guidance. This broad write capability can alter repositories in unexpected ways, damage user work, or create supply-chain risks if run against sensitive projects without clear consent.

Content

No source excerpt is available for this finding.

Intent-Code Divergence

Medium

• Category: Not specified by scanner
• Confidence: 84% confidence

Finding

The _apply_coverage_improvement docstring explicitly states the system does not auto-create tests, README, or scripts, framing those artifacts as author-intent-only. Elsewhere, _apply_reliability_improvement does auto-generate tests/test_smoke.py, which contradicts that documented intent at the module behavior level.

Content

No source excerpt is available for this finding.

subprocess module call

Medium

• Category: Dangerous Code Execution
• Confidence: 70% confidence

Finding

subprocess module calls execute external commands. Without careful input validation, this enables command injection.

Content

Scanner excerpt · scripts/self_improve.py (reported line 1129)May include surrounding context.

python

def commit_change(skill_path: Path, message: str) -> None:
    """Attempt a git commit (best-effort, non-fatal)."""
    try:
        subprocess.run(
            ["git", "add", "SKILL.md", "references/", "scripts/", "tests/"],
            cwd=str(skill_path), capture_output=True, timeout=10,
        )

subprocess module call

Medium

• Category: Dangerous Code Execution
• Confidence: 70% confidence

Finding

subprocess module calls execute external commands. Without careful input validation, this enables command injection.

Content

Scanner excerpt · scripts/self_improve.py (reported line 1133)May include surrounding context.

python

["git", "add", "SKILL.md", "references/", "scripts/", "tests/"],
            cwd=str(skill_path), capture_output=True, timeout=10,
        )
        subprocess.run(
            ["git", "commit", "-m", message],
            cwd=str(skill_path), capture_output=True, timeout=10,
        )

Missing User Warnings

Medium

• Category: Not specified by scanner
• Confidence: 96% confidence

Finding

The main loop automatically applies changes, reverts, writes memory snapshots, and may commit results without any user-facing confirmation. In a developer tool operating on repositories, silent mutation materially increases the risk of unintended file changes, destructive edits, and trust-boundary violations.

Content

No source excerpt is available for this finding.

Natural-Language Policy Violations

Medium

• Category: Not specified by scanner
• Confidence: 95% confidence

Finding

This Python file contains user-facing natural-language strings exclusively in Chinese, including the module description, CLI argument descriptions, warnings, and status messages. The stated policy forbids forcing a specific language without user opt-in, and the file does not provide any language selection mechanism or justification for a Chinese-only locale.

Content

No source excerpt is available for this finding.

Autonomous Decision Making

Medium

• Category: Excessive Agency
• Confidence: 85% confidence

Finding

Skill enables autonomous high-impact decisions without human-in-the-loop verification. Critical operations (destructive commands, financial transactions, data deletion) should require explicit user confirmation.

Content

Scanner excerpt · tests/test_learner.py (reported line 164)May include surrounding context.

python

"### 禁止行为\n- ❌ Skip evaluation\n- ❌ Ignore failures\n\n"
            "### 正确做法\n- ✅ Run full pipeline\n- ✅ Check all dims\n\n"
            "MUST run evaluation before reporting.\n"
            "即使 scores look good, NEVER skip verification.\n\n"
            "## Output Artifacts\n\n| Request | Deliverable |\n|---------|------------|\n"
            "| Run tests | JSON report |\n\n"
            "## Related Skills\n\n- **benchmark-store**: For frozen benchmarks\n",

Static analysis

No suspicious patterns detected.