Back to skill

Security audit

Improvement Learner

Security checks for vulnerabilities and agentic risk

Overview

This skill has a coherent self-improvement purpose, but it can run target code, send skill contents to an external CLI, rewrite files, delete/restore directories, and auto-commit changes with insufficient safeguards.

Install only in a disposable workspace or sandbox. Prefer --mock and avoid pointing it at untrusted skill directories unless test execution, file mutation, external LLM submission, and git commits have been disabled or reviewed.

Vulnerability Patterns
  • Unauthorized Access and Privilege EscalationObtains permissions beyond the task's legitimate needs
  • Tool Hijacking and SpoofingModifies or replaces tools so legitimate-looking calls execute attacker logic
  • Insecure Skill Coding PracticesFinds exploitable flaws such as hardcoded secrets or command injection
  • Skill Instruction HijackingAlters the agent's session goals or safety constraints when the skill loads
  • Agent Memory PoisoningWrites attacker-controlled rules into memory that affect later sessions
Findings (4)

T09 · Insecure Skill Coding Practices

Error
Location
scripts/self_improve.py:542
Finding

Arbitrary Code Execution Through Automatic Execution of Evaluated Pytest Suites

Content
View full analysis
Remediation
View remediation

T07 · Tool Hijacking and Spoofing

Error
Location
scripts/self_improve.py:1211
Finding

Automatic Git Commit Can Execute Repository-Controlled Hooks

Content
View full analysis
None: """Attempt a git commit (best-effort, non-fatal).""" try: subprocess.run( ["git", "add", "SKILL.md", "references/", "scripts/", "tests/"], cwd=str(skill_path), capture_output=True, timeout=10, ) subprocess.run( ["git", "commit", "-m", message], cwd=str(skill_path), capture_output=True, timeout=10, ) except Exception: pass # non-fatal ``` ### Technical Analysis Accepted improvements automatically invoke `git commit` in the evaluated Skill directory. Git commits can execute hooks such as `pre-commit`, `prepare-commit-msg`, `commit-msg`, and `post-commit`. Although ordinary tracked `.git/hooks` files are not transferred in a standard clone, the evaluated directory may already be a locally prepared repository, may use a malicious `core.hooksPath`, or may inherit unsafe Git configuration. The code neither disables hooks nor verifies the repository configuration before committing. The function also silently ignores exceptions and return codes. As a result, hook execution or partial staging may occur without being clearly reported to the user. ### Attack Path 1. An attacker prepares or influences a local Skill repository. 2. The repository is configured with an executable commit hook, directly or through `core.hooksPath`. 3. The victim runs the self-improvement loop against that directory. 4. A proposed modification passes the score and Pareto checks. 5. `commit_change()` stages files and runs `git commit`. 6. Git executes the configured hook. 7. The hook runs arbitrary commands with the privileges of the victim process. ### Impact Assessment A malicious hook can obtain arbit ...[truncated 515 chars]
Remediation
View remediation

T05 · Unauthorized Access and Privilege Escalation

Error
Location
scripts/self_improve.py:1094
Finding

Symlink-Following File Writes Can Modify Files Outside the Skill Directory

Content
View full analysis
bool: """Redact hardcoded secrets from SKILL.md.""" md = skill_path / "SKILL.md" if not md.exists(): return False content = md.read_text(encoding="utf-8") lowered = content.lower() if "password" not in lowered and "api_key" not in lowered: return False redacted = re.sub( r'(password|api_key)\s*[:=]\s*\S+', r'\1 = ', content, flags=re.IGNORECASE, ) if redacted != content: md.write_text(redacted, encoding="utf-8") return True return False ``` The same unvalidated `read_text()` and `write_text()` pattern is also used by other improvement functions that modify `SKILL.md` and reference files. ### Technical Analysis The code constructs `skill_path / "SKILL.md"` but does not verify that the resulting object is a regular file physically contained within the Skill directory. `Path.exists()`, `Path.read_text()`, and `Path.write_text()` follow symbolic links. An attacker-controlled Skill can therefore provide `SKILL.md` as a symbolic link to another file writable by the victim. If the linked content triggers one of the mutation routines, `write_text()` overwrites the external target. The backup-and-revert logic does not provide a reliable security boundary for external symlink targets. It operates on the Skill directory and may not restore data written through a link outside that directory. ### Attack Path 1. An attacker creates a Skill directory where `SKILL.md` is a symbolic link to a writable file outside the Skill root. 2. The external target contains text that triggers an improvement, such as a `password` or `api_key` assignment. 3. The victim runs the self-improvement loop a ...[truncated 1098 chars]
Remediation
View remediation

T09 · Insecure Skill Coding Practices

Warning
Location
scripts/self_improve.py:90
Finding

Skill Contents Are Sent to the Claude CLI by Default Without a Data-Disclosure Gate

Content
View full analysis
float: """Score a SKILL.md using LLM-as-judge. Returns 0.0–1.0. Calls `claude -p` with a rubric prompt. Falls back to regex heuristics if claude CLI is unavailable or mock=True. """ if mock or not shutil.which("claude"): return _regex_fallback_accuracy(skill_content) prompt = _ACCURACY_JUDGE_PROMPT.format( skill_content=skill_content[:8000] # cap to avoid huge prompts ) try: result = subprocess.run( ["claude", "-p", "--output-format", "json"], input=prompt, capture_output=True, text=True, timeout=120, ) ``` ### Technical Analysis Unless mock mode is explicitly enabled or no `claude` executable is found, the evaluator passes up to 8,000 characters of the target `SKILL.md` to the Claude CLI through standard input. Depending on the CLI's configuration, this content may be transmitted to an external service. No consent prompt, classification check, secret redaction, or allowlist is applied before disclosure. The project's own security scoring occurs as part of evaluation and therefore does not prevent the content from first being included in the LLM prompt. The executable is selected through `shutil.which("claude")` and later invoked by name. A spoofed executable earlier in `PATH` could capture the complete prompt locally. ### Attack Path #### External disclosure path 1. A user evaluates a private or internal Skill without `--mock`. 2. A Claude CLI is available in the environment. 3. The evaluator embeds up to 8,000 characters of `SKILL.md` in the judge prompt. 4. The CLI receives the content and may transmit it to its configured e ...[truncated 938 chars]
Remediation
View remediation
Vulnerability Patterns
  • Data ExfiltrationExternal Transmission, Env Variable Harvesting, File System Enumeration
  • Excessive AgencyUnrestricted Tool Access, Autonomous Decision Making, Scope Creep
  • Rogue AgentSelf-Modification, Session Persistence
  • Trigger AbuseOverly Broad Trigger, Shadow Command Trigger, Keyword Baiting Trigger
  • Behavioral ASTexec() Call, eval() Call, Dynamic Import
Findings (30)

Tp2

High
Category
MCP Tool Poisoning
Confidence
85% confidence
Finding

Mixing characters from multiple Unicode scripts in a single identifier is a common technique to create visually ambiguous tool names.

Content

No source excerpt is available for this finding.

Tp4

High
Category
MCP Tool Poisoning
Confidence
94% confidence
Finding

声明描述的是一个综合性的 skill 评估/自改进能力,核心应包括评分、结构优化、扣分诊断以及特定评估框架(6维、三层记忆、Pareto front)。但给出的代码只负责“追踪评估分数变化趋势”:它从 reports 目录加载历史 JSON,基于最近与更早批次均值比较来判断 improving/stable/declining,输出 Markdown 报告,附带可视化和通用建议。这与声明中的部分内容有轻微相关性(‘追踪评估分数变化趋势’),但并不足以覆盖其主要宣称能力,且缺少多个关键机制和功能,因此属于明显的描述与行为不一致。

Content

No source excerpt is available for this finding.

Ae1

High
Category
analysis-evasion
Confidence
100% confidence
Finding

Referenced artifact was not completely inspected

Content

Scanner excerpt · SKILL.md (reported line 88)May include surrounding context.

md
$ python3 scripts/self_improve.py --skill-path /path/to/skill --max-iterations 1

Ae1

High
Category
analysis-evasion
Confidence
100% confidence
Finding

Referenced artifact was not completely inspected

Content

Scanner excerpt · SKILL.md (reported line 104)May include surrounding context.

md
$ python3 scripts/self_improve.py --skill-path /path/to/skill --max-iterations 1

Ae1

High
Category
analysis-evasion
Confidence
100% confidence
Finding

Referenced artifact was not completely inspected

Content

Scanner excerpt · SKILL.md (reported line 107)May include surrounding context.

md
$ python3 scripts/self_improve.py --skill-path /path/to/skill --max-iterations 1

Ae1

High
Category
analysis-evasion
Confidence
100% confidence
Finding

Referenced artifact was not completely inspected

Content

Scanner excerpt · SKILL.md (reported line 123)May include surrounding context.

md
$ python3 scripts/self_improve.py --skill-path /path/to/skill --max-iterations 1

Ae1

High
Category
analysis-evasion
Confidence
100% confidence
Finding

Referenced artifact was not completely inspected

Content

Scanner excerpt · SKILL.md (reported line 127)May include surrounding context.

md
$ python3 scripts/self_improve.py --skill-path /path/to/skill --max-iterations 1

Context-Inappropriate Capability

High
Category
Not specified by scanner
Confidence
98% confidence
Finding

Executing arbitrary test suites is materially broader than scoring documentation quality and is dangerous when the target skill can be attacker-controlled. In this context, the mismatch makes the issue more severe because users would not reasonably expect code execution from a quality-check skill.

Content

No source excerpt is available for this finding.

Missing User Warnings

High
Category
Not specified by scanner
Confidence
98% confidence
Finding

revert_to_backup deletes the target skill directory with shutil.rmtree before restoring from backup, with no confirmation and no transactional safeguard. If paths are mis-specified or backups are stale/corrupt, this can cause destructive data loss beyond what a user expects from a self-improvement tool.

Content

No source excerpt is available for this finding.

Self-Modification

High
Category
Rogue Agent
Confidence
85% confidence
Finding

Skill modifies its own code, configuration, or behavior at runtime. Self-modification enables an agent to escalate privileges, disable safety constraints, or install persistent backdoors.

Content

Scanner excerpt · tests/test_learner.py (reported line 527)May include surrounding context.

python
(skill / "SKILL.md").write_text("original\n", encoding="utf-8")

        backup = backup_skill(skill)
        # Modify skill
        (skill / "SKILL.md").write_text("modified\n", encoding="utf-8")
        # Revert
        revert_to_backup(skill, backup)

Undeclared Tool Scope

Medium
Category
MCP Least Privilege
Confidence
91% confidence
Finding

The skill advertises shell, file read, and file write workflows through its CLI examples, but it does not declare any explicit tool scope such as allowed-tools or permissions. That creates an authorization gap: downstream systems may over-grant capabilities or operators may not realize the skill can modify files and invoke shell commands, increasing the chance of unsafe execution or privilege creep.

Content

No source excerpt is available for this finding.

Natural-Language Policy Violations

Medium
Category
Not specified by scanner
Confidence
84% confidence
Finding

The natural-language content is predominantly written as Chinese instructions and usage guidance, with no statement that users may choose another language. Under the stated policy, forcing a specific language without opt-in is a locale/language policy concern unless clearly documented and justified.

Content

No source excerpt is available for this finding.

Vague Triggers

Medium
Category
Not specified by scanner
Confidence
93% confidence
Finding

The trigger phrase self-improve is broad and likely to match routine user requests, which can cause unintended activation of a skill that reads, writes, and may invoke shell commands. In a tool-using agent environment, overbroad activation increases the risk of the wrong skill gaining control and performing modifications or evaluations outside the user's intended scope.

Content

No source excerpt is available for this finding.

Vague Triggers

Medium
Category
Not specified by scanner
Confidence
89% confidence
Finding

The trigger 质量评分 is vague and could be invoked by ordinary conversation about ratings or assessments, causing accidental skill activation. Because this skill is positioned to inspect and potentially modify skill artifacts, ambiguous triggers expand the attack surface for prompt collisions and unintended tool use.

Content

No source excerpt is available for this finding.

Vague Triggers

Medium
Category
Not specified by scanner
Confidence
94% confidence
Finding

The phrase 哪里扣分 is common conversational language and may collide with benign requests unrelated to this skill. Such ambiguous activation is risky because it can route general user input into a file- and shell-capable workflow that was not intended for the task.

Content

No source excerpt is available for this finding.

Vague Triggers

Medium
Category
Not specified by scanner
Confidence
92% confidence
Finding

The trigger 自动优化 is extremely broad and maps to a high-risk behavior class because the skill describes iterative modification and re-evaluation of files. Broad optimization language could accidentally activate autonomous editing behavior in response to unrelated requests, leading to unauthorized or surprising changes.

Content

No source excerpt is available for this finding.

Intent-Code Divergence

Medium
Category
Not specified by scanner
Confidence
96% confidence
Finding

The top-level documentation promises benchmark-based validation, implying comparison against a fixed benchmark set. The implementation instead recomputes scores from file-content heuristics, LLM judging, and test execution, so the documentation materially overstates and misdescribes what validation occurs.

Content

No source excerpt is available for this finding.

Missing User Warnings

Medium
Category
Not specified by scanner
Confidence
96% confidence
Finding

The script transmits SKILL.md content to an external evaluator without a clear warning or confirmation. In security context, silent off-box transmission of analyzed content is dangerous because users may supply proprietary prompts, internal process docs, or secrets embedded in skills.

Content

No source excerpt is available for this finding.

Context-Inappropriate Capability

Medium
Category
Not specified by scanner
Confidence
90% confidence
Finding

Invoking an external claude CLI introduces a capability not clearly justified by the stated scope and creates a data-exfiltration path for analyzed content. Because the subprocess is external and behavior depends on local configuration, it also weakens predictability and trust assumptions for a supposedly local quality evaluator.

Content

No source excerpt is available for this finding.

subprocess module call

Medium
Category
Dangerous Code Execution
Confidence
91% confidence
Finding

The script sends up to 8000 characters of SKILL.md to an external claude CLI for scoring. Even without shell injection, this is a real trust-boundary crossing and can leak proprietary or sensitive content off-process/off-host without explicit consent or sanitization.

Content

Scanner excerpt · scripts/self_improve.py (reported line 101)May include surrounding context.

python
)

    try:
        result = subprocess.run(
            ["claude", "-p", "--output-format", "json"],
            input=prompt,
            capture_output=True,

subprocess module call

Medium
Category
Dangerous Code Execution
Confidence
97% confidence
Finding

Running pytest against skill_path/tests executes arbitrary Python from the target skill repository. If the evaluated skill is untrusted, this gives an attacker a straightforward path to arbitrary code execution under the privileges of the user running the learner.

Content

Scanner excerpt · scripts/self_improve.py (reported line 544)May include surrounding context.

python
# Reliability: test results (pure-text skills without scripts/ default to 1.0)
    if has_tests:
        try:
            result = subprocess.run(
                [sys.executable, "-m", "pytest",
                 str(skill_path / "tests"), "-q", "--tb=no"],
                capture_output=True, text=True, timeout=30,

Description-Behavior Mismatch

Medium
Category
Not specified by scanner
Confidence
94% confidence
Finding

The implementation modifies more than documentation: it creates tests, references, backups, and persistent memory/state artifacts. That broader write surface means a user invoking a 'quality check' can unintentionally alter repository contents and local state in ways not implied by the skill description.

Content

No source excerpt is available for this finding.

Description-Behavior Mismatch

Medium
Category
Not specified by scanner
Confidence
95% confidence
Finding

The skill advertises evaluation and SKILL.md optimization, but it also stages and commits repository changes automatically. This hidden capability expansion increases the risk of unauthorized repository mutation and can persist attacker-influenced changes in version control.

Content

No source excerpt is available for this finding.

Context-Inappropriate Capability

Medium
Category
Not specified by scanner
Confidence
92% confidence
Finding

The self-improvement loop performs git operations that are not an obvious requirement of score evaluation. This capability can persist modifications, alter repository history, and create integrity issues, especially if changes are influenced by external scoring or untrusted repository content.

Content

No source excerpt is available for this finding.

Missing User Warnings

Medium
Category
Not specified by scanner
Confidence
97% confidence
Finding

Automatic staging and committing without user confirmation is a real integrity and auditability risk. It can make unintended changes durable, complicate rollback, and create a false appearance that modifications were intentionally reviewed and approved.

Content

No source excerpt is available for this finding.