T09 · Insecure Skill Coding Practices
- Location
scripts/self_improve.py:542- Finding
Arbitrary Code Execution Through Automatic Execution of Evaluated Pytest Suites
- Content
View full analysis
- Remediation
View remediation
Security audit
Security checks for vulnerabilities and agentic risk
This skill has a coherent self-improvement purpose, but it can run target code, send skill contents to an external CLI, rewrite files, delete/restore directories, and auto-commit changes with insufficient safeguards.
Install only in a disposable workspace or sandbox. Prefer --mock and avoid pointing it at untrusted skill directories unless test execution, file mutation, external LLM submission, and git commits have been disabled or reviewed.
scripts/self_improve.py:542Arbitrary Code Execution Through Automatic Execution of Evaluated Pytest Suites
scripts/self_improve.py:1211Automatic Git Commit Can Execute Repository-Controlled Hooks
scripts/self_improve.py:1094Symlink-Following File Writes Can Modify Files Outside the Skill Directory
scripts/self_improve.py:90Skill Contents Are Sent to the Claude CLI by Default Without a Data-Disclosure Gate
Mixing characters from multiple Unicode scripts in a single identifier is a common technique to create visually ambiguous tool names.
声明描述的是一个综合性的 skill 评估/自改进能力,核心应包括评分、结构优化、扣分诊断以及特定评估框架(6维、三层记忆、Pareto front)。但给出的代码只负责“追踪评估分数变化趋势”:它从 reports 目录加载历史 JSON,基于最近与更早批次均值比较来判断 improving/stable/declining,输出 Markdown 报告,附带可视化和通用建议。这与声明中的部分内容有轻微相关性(‘追踪评估分数变化趋势’),但并不足以覆盖其主要宣称能力,且缺少多个关键机制和功能,因此属于明显的描述与行为不一致。
Referenced artifact was not completely inspected
$ python3 scripts/self_improve.py --skill-path /path/to/skill --max-iterations 1
Referenced artifact was not completely inspected
$ python3 scripts/self_improve.py --skill-path /path/to/skill --max-iterations 1
Referenced artifact was not completely inspected
$ python3 scripts/self_improve.py --skill-path /path/to/skill --max-iterations 1
Referenced artifact was not completely inspected
$ python3 scripts/self_improve.py --skill-path /path/to/skill --max-iterations 1
Referenced artifact was not completely inspected
$ python3 scripts/self_improve.py --skill-path /path/to/skill --max-iterations 1
Executing arbitrary test suites is materially broader than scoring documentation quality and is dangerous when the target skill can be attacker-controlled. In this context, the mismatch makes the issue more severe because users would not reasonably expect code execution from a quality-check skill.
revert_to_backup deletes the target skill directory with shutil.rmtree before restoring from backup, with no confirmation and no transactional safeguard. If paths are mis-specified or backups are stale/corrupt, this can cause destructive data loss beyond what a user expects from a self-improvement tool.
Skill modifies its own code, configuration, or behavior at runtime. Self-modification enables an agent to escalate privileges, disable safety constraints, or install persistent backdoors.
(skill / "SKILL.md").write_text("original\n", encoding="utf-8")
backup = backup_skill(skill)
# Modify skill
(skill / "SKILL.md").write_text("modified\n", encoding="utf-8")
# Revert
revert_to_backup(skill, backup)
The skill advertises shell, file read, and file write workflows through its CLI examples, but it does not declare any explicit tool scope such as allowed-tools or permissions. That creates an authorization gap: downstream systems may over-grant capabilities or operators may not realize the skill can modify files and invoke shell commands, increasing the chance of unsafe execution or privilege creep.
The natural-language content is predominantly written as Chinese instructions and usage guidance, with no statement that users may choose another language. Under the stated policy, forcing a specific language without opt-in is a locale/language policy concern unless clearly documented and justified.
The trigger phrase self-improve is broad and likely to match routine user requests, which can cause unintended activation of a skill that reads, writes, and may invoke shell commands. In a tool-using agent environment, overbroad activation increases the risk of the wrong skill gaining control and performing modifications or evaluations outside the user's intended scope.
The trigger 质量评分 is vague and could be invoked by ordinary conversation about ratings or assessments, causing accidental skill activation. Because this skill is positioned to inspect and potentially modify skill artifacts, ambiguous triggers expand the attack surface for prompt collisions and unintended tool use.
The phrase 哪里扣分 is common conversational language and may collide with benign requests unrelated to this skill. Such ambiguous activation is risky because it can route general user input into a file- and shell-capable workflow that was not intended for the task.
The trigger 自动优化 is extremely broad and maps to a high-risk behavior class because the skill describes iterative modification and re-evaluation of files. Broad optimization language could accidentally activate autonomous editing behavior in response to unrelated requests, leading to unauthorized or surprising changes.
The top-level documentation promises benchmark-based validation, implying comparison against a fixed benchmark set. The implementation instead recomputes scores from file-content heuristics, LLM judging, and test execution, so the documentation materially overstates and misdescribes what validation occurs.
The script transmits SKILL.md content to an external evaluator without a clear warning or confirmation. In security context, silent off-box transmission of analyzed content is dangerous because users may supply proprietary prompts, internal process docs, or secrets embedded in skills.
Invoking an external claude CLI introduces a capability not clearly justified by the stated scope and creates a data-exfiltration path for analyzed content. Because the subprocess is external and behavior depends on local configuration, it also weakens predictability and trust assumptions for a supposedly local quality evaluator.
The script sends up to 8000 characters of SKILL.md to an external claude CLI for scoring. Even without shell injection, this is a real trust-boundary crossing and can leak proprietary or sensitive content off-process/off-host without explicit consent or sanitization.
)
try:
result = subprocess.run(
["claude", "-p", "--output-format", "json"],
input=prompt,
capture_output=True,
Running pytest against skill_path/tests executes arbitrary Python from the target skill repository. If the evaluated skill is untrusted, this gives an attacker a straightforward path to arbitrary code execution under the privileges of the user running the learner.
# Reliability: test results (pure-text skills without scripts/ default to 1.0)
if has_tests:
try:
result = subprocess.run(
[sys.executable, "-m", "pytest",
str(skill_path / "tests"), "-q", "--tb=no"],
capture_output=True, text=True, timeout=30,
The implementation modifies more than documentation: it creates tests, references, backups, and persistent memory/state artifacts. That broader write surface means a user invoking a 'quality check' can unintentionally alter repository contents and local state in ways not implied by the skill description.
The skill advertises evaluation and SKILL.md optimization, but it also stages and commits repository changes automatically. This hidden capability expansion increases the risk of unauthorized repository mutation and can persist attacker-influenced changes in version control.
The self-improvement loop performs git operations that are not an obvious requirement of score evaluation. This capability can persist modifications, alter repository history, and create integrity issues, especially if changes are influenced by external scoring or untrusted repository content.
Automatic staging and committing without user confirmation is a real integrity and auditability risk. It can make unintended changes durable, complicate rollback, and create a false appearance that modifications were intentionally reviewed and approved.