Back to skill

Security audit

skill-creator

Security checks for vulnerabilities and agentic risk

Overview

The skill is a coherent skill-building and evaluation tool, but its review helpers have unsafe local side effects and browser-injection risks that users should review before installing.

Install only if you are comfortable running a local skill-development tool that writes evaluation workspaces, invokes the Claude CLI, opens local/browser review pages, and stores feedback files. Avoid running it with elevated privileges. Prefer the static viewer or an unused random port, and be cautious evaluating untrusted skills until the HTML/JavaScript embedding is fixed with context-safe escaping or non-executable JSON storage. After use, confirm any background viewer is stopped and temporary .claude command files were removed.

Vulnerability Patterns
  • Unauthorized Access and Privilege EscalationObtains permissions beyond the task's legitimate needs
  • Insecure Skill Coding PracticesFinds exploitable flaws such as hardcoded secrets or command injection
  • Skill Instruction HijackingAlters the agent's session goals or safety constraints when the skill loads
  • Agent Memory PoisoningWrites attacker-controlled rules into memory that affect later sessions
  • Remote Payload Retrieval and ExecutionFetches external code whose behavior can change after review
Findings (3)

T09 · Insecure Skill Coding Practices

Error
Location
eval-viewer/generate_review.py:279
Finding
Stored Script Injection Through Embedded Evaluation Data<![CDATA[ ## Vulnerability Details **File Location**: `eval-viewer/generate_review.py:279-281` **Vulnerability Type**: Stored script injection in generated HTML **Risk Level**: High ### Vulnerable Code ```python data_json = json.dumps(embedded) return template.replace("/*__EMBEDDED_DATA__*/", f"const EMBEDDED_DATA = {data_json};") ``` The destination in `eval-viewer/viewer.html:647-649` is an executable script block: ```html <script> // ---- Embedded data (injected by generate_review.py) ---- /*__EMBEDDED_DATA__*/ ``` ### Technical Analysis `generate_html()` serializes evaluation data with `json.dumps()` and inserts the result directly into an executable HTML `<script>` element. The embedded object can contain evaluation prompts, generated text files, grading evidence, previous feedback, and other workspace content. JSON string escaping does not make arbitrary data safe for an HTML script context. In particular, `json.dumps()` does not neutralize the HTML parser sequence `</script>`. An evaluated output containing a payload such as: ```html </script><script>/* attacker-controlled JavaScript */</script> ``` would terminate the original script element before JavaScript parsing and introduce a new executable script element. Although most viewer rendering uses `textContent`, that protection occurs only after `EMBEDDED_DATA` has been parsed. It does not protect the initial server-generated script block. ### Attack Path 1. An untrusted or compromised Skill is included in an evaluation. 2. The Skill generates a text output containing a `</script>` sequence followed by attacker-controlled JavaScript. 3. `embed_file()` reads the generated output and includes it in the `embedded` data structure. 4. `generate_html()` serializes the structure and places it directly inside the viewer’s executable script block. 5. The user opens or refreshes the evaluation viewer. 6. The browser terminates the intended script block and executes the injected JavaScript in th ...[truncated 998 chars]
Remediation
<![CDATA[ ## Remediation Suggestions 1. Do not inject raw JSON into an executable script block. 2. Store serialized data in a non-executable element: ```html <script id="embedded-data" type="application/json"></script> ``` Then retrieve and parse its text: ```javascript const EMBEDDED_DATA = JSON.parse( document.getElementById("embedded-data").textContent ); ``` 3. Before embedding JSON into HTML, escape characters significant to the HTML parser. At minimum: ```python data_json = ( json.dumps(embedded) .replace("&", "\\u0026") .replace("<", "\\u003c") .replace(">", "\\u003e") .replace("\u2028", "\\u2028") .replace("\u2029", "\\u2029") ) ``` Escaping `<` prevents creation of a literal `</script>` sequence. 4. Add regression tests using output values containing: ```text </script> </script><script>alert(1)</script> <!-- U+2028 and U+2029 ``` 5. Add a restrictive Content Security Policy. Prefer external, locally packaged scripts and a nonce- or hash-based `script-src` policy rather than allowing arbitrary inline scripts. 6. Treat all prompts, generated artifacts, grading results, benchmark data, and previous feedback as untrusted input. ]]>

T09 · Insecure Skill Coding Practices

Error
Location
assets/eval_review.html:6
Finding
Script and HTML Injection in the Eval-Set Review Template<![CDATA[ ## Vulnerability Details **File Location**: `assets/eval_review.html:6, 41-42, 62-63`; generation instructions at `SKILL.md:414-419` **Vulnerability Type**: Unsafe placeholder substitution into HTML and JavaScript contexts **Risk Level**: High ### Vulnerable Code The template places placeholders directly into HTML contexts: ```html <title>Eval Set Review - __SKILL_NAME_PLACEHOLDER__</title> ``` ```html <h1>Eval Set Review: <span id="skill-name">__SKILL_NAME_PLACEHOLDER__</span></h1> <p class="description">Current description: <span id="skill-desc">__SKILL_DESCRIPTION_PLACEHOLDER__</span></p> ``` It also places evaluation data directly into an executable script context: ```html <script> const EVAL_DATA = __EVAL_DATA_PLACEHOLDER__; let evalItems = [...EVAL_DATA]; ``` The corresponding instructions explicitly direct the Agent to perform placeholder replacement: ```markdown 1. Read the template from `assets/eval_review.html` 2. Replace the placeholders: - `__EVAL_DATA_PLACEHOLDER__` — the JSON array of eval items (no quotes around it — it's a JS variable assignment) - `__SKILL_NAME_PLACEHOLDER__` — the skill's name - `__SKILL_DESCRIPTION_PLACEHOLDER__` — the skill's current description 3. Write to a temp file (e.g., `/tmp/eval_review_<skill-name>.html`) and open it ``` ### Technical Analysis The same replacement mechanism is used for values entering different parser contexts: - Skill name enters the HTML `<title>` and element-body contexts. - Skill description enters an HTML element-body context. - Evaluation data enters an executable JavaScript context. No context-sensitive escaping is required by the instructions or enforced by a generator. A malicious skill name or description can therefore inject HTML markup. Evaluation query data can terminate the enclosing script element with `</script>` even if the array is serialized as otherwise valid JSON. Escaping query text later with `escapeHtml()` does not mitigate this issue beca ...[truncated 1516 chars]
Remediation
<![CDATA[ ## Remediation Suggestions 1. Replace free-form placeholder substitution with a dedicated generator that applies context-sensitive encoding. 2. Set the Skill name and description through `textContent` rather than interpolating them into HTML: ```html <span id="skill-name"></span> <span id="skill-desc"></span> ``` ```javascript document.getElementById("skill-name").textContent = metadata.skillName; document.getElementById("skill-desc").textContent = metadata.skillDescription; ``` 3. Store eval data in a non-executable JSON element: ```html <script id="eval-data" type="application/json">...</script> ``` Parse it with: ```javascript const EVAL_DATA = JSON.parse( document.getElementById("eval-data").textContent ); ``` 4. Escape `<`, `>`, `&`, U+2028, and U+2029 in serialized JSON. In particular, encode `<` as `\u003c` so `</script>` cannot appear literally in the HTML source. 5. HTML-escape any data inserted into `<title>` or normal element-body contexts. 6. Avoid deriving temporary filenames directly from an unvalidated Skill name. Use a generated safe filename or strictly normalize it. 7. Add a restrictive Content Security Policy and avoid executable inline scripts where practical. 8. Add automated tests covering malicious Skill metadata and queries containing closing tags, quotes, event handlers, and script terminators. ]]>

T05 · Unauthorized Access and Privilege Escalation

Warning
Location
eval-viewer/generate_review.py:287
Finding
Evaluation Viewer Terminates Unrelated Processes on Its Selected Port<![CDATA[ ## Vulnerability Details **File Location**: `eval-viewer/generate_review.py:287-305, 434-435` **Vulnerability Type**: Unauthorized process termination and denial of service **Risk Level**: Medium ### Vulnerable Code ```python def _kill_port(port: int) -> None: """Kill any process listening on the given port.""" try: result = subprocess.run( ["lsof", "-ti", f":{port}"], capture_output=True, text=True, timeout=5, ) for pid_str in result.stdout.strip().split("\n"): if pid_str.strip(): try: os.kill(int(pid_str.strip()), signal.SIGTERM) except (ProcessLookupError, ValueError): pass if result.stdout.strip(): time.sleep(0.5) except subprocess.TimeoutExpired: pass except FileNotFoundError: print("Note: lsof not found, cannot check if port is in use", file=sys.stderr) ``` The function is called unconditionally before attempting to bind the server: ```python # Kill any existing process on the target port port = args.port _kill_port(port) ``` The code subsequently demonstrates that safe fallback behavior is already available: ```python try: server = HTTPServer(("127.0.0.1", port), handler) except OSError: # Port still in use after kill attempt — find a free one server = HTTPServer(("127.0.0.1", 0), handler) port = server.server_address[1] ``` ### Technical Analysis The viewer needs a local listening port, but it does not need control over a specific fixed port. `_kill_port()` obtains every PID reported by `lsof` for the selected port and sends `SIGTERM` without checking: - Whether the process belongs to this viewer. - Whether the process belongs to the current user. - Whether the process is critical or unrelated. - Whether the user consented to termination. - Whether selecting another port would satisfy the task. The use of an argument array prevents shel ...[truncated 1302 chars]
Remediation
<![CDATA[ ## Remediation Suggestions 1. Remove `_kill_port()` and do not automatically terminate unidentified processes. 2. Attempt to bind the requested port directly. If binding fails, either: - Bind to port `0` and let the operating system select an available port, or - Report the conflict and ask the user to select another port. 3. Prefer a non-destructive implementation: ```python port = args.port try: server = HTTPServer(("127.0.0.1", port), handler) except OSError: server = HTTPServer(("127.0.0.1", 0), handler) port = server.server_address[1] ``` 4. If replacement of an earlier viewer instance is considered necessary, maintain a viewer-specific PID file and verify the PID’s executable identity, owner, and launch token before requesting termination. 5. Require explicit user confirmation before terminating any process. 6. Avoid running the viewer with elevated privileges. ]]>
Vulnerability Patterns
  • Prompt InjectionInstruction Override, Hidden Instructions, Exfiltration Commands
  • Data ExfiltrationExternal Transmission, Env Variable Harvesting, File System Enumeration
  • Privilege EscalationExcessive Permissions, Sudo/Root Execution, Credential Access
  • Excessive AgencyUnrestricted Tool Access, Autonomous Decision Making, Scope Creep
  • Rogue AgentSelf-Modification, Session Persistence
Findings (20)

Hidden Instructions

High
Category
Prompt Injection
Content
---
name: skill-creator
description: "Create, modify, optimize skills with eval testing, benchmark analysis, and description optimization"
tags: [coding, data, visual, file-based, iterative]
Confidence
60% confidence
Finding
Hidden instructions were detected in comments or invisible text. These could contain malicious directives. Manual review is recommended.

Natural-Language Policy Violations

High
Confidence
99% confidence
Finding
The file includes a substantial Chinese-language section without any gating, translation, or indication that multilingual content is expected. This can create instruction opacity for users and models operating in English, increasing the chance that important behavioral requirements are missed or misapplied and reducing reviewability of the skill’s logic.

Vague Triggers

High
Confidence
96% confidence
Finding
The skill explicitly instructs authors to make descriptions "pushy" to counter undertriggering, which encourages overbroad matching and increases the chance the skill activates for unrelated requests. In a skill system where descriptions govern invocation, this can cause unintended execution of complex workflows, extra tool usage, and user-surprising behavior beyond the user’s actual intent.

Ae1

High
Category
analysis-evasion
Content
1. Read the template from `assets/eval_review.html`
Confidence
100% confidence
Finding
Referenced artifact was not completely inspected

Env Variable Harvesting

High
Category
Data Exfiltration
Content
# Remove CLAUDECODE env var to allow nesting claude -p inside a
    # Claude Code session. The guard is for interactive terminal conflicts;
    # programmatic subprocess usage is safe. Same pattern as run_eval.py.
    env = {k: v for k, v in os.environ.items() if k != "CLAUDECODE"}

    result = subprocess.run(
        cmd,
Confidence
70% confidence
Finding
Code enumerates, copies, or searches environment variables for secrets. Bulk environment access can collect credentials unrelated to the skill's stated purpose.

Env Variable Harvesting

High
Category
Data Exfiltration
Content
# Remove CLAUDECODE env var to allow nesting claude -p inside a
    # Claude Code session. The guard is for interactive terminal conflicts;
    # programmatic subprocess usage is safe. Same pattern as run_eval.py.
    env = {k: v for k, v in os.environ.items() if k != "CLAUDECODE"}

    result = subprocess.run(
        cmd,
Confidence
70% confidence
Finding
Code enumerates, copies, or searches environment variables for secrets. Bulk environment access can collect credentials unrelated to the skill's stated purpose.

Agent Config Directory Access

High
Category
Agent Snooping
Content
def find_project_root() -> Path:
    """Find the project root by walking up from cwd looking for .claude/.

    Mimics how Claude Code discovers its project root, so the command file
    we create ends up where claude -p will look for it.
Confidence
89% confidence
Finding
The script intentionally discovers the project root by locating a `.claude/` directory and then writes a temporary command file into `.claude/commands`. Accessing and modifying an agent configuration directory can influence agent behavior and, if run in an unexpected repository or shared workspace, may alter trusted local agent state or interfere with other commands.

Vague Triggers

Medium
Confidence
87% confidence
Finding
The skill says to "figure out where the user is in this process and then jump in and help them progress through these stages," but gives weak limits on when not to activate. That ambiguity broadens the operational scope of the skill and can lead to accidental invocation for general brainstorming, coding help, or evaluation tasks that do not actually require this skill.

Session Persistence

Medium
Category
Rogue Agent
Content
4. **Launch the viewer** with both qualitative outputs and quantitative data:
   ```bash
   nohup python <skill-creator-path>/eval-viewer/generate_review.py \
     <workspace>/iteration-N \
     --skill-name "my-skill" \
     --benchmark <workspace>/iteration-N/benchmark.json \
Confidence
78% confidence
Finding
The skill instructs launching a background process with `nohup`, which persists beyond the immediate interaction and can continue consuming resources or exposing locally served review artifacts if not properly terminated. Because the workflow also mentions capturing a PID and later killing it, the persistence is partially controlled, but it still increases operational risk if cleanup is skipped or fails.

Intent-Code Divergence

Medium
Confidence
96% confidence
Finding
The file first defines the 'Post-hoc Analyzer Agent' as analyzing a winner/loser comparison and generating actionable improvement suggestions for the losing skill, including a structured JSON output with 'improvement_suggestions'. Later, the 'Analyzing Benchmark Results' section states that the analyzer's purpose is instead to surface benchmark patterns and explicitly says 'DO NOT suggest improvements to the skill,' which is a direct contradiction in intent within the same skill documentation.

Context-Inappropriate Capability

Medium
Confidence
97% confidence
Finding
The script kills whatever process is listening on the requested port before starting its own server, even though serving a review page does not require destructive system actions. This can terminate unrelated local services, developer tools, or security-sensitive processes, causing denial of service and creating an unsafe side effect disproportionate to the tool's stated purpose.

subprocess module call

Medium
Category
Dangerous Code Execution
Content
def _kill_port(port: int) -> None:
    """Kill any process listening on the given port."""
    try:
        result = subprocess.run(
            ["lsof", "-ti", f":{port}"],
            capture_output=True, text=True, timeout=5,
        )
Confidence
90% confidence
Finding
The subprocess call itself is not shell-injection prone because it uses an argument list and the port is parsed as an integer, but it is part of functionality that enumerates and enables termination of arbitrary local processes. In this review tool context, invoking an external system utility to identify PIDs on a port creates unnecessary administrative reach and increases the blast radius of running the script.

Missing User Warnings

Medium
Confidence
95% confidence
Finding
User-entered feedback is auto-saved to /api/feedback after typing, but the UI does not clearly disclose that content is being transmitted to a server before explicit submission. In a review tool, that can leak draft comments, sensitive evaluation notes, or proprietary data to a backend unexpectedly, especially if users assume text remains local until they click submit.

subprocess module call

Medium
Category
Dangerous Code Execution
Content
# programmatic subprocess usage is safe. Same pattern as run_eval.py.
    env = {k: v for k, v in os.environ.items() if k != "CLAUDECODE"}

    result = subprocess.run(
        cmd,
        input=prompt,
        capture_output=True,
Confidence
70% confidence
Finding
subprocess module calls execute external commands. Without careful input validation, this enables command injection.

subprocess module call

Medium
Category
Dangerous Code Execution
Content
# programmatic subprocess usage is safe.
        env = {k: v for k, v in os.environ.items() if k != "CLAUDECODE"}

        process = subprocess.Popen(
            cmd,
            stdout=subprocess.PIPE,
            stderr=subprocess.DEVNULL,
Confidence
70% confidence
Finding
subprocess module calls execute external commands. Without careful input validation, this enables command injection.

Missing User Warnings

Low
Confidence
78% confidence
Finding
This markdown skill instructs the agent to save analysis results to `{output_path}`, which is a file-writing operation. The document does not include any warning, confirmation, or disclosure that it will write to the provided path or overwrite existing contents.

Missing User Warnings

Low
Confidence
85% confidence
Finding
This markdown skill instructs the agent to save results to a sibling `grading.json` file, which is a file write that affects user data/workspace state. The description does not include any warning or disclosure that the skill will create or overwrite this file.

Intent-Code Divergence

Low
Confidence
93% confidence
Finding
The top-level docstring explicitly claims no dependencies beyond the Python standard library. But `_kill_port` executes `lsof`, which is not part of the Python stdlib and may not be installed, so the documentation materially misstates what the script depends on for its advertised behavior.

Missing User Warnings

Low
Confidence
87% confidence
Finding
The code retrieves prior reviews from /api/feedback, which is a network operation involving user review data. There is no clear warning or disclosure in the visible page text that previously saved feedback will be loaded from a server.

Intent-Code Divergence

Low
Confidence
92% confidence
Finding
The comment at L1027 says the current textarea is saved to feedbackMap "but don't POST yet," implying no network submission at that stage. However, the same function proceeds to build a payload and POST it to /api/feedback at L1044-L1048 as part of the submit flow. This is a direct contradiction between inline documentation and actual behavior.

Static analysis

No suspicious patterns detected.