Back to skill

Security audit

Skill Creator Anthropic

Security checks for vulnerabilities and agentic risk

Overview

The skill is mostly a coherent skill-building and evaluation helper, but it includes unsafe local side effects and viewer vulnerabilities that users should review before installing.

Install only if you are comfortable with a skill that can run local Python helpers, invoke the Claude CLI, create and package skill files, write eval workspaces, and start a localhost review server. Avoid using it on untrusted eval outputs or untrusted skill directories until the HTML embedding, arbitrary port-kill behavior, and symlink packaging issue are fixed. Run it without elevated privileges and clean up any viewer process after review.

Vulnerability Patterns
  • Unauthorized Access and Privilege EscalationObtains permissions beyond the task's legitimate needs
  • Insecure Skill Coding PracticesFinds exploitable flaws such as hardcoded secrets or command injection
  • Skill Instruction HijackingAlters the agent's session goals or safety constraints when the skill loads
  • Agent Memory PoisoningWrites attacker-controlled rules into memory that affect later sessions
  • Remote Payload Retrieval and ExecutionFetches external code whose behavior can change after review
Findings (3)

T09 · Insecure Skill Coding Practices

Error
Location
eval-viewer/generate_review.py:279
Finding

Stored JavaScript Injection Through Unescaped Embedded Evaluation Data

Content
View full analysis
` element in `viewer.html`. JSON encoding does not make data safe for embedding in an HTML script context. In particular, it does not escape the HTML parser sequence ``. An attacker-controlled output containing a value such as: ```html ``` can terminate the original script element and introduce a new executable script element. This occurs at the HTML parsing layer before JavaScript string or JSON semantics can protect the content. The affected data is not necessarily trusted. `embed_file()` reads files produced by evaluated skills, while prompts, grading evidence, benchmark notes, and prior outputs can also contain externally influenced content. The declared workflow directs users to open this viewer, making stored payload execution a realistic attack surface. ### Attack Path 1. An attacker supplies a malicious prompt, evaluated skill, input file, or other content that causes an evaluation output file to contain a `` payload. 2. `generate_review.py` reads the generated file and places its contents in the `embedded` object. 3. `json.dumps(embedded)` preserves the literal `` sequence. 4. `generate_html()` inserts the serialized object into the executable script block in `viewer.html`. 5. ...[truncated 1122 chars]
Remediation
View remediation
``` Populate it with safely encoded text and load it with: ```javascript const EMBEDDED_DATA = JSON.parse( document.getElementById("embedded-data").textContent ); ``` 3. At minimum, replace HTML-significant characters before script-context insertion: ```python data_json = json.dumps(embedded).replace("<", "\\u003c") ``` Escaping `<` prevents construction of the `` termination sequence. Escaping `>`, `&`, U+2028, and U+2029 should also be considered for robust cross-environment safety. 4. Add a restrictive Content Security Policy that blocks inline and unauthorized external scripts. Prefer a nonce- or hash-based policy. 5. Add regression tests using output, prompt, grading, and benchmark values containing: - `` - `` - HTML event handlers - Unicode and malformed markup variants 6. Treat all evaluated outputs as hostile content, regardless of file extension. ]]>

T05 · Unauthorized Access and Privilege Escalation

Warning
Location
eval-viewer/generate_review.py:287
Finding

Viewer Startup Terminates Unrelated Processes Listening on the Selected Port

Content
View full analysis
None: """Kill any process listening on the given port.""" try: result = subprocess.run( ["lsof", "-ti", f":{port}"], capture_output=True, text=True, timeout=5, ) for pid_str in result.stdout.strip().split("\n"): if pid_str.strip(): try: os.kill(int(pid_str.strip()), signal.SIGTERM) except (ProcessLookupError, ValueError): pass if result.stdout.strip(): time.sleep(0.5) except subprocess.TimeoutExpired: pass except FileNotFoundError: print("Note: lsof not found, cannot check if port is in use", file=sys.stderr) ``` The function is invoked unconditionally before binding the review server: ```python # Kill any existing process on the target port port = args.port _kill_port(port) ``` ### Technical Analysis The viewer searches for every process listening on the selected port and sends each process `SIGTERM`. It does not verify that the process is an earlier instance of this viewer, confirm process ownership or identity, or request user authorization. The operating system limits `os.kill()` according to the current user's permissions, but this still allows the viewer to terminate unrelated applications owned by the same user. If the script is run under an elevated account, the affected scope becomes correspondingly broader. This behavior is unnecessary for the declared functionality. The code already contains a safer fallback that binds to an automatically selected free port if the preferred port is unavailable. Automatically terminating another service therefore exceeds t ...[truncated 1085 chars]
Remediation
View remediation

T09 · Insecure Skill Coding Practices

Warning
Location
scripts/package_skill.py:91
Finding

Skill Packager Follows Symbolic Links and Can Archive Files Outside the Skill Directory

Content
View full analysis
/home/user/.ssh/id_rsa ``` 3. The user runs: ```bash python -m scripts.package_skill skill-name ``` 4. `rglob()` finds `references/example.txt`. 5. `is_file()` follows the symlink and accepts the external regular file. 6. `zipf.write()` reads the external target and stores its contents as `skill-name/references/example.txt`. 7. The user distributes or upl ...[truncated 713 chars]
Remediation
View remediation
Vulnerability Patterns
  • Data ExfiltrationExternal Transmission, Env Variable Harvesting, File System Enumeration
  • Excessive AgencyUnrestricted Tool Access, Autonomous Decision Making, Scope Creep
  • Rogue AgentSelf-Modification, Session Persistence
  • Trigger AbuseOverly Broad Trigger, Shadow Command Trigger, Keyword Baiting Trigger
  • Behavioral ASTexec() Call, eval() Call, Dynamic Import
Findings (23)

Tp4

High
Category
MCP Tool Poisoning
Confidence
97% confidence
Finding

The declared description presents a broad skill-authoring and optimization tool: creating new skills, editing existing skills, improving performance, running evals, benchmarking, and optimizing descriptions for trigger accuracy. The supplied code chunk instead implements a local eval review utility. Its primary function is to discover eval runs in a workspace, read prompts/outputs/grading, generate an HTML review page, serve it through a lightweight HTTP server, and save human feedback. While this is related to evaluation workflows, it is only a reviewer/viewer component, not a system for creating or modifying skills or running/optimizing them. It also includes undeclared operational behavior such as killing processes on a port and launching a browser. Therefore the actual behavior is materially narrower and different from the declared purpose.

Content

No source excerpt is available for this finding.

Tp4

High
Category
MCP Tool Poisoning
Confidence
99% confidence
Finding

There is a clear mismatch between the declared purpose and the code’s actual behavior. The description claims functionality around skill creation, modification, optimization, evaluation, benchmarking, and trigger-description tuning. However, the supplied code only validates a local skill directory and packages it into a .skill file using zip compression while excluding build artifacts and certain directories like root-level evals. This is a materially different primary purpose, not merely an implementation detail. No evidence in the code supports the declared evaluation or optimization capabilities.

Content

No source excerpt is available for this finding.

Tp4

High
Category
MCP Tool Poisoning
Confidence
97% confidence
Finding

The declared description promises a broad skill authoring, editing, optimization, and performance-evaluation capability. The actual code only performs basic static validation of SKILL.md frontmatter and naming/length rules. While validation could be a supporting component of a larger skill-management tool, this chunk by itself does not implement the main declared capabilities and instead has a materially narrower purpose.

Content

No source excerpt is available for this finding.

Ae1

High
Category
analysis-evasion
Confidence
100% confidence
Finding

Referenced artifact was not completely inspected

Content

Scanner excerpt · SKILL.md (reported line 364)May include surrounding context.

md
1. Read the template from `assets/eval_review.html`

Env Variable Harvesting

High
Category
Data Exfiltration
Confidence
70% confidence
Finding

Code enumerates, copies, or searches environment variables for secrets. Bulk environment access can collect credentials unrelated to the skill's stated purpose.

Content

Scanner excerpt · scripts/improve_description.py (reported line 33)May include surrounding context.

python
# Remove CLAUDECODE env var to allow nesting claude -p inside a
    # Claude Code session. The guard is for interactive terminal conflicts;
    # programmatic subprocess usage is safe. Same pattern as run_eval.py.
    env = {k: v for k, v in os.environ.items() if k != "CLAUDECODE"}

    result = subprocess.run(
        cmd,

Env Variable Harvesting

High
Category
Data Exfiltration
Confidence
70% confidence
Finding

Code enumerates, copies, or searches environment variables for secrets. Bulk environment access can collect credentials unrelated to the skill's stated purpose.

Content

Scanner excerpt · scripts/run_eval.py (reported line 83)May include surrounding context.

python
# Remove CLAUDECODE env var to allow nesting claude -p inside a
    # Claude Code session. The guard is for interactive terminal conflicts;
    # programmatic subprocess usage is safe. Same pattern as run_eval.py.
    env = {k: v for k, v in os.environ.items() if k != "CLAUDECODE"}

    result = subprocess.run(
        cmd,

Agent Config Directory Access

High
Category
Agent Snooping
Confidence
85% confidence
Finding

Skill reads from agent configuration directories (.claude/, .codex/, .gemini/). These directories may contain API keys, personal settings, and other credentials that the skill has no legitimate need to access.

Content

Scanner excerpt · scripts/run_eval.py (reported line 23)May include surrounding context.

python
def find_project_root() -> Path:
    """Find the project root by walking up from cwd looking for .claude/.

    Mimics how Claude Code discovers its project root, so the command file
    we create ends up where claude -p will look for it.

Undeclared Tool Scope

Medium
Category
MCP Least Privilege
Confidence
70% confidence
Finding

Without declared permissions the skill's intent is opaque and cannot be validated.

Content

No source excerpt is available for this finding.

Vague Triggers

Medium
Category
Not specified by scanner
Confidence
96% confidence
Finding

The guidance explicitly recommends making descriptions 'pushy' and triggering even when users do not explicitly ask for the skill, which increases the chance of over-triggering on adjacent requests. In a powerful agent environment, overbroad invocation can cause unintended file operations, shell usage, subprocess launches, or evaluation workflows in contexts where the user did not intend them.

Content

No source excerpt is available for this finding.

Session Persistence

Medium
Category
Rogue Agent
Confidence
85% confidence
Finding

Using nohup to launch a background review server creates session persistence beyond the immediate interaction and can leave long-lived processes serving workspace contents or consuming system resources after the task appears finished. In shared or headless environments, this increases the risk of stale services, unintended data exposure, and orphaned processes if cleanup fails or the PID is lost.

Content

Scanner excerpt · SKILL.md (reported line 238)May include surrounding context.

  1. Launch the viewer with both qualitative outputs and quantitative data:
    bash
    nohup python <skill-creator-path>/eval-viewer/generate_review.py \
      <workspace>/iteration-N \
      --skill-name "my-skill" \
      --benchmark <workspace>/iteration-N/benchmark.json \
    

Intent-Code Divergence

Medium
Category
Not specified by scanner
Confidence
91% confidence
Finding

The manifest for skill-creator says the skill is used to create, modify, improve, and optimize skills, including benchmarking performance. But this section explicitly states that when analyzing benchmark results, the analyzer's purpose is to surface patterns and anomalies "not suggest skill improvements." That is an active contradiction between documented intent and the broader skill purpose of improving/optimizing skills via performance analysis.

Content

No source excerpt is available for this finding.

Description-Behavior Mismatch

Medium
Category
Not specified by scanner
Confidence
91% confidence
Finding

The documentation omits a security-relevant side effect: starting the viewer may terminate existing processes on the selected port. This mismatch increases operational risk because users cannot give informed consent and may unknowingly disrupt other services; while not as severe as the kill behavior itself, hidden destructive behavior is still a legitimate security concern.

Content

No source excerpt is available for this finding.

Context-Inappropriate Capability

Medium
Category
Not specified by scanner
Confidence
99% confidence
Finding

The code forcibly terminates any process listening on the requested port without verifying ownership, identity, or whether the process is safe to kill. In the context of a skill-development/eval tool, this is especially risky because users may run it on shared workstations or alongside other local services, turning a simple viewer launch into an unintended local denial-of-service action.

Content

No source excerpt is available for this finding.

subprocess module call

Medium
Category
Dangerous Code Execution
Confidence
95% confidence
Finding

The subprocess call itself is not command-injection prone because it uses an argument list and the port is parsed as an integer, but it is part of functionality that enumerates processes bound to a port and then enables terminating them. In a local developer tool this creates an unsafe side effect: running the viewer can kill unrelated local services, causing denial of service or data loss if those processes are not this tool's own prior instance.

Content

Scanner excerpt · eval-viewer/generate_review.py (reported line 291)May include surrounding context.

python
def _kill_port(port: int) -> None:
    """Kill any process listening on the given port."""
    try:
        result = subprocess.run(
            ["lsof", "-ti", f":{port}"],
            capture_output=True, text=True, timeout=5,
        )

Missing User Warnings

Medium
Category
Not specified by scanner
Confidence
95% confidence
Finding

The page auto-saves textarea contents to /api/feedback as the user types, which transmits user-entered review content over the network. Although the UI later says feedback has been saved, there is no prior warning near the textarea or instructions that feedback will be auto-posted to a server.

Content

No source excerpt is available for this finding.

Missing User Warnings

Medium
Category
Not specified by scanner
Confidence
92% confidence
Finding

The final submit action sends all review data to /api/feedback with status complete, but the interface does not clearly warn users beforehand that pressing Submit All Reviews will transmit their entered content to a backend service. This is a network transmission of user data lacking explicit disclosure.

Content

No source excerpt is available for this finding.

subprocess module call

Medium
Category
Dangerous Code Execution
Confidence
70% confidence
Finding

subprocess module calls execute external commands. Without careful input validation, this enables command injection.

Content

Scanner excerpt · scripts/improve_description.py (reported line 35)May include surrounding context.

python
# programmatic subprocess usage is safe. Same pattern as run_eval.py.
    env = {k: v for k, v in os.environ.items() if k != "CLAUDECODE"}

    result = subprocess.run(
        cmd,
        input=prompt,
        capture_output=True,

Missing User Warnings

Medium
Category
Not specified by scanner
Confidence
95% confidence
Finding

The script sends full skill content, current descriptions, eval queries, failure history, and prior results to the claude CLI for model processing. If those inputs contain proprietary code, secrets, customer data, or sensitive prompts, they leave the local process boundary and may be exposed to an external service without an explicit user-facing consent or redaction step.

Content

No source excerpt is available for this finding.

Missing User Warnings

Medium
Category
Not specified by scanner
Confidence
94% confidence
Finding

When log_dir is provided, the script writes the full prompt, model response, parsed description, and potentially the rewritten prompt/response to disk. Those logs can contain the entire skill body, evaluation queries, history, and other sensitive operational context, creating a durable local disclosure risk if the directory is shared, backed up, or readable by other users/processes.

Content

No source excerpt is available for this finding.

subprocess module call

Medium
Category
Dangerous Code Execution
Confidence
70% confidence
Finding

subprocess module calls execute external commands. Without careful input validation, this enables command injection.

Content

Scanner excerpt · scripts/run_eval.py (reported line 85)May include surrounding context.

python
# programmatic subprocess usage is safe.
        env = {k: v for k, v in os.environ.items() if k != "CLAUDECODE"}

        process = subprocess.Popen(
            cmd,
            stdout=subprocess.PIPE,
            stderr=subprocess.DEVNULL,

Missing User Warnings

Low
Category
Not specified by scanner
Confidence
83% confidence
Finding

This markdown skill description directs the agent to save results to {outputs_dir}/../grading.json, which is a file write affecting the user's workspace. The document does not include any warning or disclosure that the skill will create or overwrite this file, which is the kind of behavior the missing-warning rule covers for markdown files.

Content

No source excerpt is available for this finding.

Missing User Warnings

Low
Category
Not specified by scanner
Confidence
91% confidence
Finding

This HTML file loads fonts from Google Fonts and a JavaScript library from SheetJS CDN, which causes the user's browser to contact external services and transmit request metadata. There is no visible warning, comment, or user disclosure in the UI explaining that opening the viewer will make third-party network requests.

Content

No source excerpt is available for this finding.

Missing User Warnings

Low
Category
Not specified by scanner
Confidence
81% confidence
Finding

This code launches an external subprocess to run claude -p, which is a safety-relevant operation under the code-file criteria. Although the docstring explains the behavior for developers, there is no user-facing confirmation prompt or visible runtime disclosure when the subprocess is executed.

Content

No source excerpt is available for this finding.