Back to skill

Security audit

Lesson

Security checks for vulnerabilities and agentic risk

Overview

This is mostly a coherent skill-building toolkit, but it needs Review because it bundles unrelated content and includes unsafe local viewer behavior that can run injected browser code or terminate unrelated local processes.

Install only if you specifically need a skill-authoring and eval toolkit and are comfortable with it running local Python scripts, claude subprocesses, browser/local-server tooling, and workspace file writes. Before use, remove the bundled daily-menu directory and .codebuddy memory file, prefer static review output or a known-free port, and treat eval outputs as untrusted because the current viewer can execute injected JavaScript from crafted outputs.

Vulnerability Patterns
  • Unauthorized Access and Privilege EscalationObtains permissions beyond the task's legitimate needs
  • Insecure Skill Coding PracticesFinds exploitable flaws such as hardcoded secrets or command injection
  • Skill Instruction HijackingAlters the agent's session goals or safety constraints when the skill loads
  • Agent Memory PoisoningWrites attacker-controlled rules into memory that affect later sessions
  • Remote Payload Retrieval and ExecutionFetches external code whose behavior can change after review
Findings (2)

T09 · Insecure Skill Coding Practices

Error
Location
eval-viewer/generate_review.py:273
Finding

Stored JavaScript Injection Through Embedded Evaluation Output

Content
View full analysis
// ---- Embedded data (injected by generate_review.py) ---- /*__EMBEDDED_DATA__*/ ``` ### Technical Analysis `generate_review.py` recursively reads evaluation output files and includes their content in the `embedded` object. The resulting object is serialized with `json.dumps()` and inserted directly into an HTML ` ``` the HTML parser treats the first `` as the end of the original script element, even though the sequence appears inside a JavaScript string. The following script element is then parsed and executed. This is a stored injection issue because the payload can be placed in an evaluation output file and later executed automatically when the generated review page is opened. Evaluation outputs may be influenced by untrusted prompts, generated artifacts, tested Skills, or compromised tools. The use of `textContent` when rendering ordinary text later in `viewer.html` does not mitigate this vulnerability because execution occurs while the browser initially parses the generated HTML. ### Attack Path 1. An attacker supplies a test prompt, input artifact, Skill, or generated output that causes a file under an evaluation run's ...[truncated 1774 chars]
Remediation
View remediation
` element. 2. Store the serialized data in a non-executable element and parse its text explicitly: ```html ``` 3. Before inserting JSON into HTML, escape characters significant to the HTML parser: ```python data_json = json.dumps(embedded) data_json = ( data_json .replace("&", "\\u0026") .replace("<", "\\u003c") .replace(">", "\\u003e") .replace("\u2028", "\\u2028") .replace("\u2029", "\\u2029") ) ``` Escaping `<` is essential because it prevents construction of ``. 4. Prefer a well-reviewed HTML templating or serialization utility that explicitly supports safe JSON embedding. 5. Add regression tests using output content containing: - `` - HTML event handlers - U+2028 and U+2029 - nested JSON and multiline source files 6. Add a restrictive Content Security Policy. Avoid inline scripts where possible and use hashes or nonces for trusted scripts. Restrict `connect-src`, `img-src`, `frame-src`, and `script-src` to the minimum required origins. 7. Treat all evaluation outputs as untrusted, even when they were generated locally. ]]>

T05 · Unauthorized Access and Privilege Escalation

Warning
Location
eval-viewer/generate_review.py:287
Finding

Viewer Startup Unconditionally Terminates Unrelated Local Processes

Content
View full analysis
None: """Kill any process listening on the given port.""" try: result = subprocess.run( ["lsof", "-ti", f":{port}"], capture_output=True, text=True, timeout=5, ) for pid_str in result.stdout.strip().split("\n"): if pid_str.strip(): try: os.kill(int(pid_str.strip()), signal.SIGTERM) except (ProcessLookupError, ValueError): pass if result.stdout.strip(): time.sleep(0.5) except subprocess.TimeoutExpired: pass except FileNotFoundError: print("Note: lsof not found, cannot check if port is in use", file=sys.stderr) ``` The function is invoked unconditionally before binding the viewer: ```python # Kill any existing process on the target port port = args.port _kill_port(port) handler = partial( ReviewHandler, workspace, skill_name, feedback_path, previous, benchmark_path, ) ``` ### Technical Analysis The viewer uses `lsof` to obtain every PID listening on the selected port and sends `SIGTERM` to each one. It does not verify that a discovered process is an earlier instance of `generate_review.py`, belongs to the same workspace, or is otherwise safe to terminate. Port occupancy does not establish ownership or authorization. The process may be an unrelated development server, database proxy, notebook, debugging session, or another user application. This behavior exceeds the minimum privileges required by the declared review functionality. The code already contains a safer fallback that binds to port `0` when the requested port remains unav ...[truncated 1850 chars]
Remediation
View remediation
Vulnerability Patterns
  • Data ExfiltrationExternal Transmission, Env Variable Harvesting, File System Enumeration
  • Privilege EscalationExcessive Permissions, Sudo/Root Execution, Credential Access
  • Excessive AgencyUnrestricted Tool Access, Autonomous Decision Making, Scope Creep
  • Rogue AgentSelf-Modification, Session Persistence
  • Trigger AbuseOverly Broad Trigger, Shadow Command Trigger, Keyword Baiting Trigger
Findings (27)

Tp4

High
Category
MCP Tool Poisoning
Confidence
99% confidence
Finding

The declared description is about skill engineering and evaluation workflows, but the supplied code has an entirely different purpose: food recipe matching and menu generation. It does not create or edit skills, run evaluations, benchmark performance, analyze variance, or optimize triggering descriptions. Instead, it processes cooking ingredients and outputs menu recommendations. This is a clear primary-purpose mismatch, with unrelated functionality and triggers.

Content

No source excerpt is available for this finding.

Tp4

High
Category
MCP Tool Poisoning
Confidence
95% confidence
Finding

The declared description focuses on skill authoring and optimization: creating new skills, modifying existing skills, running evals, benchmarking performance, and improving trigger accuracy. This code does not create or modify skills, execute evals, or optimize descriptions. Instead, it implements a review/inspection tool for already-produced eval artifacts. While reviewing eval results and benchmark data is adjacent to measuring skill performance, the primary purpose here is specifically generating and serving an eval viewer and collecting human feedback. Additionally, the port-killing behavior is an undeclared operational capability. Overall, the code's main function is materially narrower and different from the declared skill-authoring and optimization purpose.

Content

No source excerpt is available for this finding.

Tp4

High
Category
MCP Tool Poisoning
Confidence
99% confidence
Finding

The declared description focuses on skill authoring, editing, optimization, evaluation, benchmarking, and trigger-description improvement. The supplied code does none of those things. Its primary purpose is packaging an existing skill directory into a .skill file after checking for existence of SKILL.md and running validation. That is a materially different function from creating/modifying skills or measuring their performance. This is not just an implementation detail of the declared purpose; packaging/distribution is a separate capability that is undeclared, while the declared capabilities are absent from the code chunk.

Content

No source excerpt is available for this finding.

Tp4

High
Category
MCP Tool Poisoning
Confidence
98% confidence
Finding

The code’s actual function is narrow metadata/schema validation for a skill’s SKILL.md file. That may be tangentially related to skill quality checks, but it does not implement the main declared capabilities: creation, editing, optimization, eval execution, benchmarking, or trigger-description optimization. This is a materially different primary purpose, so the description does not accurately represent the supplied code chunk.

Content

No source excerpt is available for this finding.

Ae1

High
Category
analysis-evasion
Confidence
100% confidence
Finding

Referenced artifact was not completely inspected

Content

Scanner excerpt · SKILL.md (reported line 364)May include surrounding context.

md
1. Read the template from `assets/eval_review.html`

Description-Behavior Mismatch

High
Category
Not specified by scanner
Confidence
99% confidence
Finding

The file’s behavior is unrelated to the declared skill purpose: it is a standalone cooking menu generator rather than tooling to create, modify, evaluate, or optimize skills. In an agent skill ecosystem, this kind of capability mismatch is dangerous because it can hide undeclared functionality, defeat review expectations, and cause the agent to invoke an unintended tool path under false metadata.

Content

No source excerpt is available for this finding.

Env Variable Harvesting

High
Category
Data Exfiltration
Confidence
70% confidence
Finding

Code enumerates, copies, or searches environment variables for secrets. Bulk environment access can collect credentials unrelated to the skill's stated purpose.

Content

Scanner excerpt · scripts/improve_description.py (reported line 33)May include surrounding context.

python
# Remove CLAUDECODE env var to allow nesting claude -p inside a
    # Claude Code session. The guard is for interactive terminal conflicts;
    # programmatic subprocess usage is safe. Same pattern as run_eval.py.
    env = {k: v for k, v in os.environ.items() if k != "CLAUDECODE"}

    result = subprocess.run(
        cmd,

Env Variable Harvesting

High
Category
Data Exfiltration
Confidence
70% confidence
Finding

Code enumerates, copies, or searches environment variables for secrets. Bulk environment access can collect credentials unrelated to the skill's stated purpose.

Content

Scanner excerpt · scripts/run_eval.py (reported line 83)May include surrounding context.

python
# Remove CLAUDECODE env var to allow nesting claude -p inside a
    # Claude Code session. The guard is for interactive terminal conflicts;
    # programmatic subprocess usage is safe. Same pattern as run_eval.py.
    env = {k: v for k, v in os.environ.items() if k != "CLAUDECODE"}

    result = subprocess.run(
        cmd,

Agent Config Directory Access

High
Category
Agent Snooping
Confidence
88% confidence
Finding

The code deliberately discovers a parent project root by searching for a .claude directory, then writes a temporary command file into .claude/commands. In a security-sensitive agent context, modifying the agent configuration/command area can influence what the downstream claude -p process sees as available skills, creating a configuration-injection surface and crossing trust boundaries beyond the current skill directory.

Content

Scanner excerpt · scripts/run_eval.py (reported line 23)May include surrounding context.

python
def find_project_root() -> Path:
    """Find the project root by walking up from cwd looking for .claude/.

    Mimics how Claude Code discovers its project root, so the command file
    we create ends up where claude -p will look for it.

Undeclared Tool Scope

Medium
Category
MCP Least Privilege
Confidence
95% confidence
Finding

The skill explicitly instructs the agent to read and write files, invoke Python and shell commands, manage background processes, and inspect environment-dependent capabilities, but it declares no tool or permission scope in its metadata. That increases the blast radius of accidental or unsafe execution because users and orchestrators cannot easily constrain what the skill may do before invocation.

Content

No source excerpt is available for this finding.

Vague Triggers

Medium
Category
Not specified by scanner
Confidence
95% confidence
Finding

The skill tells authors to make descriptions deliberately 'pushy' and to trigger on broad adjacent contexts even when users do not explicitly ask for the capability. In a system with shell, file, and process-manipulation abilities, overbroad triggering can cause the wrong skill to activate and perform invasive actions in contexts where the user only wanted lightweight help.

Content

No source excerpt is available for this finding.

Session Persistence

Medium
Category
Rogue Agent
Confidence
84% confidence
Finding

The skill instructs use of nohup to launch a background review server and later kill it, creating persistence beyond the current interaction. Background processes can continue serving local content, hold file handles, or consume resources after the user thinks the task is finished, especially if cleanup fails or the PID is mishandled.

Content

Scanner excerpt · SKILL.md (reported line 238)May include surrounding context.

  1. Launch the viewer with both qualitative outputs and quantitative data:
    bash
    nohup python <skill-creator-path>/eval-viewer/generate_review.py \
      <workspace>/iteration-N \
      --skill-name "my-skill" \
      --benchmark <workspace>/iteration-N/benchmark.json \
    

Intent-Code Divergence

Medium
Category
Not specified by scanner
Confidence
80% confidence
Finding

The benchmark section explicitly says the analyzer's purpose is to 'not suggest skill improvements,' which conflicts with the earlier post-hoc analyzer instructions centered on identifying loser weaknesses and generating improvement suggestions. This creates contradictory intent within the same documented skill file about whether the analyzer should optimize skills or only surface observations.

Content

No source excerpt is available for this finding.

Vague Triggers

Medium
Category
Not specified by scanner
Confidence
96% confidence
Finding

The trigger phrases include very common everyday expressions such as '今天吃什么', '做什么菜', '菜单', and '晚饭吃啥', which are broad enough to match many ordinary conversations. In an agent environment, this can cause the skill to activate unintentionally, leading to inappropriate context capture, user confusion, or interference with more relevant skills.

Content

No source excerpt is available for this finding.

Natural-Language Policy Violations

Medium
Category
Not specified by scanner
Confidence
95% confidence
Finding

This code file contains natural-language instructions, examples, and command help entirely in Chinese, presenting the skill as Chinese-only. Under the policy, forcing a specific language without user opt-in is a locale/language policy violation unless the restriction is clearly documented and justified, which is not present here.

Content

No source excerpt is available for this finding.

Context-Inappropriate Capability

Medium
Category
Not specified by scanner
Confidence
97% confidence
Finding

The script automatically identifies any process bound to the requested port and sends SIGTERM without verifying ownership, purpose, or obtaining user confirmation. In a developer workstation context, this can disrupt unrelated local services or terminate security-sensitive tooling, creating a local denial-of-service and unsafe operational behavior.

Content

No source excerpt is available for this finding.

Missing User Warnings

Medium
Category
Not specified by scanner
Confidence
98% confidence
Finding

Forcefully terminating whatever is listening on the chosen port is dangerous because the script makes a destructive system change unrelated to its core task of serving a review page. In the skill context, where users may run generated tooling locally, this behavior is more concerning because it can silently interfere with other development services and cause avoidable service interruption.

Content

No source excerpt is available for this finding.

subprocess module call

Medium
Category
Dangerous Code Execution
Confidence
70% confidence
Finding

subprocess module calls execute external commands. Without careful input validation, this enables command injection.

Content

Scanner excerpt · eval-viewer/generate_review.py (reported line 291)May include surrounding context.

python
def _kill_port(port: int) -> None:
    """Kill any process listening on the given port."""
    try:
        result = subprocess.run(
            ["lsof", "-ti", f":{port}"],
            capture_output=True, text=True, timeout=5,
        )

subprocess module call

Medium
Category
Dangerous Code Execution
Confidence
70% confidence
Finding

subprocess module calls execute external commands. Without careful input validation, this enables command injection.

Content

Scanner excerpt · scripts/improve_description.py (reported line 35)May include surrounding context.

python
# programmatic subprocess usage is safe. Same pattern as run_eval.py.
    env = {k: v for k, v in os.environ.items() if k != "CLAUDECODE"}

    result = subprocess.run(
        cmd,
        input=prompt,
        capture_output=True,

Missing User Warnings

Medium
Category
Not specified by scanner
Confidence
89% confidence
Finding

The script constructs a prompt containing the full skill content, eval queries/results, and prior attempt history, then sends it to an external model subprocess without any explicit consent gate, warning, or redaction step at the transmission point. If those inputs contain secrets, proprietary prompts, customer data, or sensitive test cases, the tool can exfiltrate them to a separate model runtime contrary to user expectations.

Content

No source excerpt is available for this finding.

subprocess module call

Medium
Category
Dangerous Code Execution
Confidence
70% confidence
Finding

subprocess module calls execute external commands. Without careful input validation, this enables command injection.

Content

Scanner excerpt · scripts/run_eval.py (reported line 85)May include surrounding context.

python
# programmatic subprocess usage is safe.
        env = {k: v for k, v in os.environ.items() if k != "CLAUDECODE"}

        process = subprocess.Popen(
            cmd,
            stdout=subprocess.PIPE,
            stderr=subprocess.DEVNULL,

Missing User Warnings

Low
Category
Not specified by scanner
Confidence
78% confidence
Finding

The markdown instructs the agent to save structured analysis to {output_path}, which is a file write operation affecting user data or the local filesystem. The description does not include any warning, confirmation, or explicit disclosure that the skill will write output files.

Content

No source excerpt is available for this finding.

Missing User Warnings

Low
Category
Not specified by scanner
Confidence
76% confidence
Finding

This section directs the agent to save notes to {output_path} as JSON, which again performs a file write. The markdown does not warn the user that running the skill will create or modify a file at the provided path.

Content

No source excerpt is available for this finding.

Missing User Warnings

Low
Category
Not specified by scanner
Confidence
82% confidence
Finding

This markdown skill instructs the agent to save results to {outputs_dir}/../grading.json, which is a file write that affects the user's workspace. The document does not include any explicit warning or disclosure that it will create or overwrite this file, so the write lacks user-facing notice under the markdown-file criteria.

Content

No source excerpt is available for this finding.

Natural-Language Policy Violations

Low
Category
Not specified by scanner
Confidence
93% confidence
Finding

This JSON eval file consistently defines prompts and expected outputs only in Chinese, which implies a language-specific behavior for the skill. Under the policy, locale or language constraints should be user-selectable or clearly justified; no such opt-in or justification appears in the file.

Content

No source excerpt is available for this finding.

Static analysis

No suspicious patterns detected.