Skill Eval Preflight

• Outcome: Review
• Latest audit: Sep 12, 2026 · 12:10 AM UTC
• Version: 1.0.0

Overview

This skill is mostly a coherent eval-preflight helper, but its bundled scripts can write comparison or run artifacts outside the intended skill folder if given a crafted run-group path.

Install only if you trust the skill author and use simple run-group names such as letters, digits, underscores, dots, or hyphens. Avoid passing user-supplied or path-like run-group values until the scripts validate and contain output paths under evals/runs.

A.I.G

Vulnerability Patterns

• Insecure Skill Coding Practices: Finds exploitable flaws such as hardcoded secrets or command injection
• Skill Instruction Hijacking: Alters the agent's session goals or safety constraints when the skill loads
• Agent Memory Poisoning: Writes attacker-controlled rules into memory that affect later sessions
• Remote Payload Retrieval and Execution: Fetches external code whose behavior can change after review
• Embedded Malicious Code: Ships malicious scripts inside the skill and executes them locally

Findings (2)

T09 · Insecure Skill Coding Practices

Warning

• Location: scripts/run_eval.py:34

Finding

Path Traversal Enables Artifact Creation Outside the Intended Run Directory

Content

View full analysis

Vulnerability Details

File Location: scripts/run_eval.py:34-38
Vulnerability Type: Unsanitized path construction and path traversal
Risk Level: Medium

Vulnerable Code

python

def resolve_run_dirs(skill_dir: Path, run_group: str | None, mode: str) -> tuple[str, Path]:
    resolved_group = run_group or run_stamp()
    run_root = skill_dir / "evals" / "runs" / resolved_group
    run_dir = run_root / mode
    run_dir.mkdir(parents=True, exist_ok=False)
    return resolved_group, run_dir

The resulting uncontained directory is subsequently used for writes at scripts/run_eval.py:342-344:

python

(run_dir / "summary.json").write_text(json.dumps(summary, ensure_ascii=False, indent=2) + "\n")
(run_dir / "summary.md").write_text(make_summary_md(skill_dir.name, summary))
(run_dir / "run_metadata.json").write_text(json.dumps(run_metadata, ensure_ascii=False, indent=2) + "\n")

Technical Analysis

The user-controlled --run-group argument is appended directly to the nominal evals/runs root. The implementation does not reject absolute paths, parent-directory components such as .., or path separators. It also does not resolve the resulting path and verify that it remains beneath the authorized run root.

In Python's pathlib, parent-directory components can escape the intended root, while joining an absolute path can discard the preceding path components entirely. Consequently, run_dir.mkdir() and the subsequent write_text() operations can act outside the target Skill directory.

The mode component is constrained by argparse choices, but this does not mitigate traversal through resolved_group.

Attack Path

1. The attacker can invoke run_eval.py and control --run-group.
2. The attacker supplies an absolute path or a traversal sequence, for example:

   bash

   python3 scripts/run_eval.py /path/to/skill \
     --run-group ../../../../attacker-selected-dire
   

...[truncated 1312 chars]

Remediation

View remediation

Remediation Suggestions

1. Restrict run-group names to simple identifiers:

   python

   import re
   
   RUN_GROUP_PATTERN = re.compile(r"^[A-Za-z0-9][A-Za-z0-9._-]{0,127}$")
   
   def validate_run_group(value: str) -> str:
       if value in {".", ".."} or not RUN_GROUP_PATTERN.fullmatch(value):
           raise SystemExit(
               "Error: run-group may contain only letters, digits, dots, "
               "underscores, and hyphens."
           )
       return value
   

2. Resolve and enforce containment before creating anything:

   python

   runs_root = (skill_dir / "evals" / "runs").resolve()
   resolved_group = validate_run_group(run_group or run_stamp())
   run_root = (runs_root / resolved_group).resolve()
   
   if not run_root.is_relative_to(runs_root):
       raise SystemExit("Error: run-group escapes the evals/runs directory.")
   

3. Perform the same containment check on run_dir before calling mkdir() or writing files.

4. If target directories may be writable by untrusted users, reject symlinked path components or open files through directory-relative, no-follow mechanisms to reduce symlink race risks.

5. Add regression tests covering absolute paths, ../ traversal, nested separators, ., .., and symlink-based escape attempts.

T09 · Insecure Skill Coding Practices

Warning

• Location: scripts/compare_runs.py:158

Finding

Path Traversal Enables External Summary Reads and Comparison File Overwrites

Content

View full analysis

Vulnerability Details

File Location: scripts/compare_runs.py:158-188
Vulnerability Type: Unsanitized path construction allowing traversal-based reads and writes
Risk Level: Medium

Vulnerable Code

python

skill_dir = Path(args.skill_dir).expanduser().resolve()
ensure_skill_dir(skill_dir)

run_group_dir = skill_dir / "evals" / "runs" / args.run_group
if not run_group_dir.is_dir():
    raise SystemExit(
        f"Error: run group not found: {run_group_dir}\n"
        "- next step: run run_eval.py first, then retry compare_runs.py"
    )

left_summary_path = run_group_dir / args.left_mode / "summary.json"
right_summary_path = run_group_dir / args.right_mode / "summary.json"
if not left_summary_path.is_file() or not right_summary_path.is_file():
    raise SystemExit(
        missing_summary_message(
            run_group_dir,
            args.left_mode,
            args.right_mode,
            left_summary_path,
            right_summary_path,
        )
    )

left_summary = load_json(left_summary_path)
right_summary = load_json(right_summary_path)
comparison = compare_summaries(left_summary, right_summary, args.left_mode, args.right_mode)

comparison_json_path = run_group_dir / "comparison.json"
comparison_md_path = run_group_dir / "comparison.md"
comparison_json_path.write_text(json.dumps(comparison, ensure_ascii=False, indent=2) + "\n")
comparison_md_path.write_text(make_md(skill_dir.name, args.run_group, comparison))

Technical Analysis

The required --run-group value is incorporated directly into run_group_dir without validation or containment enforcement. An absolute path can replace the intended base, and .. components can traverse out of evals/runs.

If the attacker-selected directory contains the expected mode subdirectories and valid summary.json files, the script reads those external files. It then writes comparison.json and comparison.md ...[truncated 1945 chars]

Remediation

View remediation

Remediation Suggestions

1. Apply the same strict run-group identifier validation used by run_eval.py.

2. Resolve the trusted root and candidate directory, then enforce containment before any existence check, directory iteration, read, or write:

   python

   runs_root = (skill_dir / "evals" / "runs").resolve()
   run_group = validate_run_group(args.run_group)
   run_group_dir = (runs_root / run_group).resolve()
   
   if not run_group_dir.is_relative_to(runs_root):
       raise SystemExit("Error: run-group escapes the evals/runs directory.")
   

3. Resolve each summary path and verify it remains inside run_group_dir before reading it.

4. Reject unexpected symlinks for the run-group directory, mode directories, summaries, and output files when the workspace is not fully trusted.

5. Use atomic output replacement through temporary files inside the validated directory. If overwriting existing comparisons is not required, use exclusive file creation instead.

6. Add tests proving that absolute paths, parent traversal, embedded separators, and symlink escapes are rejected before any file is read or modified.

SkillSpector

Vulnerability Patterns

• MCP Least Privilege: Underdeclared Capability, Wildcard Permission, Missing Permission Declaration
• Prompt Injection: Instruction Override, Hidden Instructions, Exfiltration Commands
• Data Exfiltration: External Transmission, Env Variable Harvesting, File System Enumeration
• Privilege Escalation: Excessive Permissions, Sudo/Root Execution, Credential Access
• Supply Chain: Unpinned Dependencies, External Script Fetching, Obfuscated Code

Findings (3)

Ae1

High

• Category: analysis-evasion
• Confidence: 100% confidence

Finding

Referenced artifact was not completely inspected

Content

Scanner excerpt · SKILL.md (reported line 56)May include surrounding context.

md

1. Confirm the target folder is a skill directory with `SKILL.md`.

Ae1

High

• Category: analysis-evasion
• Confidence: 100% confidence

Finding

Referenced artifact was not completely inspected

Content

Scanner excerpt · SKILL.md (reported line 98)May include surrounding context.

md

1. Confirm the target folder is a skill directory with `SKILL.md`.

Undeclared Tool Scope

Medium

• Category: MCP Least Privilege
• Confidence: 91% confidence

Finding

The skill instructs the agent to run local scripts against arbitrary skill paths and references capabilities consistent with file read, file write, and potentially network-adjacent behavior, but it declares no explicit tool scope such as permissions or allowed-tools. That mismatch is dangerous because consumers and enforcement layers cannot clearly constrain what the skill may access, increasing the risk of overbroad file operations or unintended external access when evaluating untrusted skills.

Content

No source excerpt is available for this finding.

Static analysis

No suspicious patterns detected.