Back to skill

Security audit

Skill Test Generate

Security checks for vulnerabilities and agentic risk

Overview

This skill is a disclosed local test-case generator, but users should avoid feeding it untrusted ZIP files.

Use this skill only on skill directories or ZIP files you trust, and choose an output JSON path you are comfortable letting it modify. Its normal operation requires the agent to read every non-hidden file in the target skill so it can produce line-referenced tests.

Vulnerability Patterns
  • Insecure Skill Coding PracticesFinds exploitable flaws such as hardcoded secrets or command injection
  • Skill Instruction HijackingAlters the agent's session goals or safety constraints when the skill loads
  • Agent Memory PoisoningWrites attacker-controlled rules into memory that affect later sessions
  • Remote Payload Retrieval and ExecutionFetches external code whose behavior can change after review
  • Embedded Malicious CodeShips malicious scripts inside the skill and executes them locally
Findings (1)

T09 · Insecure Skill Coding Practices

Warning
Location
scripts/generate.py:143
Finding

Unbounded ZIP Archive Extraction Enables Resource Exhaustion

Content
View full analysis

Vulnerability Details

File Location: scripts/generate.py, lines 143–150
Vulnerability Type: Unbounded archive extraction
Risk Level: Medium

Vulnerable Code

python
temp_dir = tempfile.mkdtemp(prefix="skill-test-gen-")
try:
    with zipfile.ZipFile(str(target_path), "r") as zf:
        zf.extractall(temp_dir)

Technical Analysis

The prepare workflow accepts a user-specified ZIP archive and extracts every member with ZipFile.extractall() without first enforcing limits on:

  • The number of archive members
  • Individual uncompressed file size
  • Aggregate uncompressed size
  • Compression ratio
  • Available disk space or extraction time

A malicious archive can contain highly compressible data that occupies little space in the ZIP but expands to a very large size. Because extraction happens before the code searches for SKILL.md, merely processing the archive is sufficient to trigger resource consumption.

Attack Path

  1. An attacker creates a ZIP bomb or another archive containing an excessive number or total size of compressed files.
  2. The archive is supplied as the target Skill, either directly or through an untrusted source.
  3. The user or Agent invokes:
    bash
    python scripts/generate.py prepare --skill /path/to/untrusted-skill.zip
    
  4. resolve_skill_path() creates a temporary directory and calls extractall() without resource limits.
  5. The archive expands until disk space, I/O capacity, or processing time is exhausted.
  6. The Skill process and other applications sharing the filesystem may fail or become unavailable.

Impact Assessment

Successful exploitation does not grant additional system privileges or arbitrary code execution based on the reviewed code. Its primary impact is local denial of service within the permissions of the account running the Skill.

An attacker may:

  • Exhaust temporary or root filesystem capacity
  • Degrade CPU and disk I/O availability
  • Cause the preparation workflow to ha ...[truncated 317 chars]
Remediation
View remediation

Remediation Suggestions

Replace unrestricted extractall() usage with a validated, member-by-member extraction routine:

  1. Inspect all ZipInfo entries before writing any files.
  2. Enforce conservative limits on:
    • Total member count
    • Maximum uncompressed size per member
    • Maximum aggregate uncompressed size
    • Maximum compression ratio
  3. Reject encrypted archives and unsupported or abnormal entry types.
  4. Confirm sufficient free disk space before extraction.
  5. Extract each approved member individually while tracking bytes written, aborting immediately if a limit is exceeded.
  6. Ensure temporary directories are deleted on every error path and after the caller no longer needs them.
  7. Consider applying execution-time and filesystem quotas when processing untrusted archives.

Example defensive preflight logic:

python
MAX_FILES = 1000
MAX_FILE_SIZE = 50 * 1024 * 1024
MAX_TOTAL_SIZE = 250 * 1024 * 1024
MAX_COMPRESSION_RATIO = 100

with zipfile.ZipFile(str(target_path), "r") as zf:
    members = zf.infolist()

    if len(members) > MAX_FILES:
        raise ValueError("Archive contains too many files")

    total_size = 0
    for member in members:
        if member.file_size > MAX_FILE_SIZE:
            raise ValueError("Archive member exceeds the size limit")

        total_size += member.file_size
        if total_size > MAX_TOTAL_SIZE:
            raise ValueError("Archive exceeds the total extraction limit")

        compressed = max(member.compress_size, 1)
        if member.file_size / compressed > MAX_COMPRESSION_RATIO:
            raise ValueError("Suspicious archive compression ratio")

    for member in members:
        zf.extract(member, temp_dir)

The extraction routine should also retain or add canonical-path containment checks as defense in depth.

Vulnerability Patterns
  • Prompt InjectionInstruction Override, Hidden Instructions, Exfiltration Commands
  • Excessive AgencyUnrestricted Tool Access, Autonomous Decision Making, Scope Creep
  • Taint TrackingDirect Taint Flow, Variable-Mediated Taint Flow, Credential Exfiltration Chain
  • MCP Least PrivilegeUnderdeclared Capability, Wildcard Permission, Missing Permission Declaration
  • MCP Tool PoisoningHidden Instructions, Unicode Deception, Parameter Description Injection
Findings (11)

Tp4

High
Category
MCP Tool Poisoning
Confidence
97% confidence
Finding

The declared purpose says the skill generates test cases, but the documented workflow also resolves local paths, enumerates files, reads all files in a target skill, decompresses zip archives, validates JSON, and rewrites files by adding fields such as summary and verified. This description-behavior gap is security-relevant because it hides filesystem and archive-handling side effects from users and calling agents, undermining informed consent and increasing the chance of unsafe invocation on sensitive local content.

Content

No source excerpt is available for this finding.

Ae1

High
Category
analysis-evasion
Confidence
100% confidence
Finding

Referenced artifact was not completely inspected

Content

Scanner excerpt · SKILL.md (reported line 53)May include surrounding context.

md
{ "file": "SKILL.md", "lines": [1, 10] }

Ae1

High
Category
analysis-evasion
Confidence
100% confidence
Finding

Referenced artifact was not completely inspected

Content

Scanner excerpt · SKILL.md (reported line 216)May include surrounding context.

md
{ "file": "SKILL.md", "lines": [1, 10] }

Undeclared Tool Scope

Medium
Category
MCP Least Privilege
Confidence
94% confidence
Finding

The skill instructs the agent to read arbitrary skill files, invoke local helper scripts, inspect installed skill directories, and write JSON output, yet it declares no explicit tool scope or permissions boundaries. This creates a confused-deputy risk where an agent may access local files or modify output files without clear least-privilege constraints, especially when the target skill path is user-controlled.

Content

No source excerpt is available for this finding.

Natural-Language Policy Violations

Medium
Category
Not specified by scanner
Confidence
94% confidence
Finding

The manifest description states the skill outputs structured JSON for test generation in Chinese, and the entire instruction set is written as if Chinese is the required operating language. There is no indication that users may choose another language or that the locale restriction is justified as region-specific, which creates a language policy concern.

Content

No source excerpt is available for this finding.

Natural-Language Policy Violations

Medium
Category
Not specified by scanner
Confidence
95% confidence
Finding

The schema requires each capability entry to be '一个简短中文描述', which imposes a specific language on skill content. This is a natural-language policy concern because the file does not offer user opt-in, multilingual alternatives, or a documented region-specific reason for requiring Chinese.

Content

No source excerpt is available for this finding.

Description-Behavior Mismatch

Medium
Category
Not specified by scanner
Confidence
98% confidence
Finding

The manifest describes the skill as automatically generating structured SFT-level functional test cases for arbitrary agent skills. However, the module docstring explicitly states that it does not perform semantic test generation and instead only provides helper subcommands for resolving skill paths, listing files, extracting metadata, and validating output JSON.

Content

No source excerpt is available for this finding.

Intent-Code Divergence

Medium
Category
Not specified by scanner
Confidence
95% confidence
Finding

The inline module documentation says 'This script does NOT perform semantic test generation,' which directly conflicts with the broader skill intent presented in the manifest as an automatic generator of SFT test cases. This is an active contradiction between documented intent and implemented/helper behavior, not merely missing detail.

Content

No source excerpt is available for this finding.

Natural-Language Policy Violations

Medium
Category
Not specified by scanner
Confidence
96% confidence
Finding

This code emits multiple user-facing error messages in Chinese, including for missing installed skills, invalid zip files, and invalid paths. Because the script does not offer any language/locale selection or document that it is intended only for a Chinese-speaking environment, it violates the language/locale policy for natural-language content.

Content

No source excerpt is available for this finding.

Natural-Language Policy Violations

Medium
Category
Not specified by scanner
Confidence
98% confidence
Finding

The validation output, success messages, parser description, subcommand help, and argument help are all presented only in Chinese. This is a natural-language policy issue because the file enforces a specific language for user interaction without exposing a setting or clearly justifying a locale restriction.

Content

No source excerpt is available for this finding.

Tainted flow: 'data' from pathlib.Path.read_text (line 475, file read) → pathlib.Path.write_text (file write)

Medium
Category
Data Flow
Confidence
65% confidence
Finding

Data from a source is assigned to a variable that is later passed to a sink, creating a variable-mediated taint flow.

Content

Scanner excerpt · scripts/generate.py (reported line 500)May include surrounding context.

python
data["verified"] = True

    # Write back
    file_path.write_text(
        json.dumps(data, ensure_ascii=False, indent=2) + "\n",
        encoding="utf-8",
    )