Back to skill

Security audit

skill-evaluation

Security checks for vulnerabilities and agentic risk

Overview

This is a legitimate skill evaluator, but its live trigger tests can expose a user's real project and environment credentials to external agent CLIs, so it needs review before installation.

Install only if you are comfortable with a write-heavy evaluation tool. Do not run live trigger evaluation from a sensitive repository or with production credentials in the environment; use a disposable workspace, test accounts, and a scrubbed environment, and review any generated optimized skill before using it.

Vulnerability Patterns
  • Insecure Skill Coding PracticesFinds exploitable flaws such as hardcoded secrets or command injection
  • Skill Instruction HijackingAlters the agent's session goals or safety constraints when the skill loads
  • Agent Memory PoisoningWrites attacker-controlled rules into memory that affect later sessions
  • Remote Payload Retrieval and ExecutionFetches external code whose behavior can change after review
  • Embedded Malicious CodeShips malicious scripts inside the skill and executes them locally
Findings (1)

T09 · Insecure Skill Coding Practices

Error
Location
scripts/run_trigger_eval.py:163
Finding

Trigger Evaluation Exposes the Host Environment and Project to External Agent CLIs

Content
View full analysis

Vulnerability Details

File Location: scripts/run_trigger_eval.py, lines 163-168
Vulnerability Type: Excessive environment and filesystem privileges in child processes
Risk Level: High

Vulnerable Code

python
env = {k: v for k, v in os.environ.items() if k != "CLAUDECODE"}

process = subprocess.Popen(
    cmd, stdout=subprocess.PIPE, stderr=subprocess.DEVNULL,
    cwd=str(project_root), env=env,
)

The other live adapters also launch agent CLIs from the detected project root while implicitly inheriting the complete parent environment:

python
result = subprocess.run(
    cmd, capture_output=True, text=True,
    timeout=timeout, cwd=str(project_root),
)

This pattern occurs in the Codex adapter at lines 363-367 and the OpenClaw adapter at lines 412-415.

Technical Analysis

The Claude adapter forwards every host environment variable except CLAUDECODE. The inherited values may include API keys, cloud credentials, access tokens, proxy credentials, and other secrets. The Codex and OpenClaw adapters do not provide an explicit environment, so they inherit the entire parent environment by default.

These agent CLIs also execute with cwd set to the detected real project root. Consequently, they may access project files, platform-specific instructions, and other data available under the invoking user's permissions. The trigger evaluator does not enforce a disposable workspace, restricted tool set, clean environment, network isolation, or approval mode.

This exceeds least privilege for trigger testing, which only needs to determine whether the temporary probe is activated. It is also inconsistent with the stronger safety boundary in agents/executor.md, which requires disposable workspaces, mocked dependencies, approval for mutations, and restrictions on high-impact tools when testing untrusted skills.

The subprocess calls use argument arrays and do not use shell=True; therefor ...[truncated 1963 chars]

Remediation
View remediation

Remediation Suggestions

  1. Run every trigger probe in a newly created disposable directory containing only the generated probe and required minimal platform configuration.
  2. Replace environment inheritance with an explicit allowlist. Pass only variables essential for process startup and the selected provider, such as a controlled PATH, a temporary HOME, locale settings, and at most the single required authentication credential.
  3. Use a temporary home/configuration directory so platform CLIs cannot load unrelated user-level rules, plugins, credentials, or persistent state.
  4. Disable network access, mutating tools, shell execution, and unrestricted file access wherever the target platform supports such controls.
  5. Require explicit user approval before running a live probe when the selected adapter cannot guarantee isolation.
  6. Apply the mandatory safety boundary from agents/executor.md to Phase 1.5 trigger evaluation as well.
  7. Do not automatically execute from an ancestor project directory. Require an explicit project root, validate it, and copy only required files into the disposable workspace.
  8. Document which credentials and external services each adapter requires before execution.
  9. Capture and report attempted tool calls and side effects rather than recording only whether the probe activated.
  10. Add regression tests verifying that unrelated environment variables and host-project files are unavailable to probe subprocesses.
Vulnerability Patterns
  • Prompt InjectionInstruction Override, Hidden Instructions, Exfiltration Commands
  • Data ExfiltrationExternal Transmission, Env Variable Harvesting, File System Enumeration
  • Privilege EscalationExcessive Permissions, Sudo/Root Execution, Credential Access
  • Excessive AgencyUnrestricted Tool Access, Autonomous Decision Making, Scope Creep
  • System Prompt LeakageDirect Leakage, Indirect Extraction, Tool-Based Exfiltration
Findings (21)

Tp4

High
Category
MCP Tool Poisoning
Confidence
98% confidence
Finding

The declared description promises a broad end-to-end skill evaluation framework covering trigger accuracy, execution quality per step, efficiency, safety, structured reporting, bad-case analysis, actionable fixes, and iterative optimization/version support. The supplied code chunk is much narrower: it is explicitly a 'standalone trigger evaluator' focused on testing whether a skill description causes activation for probe queries across several platforms. It writes temporary probe files, invokes platform CLIs or heuristics, and computes trigger precision/recall/accuracy. While trigger accuracy is one subset of the declared purpose, the code does not implement the majority of the promised evaluation dimensions or optimization/reporting features. Therefore the description materially overstates the code's actual behavior.

Content

No source excerpt is available for this finding.

Tp4

High
Category
MCP Tool Poisoning
Confidence
99% confidence
Finding

The code’s primary purpose is clearly a static security/safety scan of skill files, especially detecting prompt injection, dangerous commands, secrets, network calls, and file access patterns. That is only one narrow slice of the declared functionality ('safety') and does not implement the broader promised capabilities such as step-by-step diagnosis of skill behavior, trigger accuracy measurement, execution correctness/quality assessment, efficiency benchmarking, bad-case analysis, actionable optimization workflows, or version comparisons. This is therefore a material description-behavior mismatch, not merely an implementation detail difference.

Content

No source excerpt is available for this finding.

Anti-Refusal Statement

High
Category
Anti-Refusal
Confidence
85% confidence
Finding

Skill instructs the agent to omit warnings, disclaimers, or ethical commentary. Stripping safety caveats hides risk from the user and is a common jailbreak preamble.

Content

Scanner excerpt · agents/executor.md (reported line 152)May include surrounding context.

md
## Principles

- **Record, don't judge.** Write what happened, not whether it's good or bad.
  Scoring is Phase 4's job.

- **Be complete.** Capture the full actual_output even if it's long. Truncation

Hidden Instructions

High
Category
Prompt Injection
Confidence
70% confidence
Finding

Hidden instructions were detected in comments or invisible text. These could contain malicious directives. Manual review is recommended.

Content

Scanner excerpt · agents/reporter.md (reported line 176)May include surrounding context.

  1. Append optimization metadata as an HTML comment at the end:
markdown
<!-- EVALUATION OPTIMIZATION METADATA
optimization_id: OPT-{N}
from_version: v{N}
to_version: v{N+1}

Hidden Instructions

High
Category
Prompt Injection
Confidence
70% confidence
Finding

Hidden instructions were detected in comments or invisible text. These could contain malicious directives. Manual review is recommended.

Content

Scanner excerpt · references/schemas.md (reported line 374)May include surrounding context.

md
... (fixed skill content) ...

<!-- EVALUATION OPTIMIZATION METADATA
optimization_id: OPT-{N}
from_version: v1
to_version: v2

Env Variable Harvesting

High
Category
Data Exfiltration
Confidence
70% confidence
Finding

Code enumerates, copies, or searches environment variables for secrets. Bulk environment access can collect credentials unrelated to the skill's stated purpose.

Content

Scanner excerpt · scripts/run_trigger_eval.py (reported line 163)May include surrounding context.

python
if model:
            cmd.extend(["--model", model])

        env = {k: v for k, v in os.environ.items() if k != "CLAUDECODE"}

        process = subprocess.Popen(
            cmd, stdout=subprocess.PIPE, stderr=subprocess.DEVNULL,

Direct Prompt Extraction

High
Category
System Prompt Leakage
Confidence
85% confidence
Finding

Skill contains instructions that could directly expose system prompts, internal rules, or hidden instructions to users or external parties.

Content

Scanner excerpt · scripts/run_trigger_eval.py (reported line 283)May include surrounding context.

python
def get_skill_dir(self, project_root: Path) -> Path:
        rules_dir = project_root / ".cursor" / "rules"
        rules_dir.mkdir(parents=True, exist_ok=True)
        return rules_dir

    def write_probe_file(self, skill_dir: Path, probe_name: str,
                         skill_name: str, description: str) -> Path:

Description-Behavior Mismatch

High
Category
Not specified by scanner
Confidence
97% confidence
Finding

The manifest describes a comprehensive evaluator that measures trigger accuracy, step execution quality, efficiency, safety, structured reporting, bad-case analysis, and iterative optimization support. This file implements only a regex-based static safety scan over SKILL.md and bundled files, producing safety findings and a score, with no logic for broader quality evaluation dimensions.

Content

No source excerpt is available for this finding.

Credential Access

High
Category
Privilege Escalation
Confidence
60% confidence
Finding

Code accesses credential files (SSH keys, AWS credentials, etc.). This could indicate credential theft attempts.

Content

Scanner excerpt · scripts/safety_scanner.py (reported line 44)May include surrounding context.

python
"File write operation (verify output scope is intentional)", "file_write"),
    (r'\bprint\s*\(.*(?:error|exception|traceback)', "MEDIUM",
     "Printing error details that may leak internal info", "info_leak"),
    (r'\.env\b', "MEDIUM", "Reference to .env file (may contain secrets)", "credential_exposure"),

    # Low severity
    (r'#\s*TODO|#\s*FIXME|#\s*HACK', "LOW", "Unresolved TODO/FIXME in skill code", "code_quality"),

Undeclared Tool Scope

Medium
Category
MCP Least Privilege
Confidence
87% confidence
Finding

The skill instructs broad file creation/modification and references scripts, network-like trigger testing, shell/CLI use, and environment interaction, but it does not declare any explicit tool restrictions or least-privilege scope. In a skill that evaluates arbitrary third-party skills, missing tool boundaries increases the chance of unintended filesystem changes, unsafe execution of helper scripts, or overbroad access when handling untrusted inputs.

Content

No source excerpt is available for this finding.

Vague Triggers

Medium
Category
Not specified by scanner
Confidence
95% confidence
Finding

The invocation guidance is very broad and may activate the skill for generic requests about evaluation, benchmarking, or prompt quality rather than narrowly scoped skill-assessment tasks. Over-triggering a write-capable skill can cause unnecessary file generation, unintended workflow changes, or evaluation of the wrong target in contexts involving untrusted or sensitive repositories.

Content

No source excerpt is available for this finding.

Missing User Warnings

Medium
Category
Not specified by scanner
Confidence
91% confidence
Finding

The skill mandates creation of multiple versioned files and may generate an optimized replacement skill, but it does not prominently require explicit user consent before making these writes. In a repository containing untrusted or production prompts, automatic artifact generation and modification can create integrity risks, noisy commits, or accidental propagation of unsafe changes.

Content

No source excerpt is available for this finding.

Autonomous Decision Making

Medium
Category
Excessive Agency
Confidence
75% confidence
Finding

Skill enables autonomous high-impact decisions without human-in-the-loop verification. Critical operations (destructive commands, financial transactions, data deletion) should require explicit user confirmation.

Content

Scanner excerpt · references/rubrics.md (reported line 180)May include surrounding context.

(ref: SKILL.md Step 3 'use querySelector'). Actual used regex matching."

"completion=0: Step 4 'validate input' was not executed. Model skipped directly to submission without checking required fields."

text

subprocess module call

Medium
Category
Dangerous Code Execution
Confidence
70% confidence
Finding

subprocess module calls execute external commands. Without careful input validation, this enables command injection.

Content

Scanner excerpt · scripts/run_trigger_eval.py (reported line 165)May include surrounding context.

python
env = {k: v for k, v in os.environ.items() if k != "CLAUDECODE"}

        process = subprocess.Popen(
            cmd, stdout=subprocess.PIPE, stderr=subprocess.DEVNULL,
            cwd=str(project_root), env=env,
        )

subprocess module call

Medium
Category
Dangerous Code Execution
Confidence
70% confidence
Finding

subprocess module calls execute external commands. Without careful input validation, this enables command injection.

Content

Scanner excerpt · scripts/run_trigger_eval.py (reported line 363)May include surrounding context.

python
cmd.extend(["--model", model])

        try:
            result = subprocess.run(
                cmd, capture_output=True, text=True,
                timeout=timeout, cwd=str(project_root),
            )

subprocess module call

Medium
Category
Dangerous Code Execution
Confidence
70% confidence
Finding

subprocess module calls execute external commands. Without careful input validation, this enables command injection.

Content

Scanner excerpt · scripts/run_trigger_eval.py (reported line 412)May include surrounding context.

python
cmd.extend(["--model", model])

        try:
            result = subprocess.run(
                cmd, capture_output=True, text=True,
                timeout=timeout, cwd=str(project_root),
            )

Unsafe Defaults

Medium
Category
Tool Misuse
Confidence
70% confidence
Finding

Tool defaults are unsafe or overly permissive (e.g. disabled TLS verification, no authentication, world-writable permissions). Unsafe defaults widen the attack surface.

Content

Scanner excerpt · scripts/safety_scanner.py (reported line 29)May include surrounding context.

python
"Shell subprocess with shell=True", "code_injection"),

    # High severity
    (r'\bchmod\s+777\b', "HIGH", "Setting world-writable permissions", "permission_risk"),
    (r'\bsudo\b', "HIGH", "Requesting elevated privileges", "privilege_escalation"),
    (r'(\/etc\/passwd|\/etc\/shadow)', "HIGH", "Accessing system credential files", "system_access"),
    (r'\bos\.environ\b', "HIGH", "Accessing environment variables (potential secret exposure)", "info_leak"),

Missing User Warnings

Medium
Category
Not specified by scanner
Confidence
89% confidence
Finding

This code performs a file write via Path(args.output).write_text(output_json). While the write is conditional on the --output argument, there is no confirmation prompt, user-facing warning near the write, or comment/docstring specifically disclosing that the tool will overwrite/create the target file.

Content

No source excerpt is available for this finding.

Missing User Warnings

Low
Category
Not specified by scanner
Confidence
88% confidence
Finding

The script writes evaluation results to an arbitrary path supplied by --output, but there is no confirmation prompt or explicit user-facing warning at the point of the file write. Although the CLI argument name implies saving output, the code path performing the write does not disclose overwrite or filesystem impact behavior.

Content

No source excerpt is available for this finding.

Intent-Code Divergence

Low
Category
Not specified by scanner
Confidence
84% confidence
Finding

The module docstring states it scans SKILL.md and bundled scripts for dangerous patterns. However, the implementation also scans references/ markdown files for instruction patterns, which is an additional scope not reflected in the documentation.

Content

No source excerpt is available for this finding.

Missing User Warnings

Low
Category
Not specified by scanner
Confidence
88% confidence
Finding

This Python file performs a file write via Path(args.output).write_text(output_json). Although writing an output report is part of the tool's behavior, there is no explicit user disclosure at the point of execution beyond the CLI flag name, no confirmation prompt, and no warning about overwriting or modifying files.

Content

No source excerpt is available for this finding.

Static analysis

No suspicious patterns detected.