Security audit

PluginEval Core

Security checks for vulnerabilities and agentic risk

Overview

The skill is mostly a local quality evaluator, but its optional write mode can modify or delete skill files and has unsafe path handling, so it needs review before installation.

Use the read-only evaluation modes by default. Only run --auto-fix --allow-write on skill directories you trust and have checked for symlinks, because write mode can rewrite SKILL.md, remove empty reference files, and delete old backups.

A.I.G

Vulnerability Patterns

Insecure Skill Coding PracticesFinds exploitable flaws such as hardcoded secrets or command injection
Skill Instruction HijackingAlters the agent's session goals or safety constraints when the skill loads
Agent Memory PoisoningWrites attacker-controlled rules into memory that affect later sessions
Remote Payload Retrieval and ExecutionFetches external code whose behavior can change after review
Embedded Malicious CodeShips malicious scripts inside the skill and executes them locally

Findings (1)

T09 · Insecure Skill Coding Practices

Error

Location: scripts/eval.py:328
Finding: Auto-Fix Follows an Unvalidated SKILL.md Symbolic Link
Content: View full analysis
/SKILL.md`. It verifies only that the path exists. It does not verify that `SKILL.md` is a regular file, reject symbolic links, canonicalize the path, or confirm that the resolved file remains inside the resolved skill directory. Python operations including `Path.exists()`, `Path.read_text()`, `Path.write_text()`, and the `shutil.copy()` operation used by `create_backup()` follow symbolic links. Consequently, when auto-fix runs with `--allow-write`, a crafted `SKILL.md` symbolic link can redirect the write operation to another file writable by the invoking account. Exploitation requires the linked file's contents and detected anti-patterns to produce an applicable fix. For example, a Markdown-like target containing a heading but no trigger phrase can be modified by `fix_missing_trigger()`. The explicit `--allow-write` requirement reduces accidental exposure but does not make an attacker-controlled skill directory safe to process. ### Attack Path 1. An attacker prepares a skill directory under their control. 2. The attacker creates `SKILL.md` as a symbolic link to a file writable by the user who will run the evaluator. 3. The linked target contains content that triggers an auto-fix, such as a Markdown heading w ...[truncated 1308 chars]
Remediation: View remediation

SkillSpector

Vulnerability Patterns

Trigger AbuseOverly Broad Trigger, Shadow Command Trigger, Keyword Baiting Trigger
MCP Least PrivilegeUnderdeclared Capability, Wildcard Permission, Missing Permission Declaration
MCP Tool PoisoningHidden Instructions, Unicode Deception, Parameter Description Injection
Prompt InjectionInstruction Override, Hidden Instructions, Exfiltration Commands
Data ExfiltrationExternal Transmission, Env Variable Harvesting, File System Enumeration

Findings (6)

Tp4

High

Category: MCP Tool Poisoning
Confidence: 96% confidence
Finding: The skill is described as a self-contained quality evaluator, but the analyzer indicates hidden or undocumented behaviors that can modify skill files, create backups, and delete files during auto-fix mode. That mismatch is dangerous because operators may run the skill expecting passive analysis while it performs destructive or integrity-altering actions on local content.
Content: No source excerpt is available for this finding.

Intent-Code Divergence

High

Category: Not specified by scanner
Confidence: 98% confidence
Finding: The tool advertises preview-only behavior unless --allow-write is supplied, but create_backup() unconditionally deletes older backup files as part of backup rotation. In write mode this can remove files without additional confirmation, and because backup deletion is separate from the target write, the implementation undermines the stated safety model and can cause unintended data loss in the skill directory.
Content: No source excerpt is available for this finding.

Undeclared Tool Scope

Medium

Category: MCP Least Privilege
Confidence: 89% confidence
Finding: The skill documentation advertises executable evaluation commands and the static analyzer detected file read/write capabilities, but the manifest declares no explicit tool scope such as permissions or allowed-tools. This creates an authorization gap where an agent or user may invoke file-modifying behavior without clear upfront disclosure or restriction, increasing the chance of unintended access to local skill files.
Content: No source excerpt is available for this finding.

Vague Triggers

Medium

Category: Not specified by scanner
Confidence: 95% confidence
Finding: The manifest trigger list includes generic phrases like "evaluate skill," "check quality," and "quality score" without narrowing context, exclusions, or negative examples. In a markdown/manifest file, these broad triggers can cause unintended invocation because they plausibly match ordinary conversation about quality assessment.
Content: No source excerpt is available for this finding.

Description-Behavior Mismatch

Medium

Category: Not specified by scanner
Confidence: 83% confidence
Finding: The module docstring advertises 'LLM judge (Layer 2)' and the CLI exposes --layer2 and --deep modes, yet no evaluate_layer2 implementation exists in this file. For a skill described as self-contained, exposing a missing or implicitly external evaluation layer is inconsistent with the stated scope.
Content: No source excerpt is available for this finding.

Description-Behavior Mismatch

Medium

Category: Not specified by scanner
Confidence: 91% confidence
Finding: The manifest description frames the skill as a quality evaluation engine with scoring, anti-pattern detection, and badges. However, this file includes backup creation, file deletion, and content rewriting logic for SKILL.md and reference files, which goes beyond pure evaluation into remediation and mutation of the evaluated skill.
Content: No source excerpt is available for this finding.

Static analysis

No suspicious patterns detected.