Back to skill

Security audit

evaluation-framework

Security checks for vulnerabilities and agentic risk

Overview

This is a documentation-only evaluation framework; the main risks are broad activation and example commands or automation templates that users should scope carefully.

Before installing, expect this skill to influence evaluation and quality-gate conversations. Review its broad triggers if you want narrower activation, and do not copy the auto-deploy or unpinned pip examples into production workflows without explicit human approval and pinned, isolated tooling.

Vulnerability Patterns
  • Insecure DependenciesIntroduces malicious components through unsafe dependency sources
  • Skill Instruction HijackingAlters the agent's session goals or safety constraints when the skill loads
  • Agent Memory PoisoningWrites attacker-controlled rules into memory that affect later sessions
  • Remote Payload Retrieval and ExecutionFetches external code whose behavior can change after review
  • Embedded Malicious CodeShips malicious scripts inside the skill and executes them locally
Findings (1)

T08 · Insecure Dependencies

Warning
Location
modules/quality-metrics.md:184
Finding
Unpinned Third-Party Package Installation and Immediate Execution<![CDATA[ ## Vulnerability Details **File Location**: `modules/quality-metrics.md:184-186` **Vulnerability Type**: Unpinned dependency installation from a mutable package registry **Risk Level**: Medium ### Vulnerable Code ```text **Measure**: `pip install textstat; python -c "import textstat,sys; print(textstat.flesch_reading_ease(open(sys.argv[1]).read()))" file.md`. ``` ### Technical Analysis The documented measurement procedure instructs an agent or user to install `textstat` without specifying a version, package hash, trusted index, or locked dependency set. It then immediately imports the installed package. Because the dependency is resolved from mutable package-registry state at execution time, the code ultimately executed may differ from what was reviewed. Installation can also execute package build hooks, while the subsequent import executes the package's initialization code. The same risk extends to transitive dependencies selected by the package resolver. This is a supply-chain weakness rather than evidence that the named package is currently malicious. Exploitation would require compromise of the package, one of its dependencies, the configured package index, or the dependency-resolution path. ### Attack Path 1. An attacker compromises a release of `textstat`, one of its transitive dependencies, or a package index used by the environment. 2. The compromised distribution includes malicious installation, build, or import-time code. 3. A user or agent follows the documented reading-level measurement procedure. 4. `pip install textstat` resolves and installs the attacker-controlled distribution because no audited version or integrity hash is required. 5. The following `python -c` command imports `textstat`, executing its import-time code. 6. The payload runs with the operating-system permissions and environmental access of the user who invoked the command. ### Impact Assessment Successful exploitation could provide arbitrary code execution un ...[truncated 507 chars]
Remediation
<![CDATA[ ## Remediation Suggestions 1. Pin `textstat` and all transitive dependencies to reviewed versions in a lock file or requirements file. 2. Record cryptographic hashes and enforce them during installation, for example: ```bash python -m pip install --require-hashes -r requirements.txt ``` 3. Configure an approved package index explicitly and disable unintended supplemental indexes to reduce dependency-confusion exposure. 4. Perform installation inside an isolated virtual environment or disposable container rather than the user's global Python environment. 5. Avoid runtime package installation when possible. Declare the measurement tool as a reviewed development dependency and provision it during a controlled setup phase. 6. Scan and periodically update the locked dependency set using an established dependency-review process. 7. Separate installation from execution so dependency changes can be inspected before the package is imported. ]]>
Vulnerability Patterns
  • Prompt InjectionInstruction Override, Hidden Instructions, Exfiltration Commands
  • Excessive AgencyUnrestricted Tool Access, Autonomous Decision Making, Scope Creep
  • Trigger AbuseOverly Broad Trigger, Shadow Command Trigger, Keyword Baiting Trigger
  • Data ExfiltrationExternal Transmission, Env Variable Harvesting, File System Enumeration
  • Privilege EscalationExcessive Permissions, Sudo/Root Execution, Credential Access
Findings (4)

Anti-Refusal Statement

High
Category
Anti-Refusal
Content
| Score | Meaning |
|-------|---------|
| 2 | Fully satisfied, no caveats |
| 1 | Partially satisfied, named caveat |
| 0 | Not satisfied or absent |
Confidence
80% confidence
Finding
Skill instructs the agent to omit warnings, disclaimers, or ethical commentary. Stripping safety caveats hides risk from the user and is a common jailbreak preamble.

Vague Triggers

Medium
Confidence
94% confidence
Finding
The trigger list includes very broad terms such as "evaluation," "scoring," "metrics," and "quality," which can cause the skill to activate in many unrelated contexts. This increases the chance of unintended invocation, polluting agent behavior or influencing decisions when the user did not intend to use this framework, though it does not directly create code execution or data exfiltration risk.

Autonomous Decision Making

Medium
Category
Excessive Agency
Content
```yaml
thresholds:
  production_deployment:
    90-100: Auto-deploy
    80-89:  Deploy with manual verification
    0-79:   Block deployment
Confidence
85% confidence
Finding
Skill enables autonomous high-impact decisions without human-in-the-loop verification. Critical operations (destructive commands, financial transactions, data deletion) should require explicit user confirmation.

Autonomous Decision Making

Medium
Category
Excessive Agency
Content
```yaml
thresholds:
  production_deployment:
    90-100: Auto-deploy
    80-89:  Deploy with manual verification
    0-79:   Block deployment
Confidence
85% confidence
Finding
Skill enables autonomous high-impact decisions without human-in-the-loop verification. Critical operations (destructive commands, financial transactions, data deletion) should require explicit user confirmation.

Static analysis

No suspicious patterns detected.