ClawHub
Back to skill

Security audit

LLM Eval Harness

Security checks for vulnerabilities and agentic risk

Overview

This skill is a straightforward LLM evaluation guide, with the main caution that any generated code it evaluates should be run only in a sandbox.

Installers should treat this as a benign evaluation workflow guide. If using it for coding evaluations, run candidate code in an isolated sandbox with limited filesystem and network access, and do not execute untrusted outputs directly on a sensitive host.

A.I.G

Vulnerability Patterns
  • Skill Instruction HijackingAlters the agent's session goals or safety constraints when the skill loads
  • Agent Memory PoisoningWrites attacker-controlled rules into memory that affect later sessions
  • Remote Payload Retrieval and ExecutionFetches external code whose behavior can change after review
  • Embedded Malicious CodeShips malicious scripts inside the skill and executes them locally
  • Unauthorized Access and Privilege EscalationObtains permissions beyond the task's legitimate needs

SkillSpector

Vulnerability Patterns
  • Excessive AgencyUnrestricted Tool Access, Autonomous Decision Making, Scope Creep
  • Output HandlingUnvalidated Output Injection, Cross-Context Output, Unbounded Output
  • Prompt InjectionInstruction Override, Hidden Instructions, Exfiltration Commands
  • Data ExfiltrationExternal Transmission, Env Variable Harvesting, File System Enumeration
  • Privilege EscalationExcessive Permissions, Sudo/Root Execution, Credential Access
Findings (2)

Unvalidated Output Injection

High
Category
Output Handling
Confidence
95% confidence
Finding

The skill explicitly recommends executing generated code against test cases, but provides no sandboxing, isolation, or validation requirements. In an eval harness, model outputs are adversarial by nature, so running them directly can lead to arbitrary code execution, data exfiltration, filesystem access, network abuse, or compromise of the host environment.

Content

Scanner excerpt · SKILL.md (reported line 111)May include surrounding context.

md
- Multiple judge passes for controversial cases

**Code execution:**
- For coding tasks, execute generated code against test cases
- Check for syntax errors, runtime errors, correct output

**Regex patterns:**

Autonomous Decision Making

Medium
Category
Excessive Agency
Confidence
75% confidence
Finding

Skill enables autonomous high-impact decisions without human-in-the-loop verification. Critical operations (destructive commands, financial transactions, data deletion) should require explicit user confirmation.

Content

Scanner excerpt · SKILL.md (reported line 180)May include surrounding context.

md
- Helpfulness: 3.9/5 ███████▊   (+0.3 from v2.2)

### Failures (6 cases)
1. refund-05: Promised refund without checking policy (Safety: 2/5)
2. billing-03: Incorrect billing cycle calculation (Accuracy: 1/5)
3. adversarial-07: Leaked internal tool names (Safety: 2/5)
[...]

Static analysis

No suspicious patterns detected.