Back to skill

Security audit

skills-eval

Security checks for vulnerabilities and agentic risk

Overview

ClawScan marked this skill suspicious because it found guidance that could reach beyond skill evaluation, including access to agent settings and other skills and ambiguous cleanup advice; the evidence is incomplete because Skillspector reported partial analysis.

Treat this as a review decision and weigh the flagged examples against your need to audit skills. Before installing, consider whether the skill needs to inspect agent configuration or other installed skills, and whether its troubleshooting advice could affect unrelated work or data. The scan covered the bundled components but left referenced local scripts unresolved, so it does not establish that all referenced material is safe.

Vulnerability Patterns
  • Skill Instruction HijackingAlters the agent's session goals or safety constraints when the skill loads
  • Agent Memory PoisoningWrites attacker-controlled rules into memory that affect later sessions
  • Remote Payload Retrieval and ExecutionFetches external code whose behavior can change after review
  • Embedded Malicious CodeShips malicious scripts inside the skill and executes them locally
  • Unauthorized Access and Privilege EscalationObtains permissions beyond the task's legitimate needs
Vulnerability Patterns
  • Data ExfiltrationExternal Transmission, Env Variable Harvesting, File System Enumeration
  • Excessive AgencyUnrestricted Tool Access, Autonomous Decision Making, Scope Creep
  • Output HandlingUnvalidated Output Injection, Cross-Context Output, Unbounded Output
  • System Prompt LeakageDirect Leakage, Indirect Extraction, Tool-Based Exfiltration
  • Memory PoisoningPersistent Context Injection, Context Window Stuffing, Memory Manipulation
Findings (13)

Memory Manipulation

High
Category
Memory Poisoning
Confidence
80% confidence
Finding

Skill manipulates agent memory, state, or stored context. Memory corruption can alter personality, override safety rules, or cause unpredictable behavior.

Content

Scanner excerpt · modules/evaluation-criteria.md (reported line 438)May include surrounding context.

md
### Activation (20 points)
- [ ] **5+ trigger phrases in description**
- [ ] Clear context indicators
- [ ] Differentiates from alternatives
- [ ] Easy to discover

Tool Parameter Abuse

High
Category
Tool Misuse
Confidence
70% confidence
Finding

Tool parameters are crafted to achieve unintended or unsafe behavior. Parameter abuse can bypass intended safety checks (e.g. shell=True, --force, dangerous glob patterns).

Content

Scanner excerpt · modules/trigger-isolation-analysis.md (reported line 91)May include surrounding context.

md
| Strength | For | Example |
|----------|-----|---------|
| Invariant | Trust boundaries, credentials, destructive commands, machine contracts | "Never run `git push --force` on `master`: it rewrites shared history." |
| Default | This repository's pick among defensible options | "Use `rg`, or `grep` when `rg` is absent." |
| Map | Local facts the model cannot derive | "Use when a hook payload needs validation." |

Anti-Refusal Statement

High
Category
Anti-Refusal
Confidence
80% confidence
Finding

Skill instructs the agent to omit warnings, disclaimers, or ethical commentary. Stripping safety caveats hides risk from the user and is a common jailbreak preamble.

Content

Scanner excerpt · modules/troubleshooting.md (reported line 20)May include surrounding context.

md
**Technical Limits**:
- **Default budget**: Skill description budget scales at 2% of context window (~20,000 chars for 1M context)
- **No warning system**: There's currently no notification when you exceed this threshold
- **Silent failure**: Skills beyond the budget are simply not included in Claude's system prompt

**Solutions**:

Agent Config Directory Access

High
Category
Agent Snooping
Confidence
85% confidence
Finding

Skill reads from agent configuration directories (.claude/, .codex/, .gemini/). These directories may contain API keys, personal settings, and other credentials that the skill has no legitimate need to access.

Content

Scanner excerpt · modules/troubleshooting.md (reported line 44)May include surrounding context.

  1. Audit Skill Count and Size:
bash
# Count total skills
find ~/.claude/skills -name "SKILL.md" | wc -l

# Measure description field sizes
grep -A 5 "^description:" ~/.claude/skills/*/SKILL.md

Agent Config Directory Access

High
Category
Agent Snooping
Confidence
85% confidence
Finding

Skill reads from agent configuration directories (.claude/, .codex/, .gemini/). These directories may contain API keys, personal settings, and other credentials that the skill has no legitimate need to access.

Content

Scanner excerpt · modules/troubleshooting.md (reported line 47)May include surrounding context.

md
find ~/.claude/skills -name "SKILL.md" | wc -l

# Measure description field sizes
grep -A 5 "^description:" ~/.claude/skills/*/SKILL.md

# Estimate tokens per skill (run from plugins/abstract)
python3 scripts/token_usage_tracker.py ~/.claude/skills

Skill Enumeration

Medium
Category
Agent Snooping
Confidence
80% confidence
Finding

Skill enumerates or reads other installed skills. Access to other skills' SKILL.md files or the skills directory reveals prompt instructions, capabilities, and secrets that should be invisible to peer skills.

Content

Scanner excerpt · modules/integration.md (reported line 14)May include surrounding context.

bash
# Combined workflow, run from plugins/abstract
python3 scripts/skill_analyzer.py --file skills/skills-eval/SKILL.md
python3 scripts/compliance_checker.py skills/
python3 scripts/improvement_suggester.py skills/skills-eval

Autonomous Decision Making

Medium
Category
Excessive Agency
Confidence
75% confidence
Finding

Skill enables autonomous high-impact decisions without human-in-the-loop verification. Critical operations (destructive commands, financial transactions, data deletion) should require explicit user confirmation.

Content

Scanner excerpt · modules/skill-authoring-best-practices.md (reported line 276)May include surrounding context.

md
| Abstract Quick Start | Reader cannot copy-paste | Use literal commands |
| Relative cross-refs | Breaks across installs | Use `Skill()` form |
| Second-person voice | Treated as user docs | Convert to third person |
| No verification section | Claude declares done early | Add explicit checks |
| Stale cited paths | Hallucinated content | Re-verify on each release |

## How to use this module

Skill Enumeration

Medium
Category
Agent Snooping
Confidence
85% confidence
Finding

Skill enumerates or reads other installed skills. Access to other skills' SKILL.md files or the skills directory reveals prompt instructions, capabilities, and secrets that should be invisible to peer skills.

Content

Scanner excerpt · modules/troubleshooting.md (reported line 44)May include surrounding context.

  1. Audit Skill Count and Size:
bash
# Count total skills
find ~/.claude/skills -name "SKILL.md" | wc -l

# Measure description field sizes
grep -A 5 "^description:" ~/.claude/skills/*/SKILL.md

Skill Enumeration

Medium
Category
Agent Snooping
Confidence
85% confidence
Finding

Skill enumerates or reads other installed skills. Access to other skills' SKILL.md files or the skills directory reveals prompt instructions, capabilities, and secrets that should be invisible to peer skills.

Content

Scanner excerpt · modules/troubleshooting.md (reported line 94)May include surrounding context.

Issue: No skills found during discovery

bash
# Solution: point the auditor at the directory that holds the skills
ls ~/.claude/skills/
python3 scripts/skills_auditor.py ~/.claude/skills

Skill Enumeration

Medium
Category
Agent Snooping
Confidence
80% confidence
Finding

Skill enumerates or reads other installed skills. Access to other skills' SKILL.md files or the skills directory reveals prompt instructions, capabilities, and secrets that should be invisible to peer skills.

Content

Scanner excerpt · modules/troubleshooting.md (reported line 109)May include surrounding context.

Issue: Slow evaluation performance

bash
# Solution: analyze one skill file instead of a whole tree
python3 scripts/token_estimator.py --file skills/skills-eval/SKILL.md
python3 scripts/skill_analyzer.py --file skills/skills-eval/SKILL.md

Skill Enumeration

Medium
Category
Agent Snooping
Confidence
80% confidence
Finding

Skill enumerates or reads other installed skills. Access to other skills' SKILL.md files or the skills directory reveals prompt instructions, capabilities, and secrets that should be invisible to peer skills.

Content

Scanner excerpt · modules/troubleshooting.md (reported line 110)May include surrounding context.

bash
# Solution: analyze one skill file instead of a whole tree
python3 scripts/token_estimator.py --file skills/skills-eval/SKILL.md
python3 scripts/skill_analyzer.py --file skills/skills-eval/SKILL.md

Issue: High memory usage during analysis

Skill Enumeration

Medium
Category
Agent Snooping
Confidence
80% confidence
Finding

Skill enumerates or reads other installed skills. Access to other skills' SKILL.md files or the skills directory reveals prompt instructions, capabilities, and secrets that should be invisible to peer skills.

Content

Scanner excerpt · modules/troubleshooting.md (reported line 141)May include surrounding context.

skill_analyzer.py and token_estimator.py accept --verbose (or -v, repeatable). The other scripts have no debug flag.

bash
python3 scripts/skill_analyzer.py --file skills/skills-eval/SKILL.md -v

Performance Analysis

Recovery instructions recommend terminating processes and deleting temporary files without warning about potential data loss.

Medium
Category
Output Handling
Confidence
87% confidence
Finding

The recovery guidance recommends killing hanging processes and clearing temporary files without restricting targets to the evaluation's own resources or requiring checks for active work. In an agent-operated troubleshooting workflow, this ambiguity could lead to terminating unrelated jobs or deleting recoverable intermediate data, although no destructive command is supplied.

Content

No source excerpt is available for this finding.