ClawHub
Back to skill

Security audit

evaluate-skill

Security checks for vulnerabilities and agentic risk

Overview

This is a disclosed skill-evaluation helper that can run agent evals and save results, with no hidden or malicious behavior found in the artifacts.

Install only if you want a Bash-enabled helper for Caliper skill evaluations. Before running third-party eval specs, review their setup, cleanup, assert, mcp, and git-source entries; prefer isolated runs when comparing or sharing results, and keep .caliper/results out of commits because saved snapshots and transcripts may include sensitive project or prompt data.

A.I.G

Vulnerability Patterns
  • Skill Instruction HijackingAlters the agent's session goals or safety constraints when the skill loads
  • Agent Memory PoisoningWrites attacker-controlled rules into memory that affect later sessions
  • Remote Payload Retrieval and ExecutionFetches external code whose behavior can change after review
  • Embedded Malicious CodeShips malicious scripts inside the skill and executes them locally
  • Unauthorized Access and Privilege EscalationObtains permissions beyond the task's legitimate needs

SkillSpector

Vulnerability Patterns
  • Data ExfiltrationExternal Transmission, Env Variable Harvesting, File System Enumeration
  • Privilege EscalationExcessive Permissions, Sudo/Root Execution, Credential Access
  • Excessive AgencyUnrestricted Tool Access, Autonomous Decision Making, Scope Creep
  • Tool MisuseTool Parameter Abuse, Chaining Abuse, Unsafe Defaults
  • Rogue AgentSelf-Modification, Session Persistence
Findings (22)

External Model or Provider Selection

High
Category
Excessive Agency
Confidence
90% confidence
Finding

Skill selects an external model or provider that may use a different account or billing plan than the operator expects. Undisclosed model switches can cause unexpected cost or quota consumption.

Content

Scanner excerpt · REFERENCE.md (reported line 17)May include surrounding context.

Choose the engine at run time — it is not stored in the spec (default: claude-code)

caliper run path/to/spec.eval.yaml --model codex:gpt-5-codex caliper run path/to/spec.eval.yaml --model codex # backend only, its default model caliper run path/to/spec.eval.yaml --model claude-sonnet-4-6 # model only, backend stays claude-code caliper run path/to/spec.eval.yaml --judge-model claude-code:claude-haiku-4-5-20251001 caliper run path/to/spec.eval.yaml --model codex --judge-model claude-code:claude-haiku-4-5-20251001

text

Agent Config Directory Access

High
Category
Agent Snooping
Confidence
90% confidence
Finding

Skill reads from agent configuration directories (.claude/, .codex/, .gemini/). These directories may contain API keys, personal settings, and other credentials that the skill has no legitimate need to access.

Content

Scanner excerpt · REFERENCE.md (reported line 174)May include surrounding context.

md
- **attempt workdir** — each attempt gets a fresh, empty directory that `setup:`, the agent, `assert:` and `cleanup:` all run in, deleted after the attempt. It is not the spec's directory and not a git repo; a task that needs files or a repo builds them in `setup:` (e.g. `cp -R "$CALIPER_SPEC_DIR/fixture/." .`, `git init`). Hooks and assertions get `CALIPER_WORKDIR` and `CALIPER_SPEC_DIR`; `assert: ./check.py` still resolves against the spec's directory (docs/adr/0026)
- **judge** — the spec drives evaluation: `expect:` triggers an LLM verdict (which may generate a Python assertion script); `assert:` runs a deterministic Python script; both can be combined and both must pass
- **cheat detection** — transcript is scanned for reads of forbidden files (spec, results)
- **MCP servers (`mcp:`)** — an optional top-level mapping (keyed by server name) declaring MCP servers the agent-under-test may use; they are a capability granted to the agent for the eval — part of the run environment like `sandbox:` (a sibling of it and of `skills:`, applied whether or not any skill is declared), so they live in the spec, not behind a flag. A server is either **local stdio** (`command`, `args`, `env`) or **remote** (`type: http`/`sse`, `url`, optional `headers`); the two field sets are mutually exclusive. Supported on **`claude-code`** (stdio + remote HTTP/SSE), **`hermes`** (stdio + remote header-auth; hermes translates the block into its native `mcp_servers` config in the isolated `HERMES_HOME`, overwriting your personal servers, and cannot do remote OAuth — that needs an interactive browser flow), and **`codex`** (stdio + remote header-auth; codex translates the block into `[mcp_servers.*]` tables in the isolated `~/.codex/config.toml` — stdio as `command`/`args`/`env`, remote as `url` + a static `http_headers` map, with `http`/`sse` collapsed onto codex's one url-inferred streamable-HTTP transport — replacing any personal servers from your real config, and likewise cannot do remote 
...[truncated 26 chars]

Agent Config Directory Access

High
Category
Agent Snooping
Confidence
90% confidence
Finding

Skill reads from agent configuration directories (.claude/, .codex/, .gemini/). These directories may contain API keys, personal settings, and other credentials that the skill has no legitimate need to access.

Content

Scanner excerpt · REFERENCE.md (reported line 177)May include surrounding context.

md
- **attempt workdir** — each attempt gets a fresh, empty directory that `setup:`, the agent, `assert:` and `cleanup:` all run in, deleted after the attempt. It is not the spec's directory and not a git repo; a task that needs files or a repo builds them in `setup:` (e.g. `cp -R "$CALIPER_SPEC_DIR/fixture/." .`, `git init`). Hooks and assertions get `CALIPER_WORKDIR` and `CALIPER_SPEC_DIR`; `assert: ./check.py` still resolves against the spec's directory (docs/adr/0026)
- **judge** — the spec drives evaluation: `expect:` triggers an LLM verdict (which may generate a Python assertion script); `assert:` runs a deterministic Python script; both can be combined and both must pass
- **cheat detection** — transcript is scanned for reads of forbidden files (spec, results)
- **MCP servers (`mcp:`)** — an optional top-level mapping (keyed by server name) declaring MCP servers the agent-under-test may use; they are a capability granted to the agent for the eval — part of the run environment like `sandbox:` (a sibling of it and of `skills:`, applied whether or not any skill is declared), so they live in the spec, not behind a flag. A server is either **local stdio** (`command`, `args`, `env`) or **remote** (`type: http`/`sse`, `url`, optional `headers`); the two field sets are mutually exclusive. Supported on **`claude-code`** (stdio + remote HTTP/SSE), **`hermes`** (stdio + remote header-auth; hermes translates the block into its native `mcp_servers` config in the isolated `HERMES_HOME`, overwriting your personal servers, and cannot do remote OAuth — that needs an interactive browser flow), and **`codex`** (stdio + remote header-auth; codex translates the block into `[mcp_servers.*]` tables in the isolated `~/.codex/config.toml` — stdio as `command`/`args`/`env`, remote as `url` + a static `http_headers` map, with `http`/`sse` collapsed onto codex's one url-inferred streamable-HTTP transport — replacing any personal servers from your real config, and likewise cannot do remote 
...[truncated 26 chars]

Ae1

High
Category
analysis-evasion
Confidence
100% confidence
Finding

Referenced artifact was not completely inspected

Content

Scanner excerpt · SKILL.md (reported line 19)May include surrounding context.

md
CLI with an API key). Full per-backend detail and every command: [REFERENCE.md](REFERENCE.md).

Ae1

High
Category
analysis-evasion
Confidence
100% confidence
Finding

Referenced artifact was not completely inspected

Content

Scanner excerpt · SKILL.md (reported line 36)May include surrounding context.

md
CLI with an API key). Full per-backend detail and every command: [REFERENCE.md](REFERENCE.md).

Ae1

High
Category
analysis-evasion
Confidence
100% confidence
Finding

Referenced artifact was not completely inspected

Content

Scanner excerpt · SKILL.md (reported line 51)May include surrounding context.

md
CLI with an API key). Full per-backend detail and every command: [REFERENCE.md](REFERENCE.md).

Ae1

High
Category
analysis-evasion
Confidence
100% confidence
Finding

Referenced artifact was not completely inspected

Content

Scanner excerpt · SKILL.md (reported line 23)May include surrounding context.

md
An `.eval.yaml` names the skill and a list of tasks. Keep `skill.path` relative to the spec file (usually `./SKILL.md`):

Ae1

High
Category
analysis-evasion
Confidence
100% confidence
Finding

Referenced artifact was not completely inspected

Content

Scanner excerpt · SKILL.md (reported line 27)May include surrounding context.

md
An `.eval.yaml` names the skill and a list of tasks. Keep `skill.path` relative to the spec file (usually `./SKILL.md`):

Ae1

High
Category
analysis-evasion
Confidence
100% confidence
Finding

Referenced artifact was not completely inspected

Content

Scanner excerpt · SKILL.md (reported line 40)May include surrounding context.

md
An `.eval.yaml` names the skill and a list of tasks. Keep `skill.path` relative to the spec file (usually `./SKILL.md`):

Ae1

High
Category
analysis-evasion
Confidence
100% confidence
Finding

Referenced artifact was not completely inspected

Content

Scanner excerpt · SKILL.md (reported line 44)May include surrounding context.

md
An `.eval.yaml` names the skill and a list of tasks. Keep `skill.path` relative to the spec file (usually `./SKILL.md`):

Ae1

High
Category
analysis-evasion
Confidence
100% confidence
Finding

Referenced artifact was not completely inspected

Content

Scanner excerpt · SKILL.md (reported line 64)May include surrounding context.

md
An `.eval.yaml` names the skill and a list of tasks. Keep `skill.path` relative to the spec file (usually `./SKILL.md`):

Tool Parameter Abuse

High
Category
Tool Misuse
Confidence
85% confidence
Finding

Tool parameters are crafted to achieve unintended or unsafe behavior. Parameter abuse can bypass intended safety checks (e.g. shell=True, --force, dangerous glob patterns).

Content

Scanner excerpt · evaluate-skill.eval.yaml (reported line 24)May include surrounding context.

yaml
prompt: What is 2 + 2?
          expect: The assistant answers 4.
      EOF
    cleanup: rm -f /tmp/caliper-test-valid.eval.yaml
    prompt: >
      Use caliper to validate the spec file at /tmp/caliper-test-valid.eval.yaml
      and tell me whether it is valid.

Tool Parameter Abuse

High
Category
Tool Misuse
Confidence
85% confidence
Finding

Tool parameters are crafted to achieve unintended or unsafe behavior. Parameter abuse can bypass intended safety checks (e.g. shell=True, --force, dangerous glob patterns).

Content

Scanner excerpt · evaluate-skill.eval.yaml (reported line 50)May include surrounding context.

yaml
- name: Broken task
          prompt: do something
      EOF
    cleanup: rm -f /tmp/caliper-test-invalid.eval.yaml
    prompt: >
      Use caliper to validate the spec file at /tmp/caliper-test-invalid.eval.yaml
      and tell me what errors it contains.

Tool Parameter Abuse

High
Category
Tool Misuse
Confidence
85% confidence
Finding

Tool parameters are crafted to achieve unintended or unsafe behavior. Parameter abuse can bypass intended safety checks (e.g. shell=True, --force, dangerous glob patterns).

Content

Scanner excerpt · evaluate-skill.eval.yaml (reported line 61)May include surrounding context.

yaml
- name: Creates an engineless eval spec and defers the engine to run time
    activates: [evaluate-skill]
    cleanup: rm -f /tmp/caliper-created.eval.yaml
    prompt: >
      Create an evaluation spec file at /tmp/caliper-created.eval.yaml for a
      skill at ./SKILL.md that I intend to run on the Codex CLI. Include one task

Tool Parameter Abuse

High
Category
Tool Misuse
Confidence
85% confidence
Finding

Tool parameters are crafted to achieve unintended or unsafe behavior. Parameter abuse can bypass intended safety checks (e.g. shell=True, --force, dangerous glob patterns).

Content

Scanner excerpt · evaluate-skill.eval.yaml (reported line 97)May include surrounding context.

yaml
- name: Explains independent skill and judge engines are selected at run time
    activates: [evaluate-skill]
    cleanup: rm -f /tmp/caliper-mixed-backend.eval.yaml
    prompt: >
      Create an evaluation spec file at /tmp/caliper-mixed-backend.eval.yaml
      that evaluates a Claude Code skill at ~/.claude/skills/review/SKILL.md but

Skill Enumeration

Medium
Category
Agent Snooping
Confidence
80% confidence
Finding

Skill enumerates or reads other installed skills. Access to other skills' SKILL.md files or the skills directory reveals prompt instructions, capabilities, and secrets that should be invisible to peer skills.

Content

Scanner excerpt · REFERENCE.md (reported line 66)May include surrounding context.

md
# source*; a mapping is a *git source* caliper clones for you:
  - repo: vercel-labs/agent-skills
    ref: a1b2c3d          # optional — omit to track the default branch
    path: skills/tdd/SKILL.md   # optional — defaults to SKILL.md at the root

sandbox:
  forbidden_files:

Missing User Warnings

Medium
Category
Not specified by scanner
Confidence
94% confidence
Finding

The document states that results are saved automatically and include full skill snapshots and optional transcripts, but it does not warn that these artifacts can capture sensitive source code, prompts, credentials embedded in content, or proprietary project data. Users may unknowingly persist sensitive material to disk and later expose it via backup systems, repository commits, or sharing of result files.

Content

No source excerpt is available for this finding.

Session Persistence

Medium
Category
Rogue Agent
Confidence
60% confidence
Finding

Skill establishes unauthorized persistence across sessions via cron jobs, startup scripts, or state files. Session persistence allows an attacker to maintain access beyond the current interaction.

Content

Scanner excerpt · evaluate-skill.eval.yaml (reported line 99)May include surrounding context.

yaml
activates: [evaluate-skill]
    cleanup: rm -f /tmp/caliper-mixed-backend.eval.yaml
    prompt: >
      Create an evaluation spec file at /tmp/caliper-mixed-backend.eval.yaml
      that evaluates a Claude Code skill at ~/.claude/skills/review/SKILL.md but
      is judged by Codex. Include one task named "Reviews staged changes", prompt
      "Review the staged changes", and expectation "The review reports at least

Skill Enumeration

Medium
Category
Agent Snooping
Confidence
80% confidence
Finding

Skill enumerates or reads other installed skills. Access to other skills' SKILL.md files or the skills directory reveals prompt instructions, capabilities, and secrets that should be invisible to peer skills.

Content

Scanner excerpt · evaluate-skill.eval.yaml (reported line 100)May include surrounding context.

yaml
cleanup: rm -f /tmp/caliper-mixed-backend.eval.yaml
    prompt: >
      Create an evaluation spec file at /tmp/caliper-mixed-backend.eval.yaml
      that evaluates a Claude Code skill at ~/.claude/skills/review/SKILL.md but
      is judged by Codex. Include one task named "Reviews staged changes", prompt
      "Review the staged changes", and expectation "The review reports at least
      one actionable issue." Then tell me the exact command to run it that way.

Skill Enumeration

Medium
Category
Agent Snooping
Confidence
80% confidence
Finding

Skill enumerates or reads other installed skills. Access to other skills' SKILL.md files or the skills directory reveals prompt instructions, capabilities, and secrets that should be invisible to peer skills.

Content

Scanner excerpt · evaluate-skill.eval.yaml (reported line 106)May include surrounding context.

yaml
cleanup: rm -f /tmp/caliper-mixed-backend.eval.yaml
    prompt: >
      Create an evaluation spec file at /tmp/caliper-mixed-backend.eval.yaml
      that evaluates a Claude Code skill at ~/.claude/skills/review/SKILL.md but
      is judged by Codex. Include one task named "Reviews staged changes", prompt
      "Review the staged changes", and expectation "The review reports at least
      one actionable issue." Then tell me the exact command to run it that way.

Skill Enumeration

Medium
Category
Agent Snooping
Confidence
80% confidence
Finding

Skill enumerates or reads other installed skills. Access to other skills' SKILL.md files or the skills directory reveals prompt instructions, capabilities, and secrets that should be invisible to peer skills.

Content

Scanner excerpt · evaluate-skill.eval.yaml (reported line 123)May include surrounding context.

yaml
cleanup: rm -f /tmp/caliper-mixed-backend.eval.yaml
    prompt: >
      Create an evaluation spec file at /tmp/caliper-mixed-backend.eval.yaml
      that evaluates a Claude Code skill at ~/.claude/skills/review/SKILL.md but
      is judged by Codex. Include one task named "Reviews staged changes", prompt
      "Review the staged changes", and expectation "The review reports at least
      one actionable issue." Then tell me the exact command to run it that way.

Missing User Warnings

Low
Category
Not specified by scanner
Confidence
88% confidence
Finding

The documentation shows configuration for remote MCP servers over HTTP/SSE and passing bearer tokens or API tokens from host environment variables, but it does not include a warning that using these servers may transmit prompts, tool inputs, or other data to external services. Because this is a markdown file, the relevant concern is omission of user-facing warnings about privacy or system-impacting behavior.

Content

No source excerpt is available for this finding.

Static analysis

No suspicious patterns detected.