ClawHub
Back to skill

Security audit

grill-skill

Security checks for vulnerabilities and agentic risk

Overview

The skill is a disclosed eval-building workflow that reads a user-selected skill, writes eval specs with confirmation, and runs Caliper tests; its sensitive behaviors are expected for that purpose.

Before installing, be aware this skill can read and edit the skill files you point it at and can run Caliper, which may execute task setup/cleanup commands and load your local agent customizations unless isolated. Use it in a project you trust, review generated eval YAML before confirming writes, and use isolated Caliper mode for portable or shared results.

A.I.G

Vulnerability Patterns
  • Skill Instruction HijackingAlters the agent's session goals or safety constraints when the skill loads
  • Agent Memory PoisoningWrites attacker-controlled rules into memory that affect later sessions
  • Remote Payload Retrieval and ExecutionFetches external code whose behavior can change after review
  • Embedded Malicious CodeShips malicious scripts inside the skill and executes them locally
  • Unauthorized Access and Privilege EscalationObtains permissions beyond the task's legitimate needs

SkillSpector

Vulnerability Patterns
  • Data ExfiltrationExternal Transmission, Env Variable Harvesting, File System Enumeration
  • Privilege EscalationExcessive Permissions, Sudo/Root Execution, Credential Access
  • Excessive AgencyUnrestricted Tool Access, Autonomous Decision Making, Scope Creep
  • Tool MisuseTool Parameter Abuse, Chaining Abuse, Unsafe Defaults
  • Behavioral ASTexec() Call, eval() Call, Dynamic Import
Findings (23)

Agent Config Directory Access

High
Category
Agent Snooping
Confidence
90% confidence
Finding

Skill reads from agent configuration directories (.claude/, .codex/, .gemini/). These directories may contain API keys, personal settings, and other credentials that the skill has no legitimate need to access.

Content

Scanner excerpt · REFERENCE.md (reported line 230)May include surrounding context.

md
## MCP servers (`mcp:`)

If the skill under test needs MCP tools, declare them in a top-level `mcp:` block (a mapping keyed by server name) — a capability granted to the agent-under-test for the eval, part of the run environment like `sandbox:` (a sibling of it and of `skills:`), so they belong in the spec, not on the command line. A server is either **local stdio** (a `command`, optional `args`, optional `env`) or **remote** (`type: http`/`sse`, a `url`, optional `headers` for auth); the two field sets are mutually exclusive. Supported on **`claude-code`** (stdio + remote HTTP/SSE), **`hermes`** (stdio + remote header-auth; not remote OAuth), and **`codex`** (stdio + remote header-auth, translated into `[mcp_servers.*]` tables in the isolated `~/.codex/config.toml`; not remote OAuth). A tool call appears in the transcript as a namespaced name — `mcp__<server>__<tool>` on `claude-code` and `codex`, `mcp_<server>_<tool>` on `hermes` — so an `expect:` criterion can check the skill actually used it; word it around behaviour, not one backend's spelling, if the spec runs under more than one engine. Put secrets in a host env var and reference it as `${VAR}` inside a stdio `env:`, a remote `headers:`, or a remote `url:` — it resolves at the harness boundary from your shell at run time and never lands in the committed spec (an unset var fails the run). Running an `mcp:` spec on a backend that can't honor it is a hard error, not a silent no-op: `pi` has no MCP by design and will not honor `mcp:` natively — expose the capability as a CLI tool the skill drives or a pi extension, or run the eval on `claude-code`/`hermes`/`codex`. By default a run also loads user skills, plugins, rules and settings, alongside the user's MCP servers and account connectors (Gmail, Drive, GitHub, and the like), merged with `mcp:` — the declared server wins a name clash — so a task may rely on a connector the user has. The judge keeps its existing connector isolation, and `pi` has nothing to load. F
...[truncated 25 chars]

Ae1

High
Category
analysis-evasion
Confidence
100% confidence
Finding

Referenced artifact was not completely inspected

Content

Scanner excerpt · SKILL.md (reported line 13)May include surrounding context.

md
`/grill-skill [path]` — optional path to a `SKILL.md`.

Ae1

High
Category
analysis-evasion
Confidence
100% confidence
Finding

Referenced artifact was not completely inspected

Content

Scanner excerpt · SKILL.md (reported line 16)May include surrounding context.

md
`/grill-skill [path]` — optional path to a `SKILL.md`.

Ae1

High
Category
analysis-evasion
Confidence
100% confidence
Finding

Referenced artifact was not completely inspected

Content

Scanner excerpt · SKILL.md (reported line 20)May include surrounding context.

md
`/grill-skill [path]` — optional path to a `SKILL.md`.

Ae1

High
Category
analysis-evasion
Confidence
100% confidence
Finding

Referenced artifact was not completely inspected

Content

Scanner excerpt · SKILL.md (reported line 24)May include surrounding context.

md
`/grill-skill [path]` — optional path to a `SKILL.md`.

Ae1

High
Category
analysis-evasion
Confidence
100% confidence
Finding

Referenced artifact was not completely inspected

Content

Scanner excerpt · SKILL.md (reported line 40)May include surrounding context.

md
`/grill-skill [path]` — optional path to a `SKILL.md`.

Ae1

High
Category
analysis-evasion
Confidence
100% confidence
Finding

Referenced artifact was not completely inspected

Content

Scanner excerpt · SKILL.md (reported line 60)May include surrounding context.

md
`/grill-skill [path]` — optional path to a `SKILL.md`.

Ae1

High
Category
analysis-evasion
Confidence
100% confidence
Finding

Referenced artifact was not completely inspected

Content

Scanner excerpt · SKILL.md (reported line 61)May include surrounding context.

md
`/grill-skill [path]` — optional path to a `SKILL.md`.

Tool Parameter Abuse

High
Category
Tool Misuse
Confidence
90% confidence
Finding

Tool parameters are crafted to achieve unintended or unsafe behavior. Parameter abuse can bypass intended safety checks (e.g. shell=True, --force, dangerous glob patterns).

Content

Scanner excerpt · grill-skill.eval.yaml (reported line 21)May include surrounding context.

yaml
- name: Interviews before generating — asks, then stops (does not run ahead)
    activates: [grill-skill]
    setup: |
      rm -rf /tmp/grill-fresh
      mkdir -p /tmp/grill-fresh
      cat > /tmp/grill-fresh/SKILL.md << 'EOF'
      ---

Tool Parameter Abuse

High
Category
Tool Misuse
Confidence
90% confidence
Finding

Tool parameters are crafted to achieve unintended or unsafe behavior. Parameter abuse can bypass intended safety checks (e.g. shell=True, --force, dangerous glob patterns).

Content

Scanner excerpt · grill-skill.eval.yaml (reported line 35)May include surrounding context.

yaml
- name: Interviews before generating — asks, then stops (does not run ahead)
    activates: [grill-skill]
    setup: |
      rm -rf /tmp/grill-fresh
      mkdir -p /tmp/grill-fresh
      cat > /tmp/grill-fresh/SKILL.md << 'EOF'
      ---

Tool Parameter Abuse

High
Category
Tool Misuse
Confidence
90% confidence
Finding

Tool parameters are crafted to achieve unintended or unsafe behavior. Parameter abuse can bypass intended safety checks (e.g. shell=True, --force, dangerous glob patterns).

Content

Scanner excerpt · grill-skill.eval.yaml (reported line 61)May include surrounding context.

yaml
- name: Interviews before generating — asks, then stops (does not run ahead)
    activates: [grill-skill]
    setup: |
      rm -rf /tmp/grill-fresh
      mkdir -p /tmp/grill-fresh
      cat > /tmp/grill-fresh/SKILL.md << 'EOF'
      ---

Tool Parameter Abuse

High
Category
Tool Misuse
Confidence
90% confidence
Finding

Tool parameters are crafted to achieve unintended or unsafe behavior. Parameter abuse can bypass intended safety checks (e.g. shell=True, --force, dangerous glob patterns).

Content

Scanner excerpt · grill-skill.eval.yaml (reported line 82)May include surrounding context.

yaml
- name: Interviews before generating — asks, then stops (does not run ahead)
    activates: [grill-skill]
    setup: |
      rm -rf /tmp/grill-fresh
      mkdir -p /tmp/grill-fresh
      cat > /tmp/grill-fresh/SKILL.md << 'EOF'
      ---

Tool Parameter Abuse

High
Category
Tool Misuse
Confidence
90% confidence
Finding

Tool parameters are crafted to achieve unintended or unsafe behavior. Parameter abuse can bypass intended safety checks (e.g. shell=True, --force, dangerous glob patterns).

Content

Scanner excerpt · grill-skill.eval.yaml (reported line 110)May include surrounding context.

yaml
- name: Interviews before generating — asks, then stops (does not run ahead)
    activates: [grill-skill]
    setup: |
      rm -rf /tmp/grill-fresh
      mkdir -p /tmp/grill-fresh
      cat > /tmp/grill-fresh/SKILL.md << 'EOF'
      ---

Tool Parameter Abuse

High
Category
Tool Misuse
Confidence
90% confidence
Finding

Tool parameters are crafted to achieve unintended or unsafe behavior. Parameter abuse can bypass intended safety checks (e.g. shell=True, --force, dangerous glob patterns).

Content

Scanner excerpt · grill-skill.eval.yaml (reported line 123)May include surrounding context.

yaml
- name: Interviews before generating — asks, then stops (does not run ahead)
    activates: [grill-skill]
    setup: |
      rm -rf /tmp/grill-fresh
      mkdir -p /tmp/grill-fresh
      cat > /tmp/grill-fresh/SKILL.md << 'EOF'
      ---

Tool Parameter Abuse

High
Category
Tool Misuse
Confidence
90% confidence
Finding

Tool parameters are crafted to achieve unintended or unsafe behavior. Parameter abuse can bypass intended safety checks (e.g. shell=True, --force, dangerous glob patterns).

Content

Scanner excerpt · grill-skill.eval.yaml (reported line 21)May include surrounding context.

yaml
- name: Interviews before generating — asks, then stops (does not run ahead)
    activates: [grill-skill]
    setup: |
      rm -rf /tmp/grill-fresh
      mkdir -p /tmp/grill-fresh
      cat > /tmp/grill-fresh/SKILL.md << 'EOF'
      ---

Tool Parameter Abuse

High
Category
Tool Misuse
Confidence
90% confidence
Finding

Tool parameters are crafted to achieve unintended or unsafe behavior. Parameter abuse can bypass intended safety checks (e.g. shell=True, --force, dangerous glob patterns).

Content

Scanner excerpt · grill-skill.eval.yaml (reported line 35)May include surrounding context.

yaml
- name: Interviews before generating — asks, then stops (does not run ahead)
    activates: [grill-skill]
    setup: |
      rm -rf /tmp/grill-fresh
      mkdir -p /tmp/grill-fresh
      cat > /tmp/grill-fresh/SKILL.md << 'EOF'
      ---

Tool Parameter Abuse

High
Category
Tool Misuse
Confidence
90% confidence
Finding

Tool parameters are crafted to achieve unintended or unsafe behavior. Parameter abuse can bypass intended safety checks (e.g. shell=True, --force, dangerous glob patterns).

Content

Scanner excerpt · grill-skill.eval.yaml (reported line 61)May include surrounding context.

yaml
- name: Detects an existing eval and asks before touching it
    activates: [grill-skill]
    setup: |
      rm -rf /tmp/grill-existing
      mkdir -p /tmp/grill-existing
      cat > /tmp/grill-existing/SKILL.md << 'EOF'
      ---

Tool Parameter Abuse

High
Category
Tool Misuse
Confidence
90% confidence
Finding

Tool parameters are crafted to achieve unintended or unsafe behavior. Parameter abuse can bypass intended safety checks (e.g. shell=True, --force, dangerous glob patterns).

Content

Scanner excerpt · grill-skill.eval.yaml (reported line 82)May include surrounding context.

yaml
- name: Detects an existing eval and asks before touching it
    activates: [grill-skill]
    setup: |
      rm -rf /tmp/grill-existing
      mkdir -p /tmp/grill-existing
      cat > /tmp/grill-existing/SKILL.md << 'EOF'
      ---

Tool Parameter Abuse

High
Category
Tool Misuse
Confidence
90% confidence
Finding

Tool parameters are crafted to achieve unintended or unsafe behavior. Parameter abuse can bypass intended safety checks (e.g. shell=True, --force, dangerous glob patterns).

Content

Scanner excerpt · grill-skill.eval.yaml (reported line 110)May include surrounding context.

yaml
- name: Serializes a valid 3-task spec when the structure is given
    activates: [grill-skill]
    setup: |
      rm -rf /tmp/grill-serialize
      mkdir -p /tmp/grill-serialize
      cat > /tmp/grill-serialize/SKILL.md << 'EOF'
      ---

Tool Parameter Abuse

High
Category
Tool Misuse
Confidence
90% confidence
Finding

Tool parameters are crafted to achieve unintended or unsafe behavior. Parameter abuse can bypass intended safety checks (e.g. shell=True, --force, dangerous glob patterns).

Content

Scanner excerpt · grill-skill.eval.yaml (reported line 123)May include surrounding context.

yaml
- name: Serializes a valid 3-task spec when the structure is given
    activates: [grill-skill]
    setup: |
      rm -rf /tmp/grill-serialize
      mkdir -p /tmp/grill-serialize
      cat > /tmp/grill-serialize/SKILL.md << 'EOF'
      ---

Skill Enumeration

Medium
Category
Agent Snooping
Confidence
80% confidence
Finding

Skill enumerates or reads other installed skills. Access to other skills' SKILL.md files or the skills directory reveals prompt instructions, capabilities, and secrets that should be invisible to peer skills.

Content

Scanner excerpt · REFERENCE.md (reported line 116)May include surrounding context.

md
# source*; a mapping is a *git source* caliper clones for you:
  - repo: vercel-labs/agent-skills
    ref: a1b2c3d          # optional — omit to track the default branch
    path: skills/tdd/SKILL.md   # optional — defaults to SKILL.md at the root
                          # a symlink out of the cloned repo refuses the run

sandbox:

Skill Enumeration

Medium
Category
Agent Snooping
Confidence
80% confidence
Finding

Skill enumerates or reads other installed skills. Access to other skills' SKILL.md files or the skills directory reveals prompt instructions, capabilities, and secrets that should be invisible to peer skills.

Content

Scanner excerpt · REFERENCE.md (reported line 204)May include surrounding context.

The spec file lives next to the skill and shares its directory name:

text
skills/my-skill/SKILL.md
skills/my-skill/my-skill.eval.yaml   ← generated here

Autonomous Decision Making

Medium
Category
Excessive Agency
Confidence
75% confidence
Finding

Skill enables autonomous high-impact decisions without human-in-the-loop verification. Critical operations (destructive commands, financial transactions, data deletion) should require explicit user confirmation.

Content

Scanner excerpt · grill-skill.eval.yaml (reported line 46)May include surrounding context.

yaml
the SKILL.md, gives some understanding of the skill, and asks the user for
      input before generating anything — then STOPS to wait. The number of
      questions does not matter. Fail if the agent skips the interview: it
      invents the user's answers, generates the eval tasks itself without
      asking, or writes any .eval.yaml file in this turn.
    assert: |
      import glob

Static analysis

No suspicious patterns detected.