SkillProbe

• Outcome: Review
• Latest audit: Sep 12, 2026 · 6:07 AM UTC
• Version: 1.0.0

Overview

SkillProbe is a coherent skill-evaluation workflow, but it evaluates untrusted skills by telling a tool-capable sub-agent to apply their full instructions without clear containment.

Install only if you will run evaluations in a sandboxed or low-privilege agent environment. Do not use it to evaluate unknown skills in a session that has sensitive files, credentials, network tools, or write access available to sub-agents unless you have added separate containment and logging.

A.I.G

Vulnerability Patterns

• Skill Instruction Hijacking: Alters the agent's session goals or safety constraints when the skill loads
• Tool Hijacking and Spoofing: Modifies or replaces tools so legitimate-looking calls execute attacker logic
• Agent Memory Poisoning: Writes attacker-controlled rules into memory that affect later sessions
• Remote Payload Retrieval and Execution: Fetches external code whose behavior can change after review
• Embedded Malicious Code: Ships malicious scripts inside the skill and executes them locally

Findings (2)

T01 · Skill Instruction Hijacking

Error

• Location: DISPATCH_PROTOCOL.md:46

Finding

Untrusted Skill Instructions Are Directly Applied by a Tool-Capable Sub-Agent

Content

View full analysis

Vulnerability Details

File Location: DISPATCH_PROTOCOL.md, lines 46-59
Vulnerability Type: Untrusted instruction execution and prompt hijacking
Risk Level: High

Vulnerable Code

text

You are a task executor. Read the skill below first, then apply its guidance to every task.

## Skill to Apply
{SKILL_CONTENT}

---

Rules:
- Understand the skill content FIRST
- Apply its methodology to each task
- Answer each task independently

Tasks:
{TASK_LIST}

Technical Analysis

The dispatch protocol inserts the complete contents of the Skill being evaluated into the instruction context of Sub-Agent B. It then explicitly directs that agent to understand and apply the embedded instructions.

The candidate Skill is an untrusted input because SkillProbe is intended to evaluate third-party Skills before users decide whether to install or retain them. The protocol does not establish an instruction-versus-data boundary, detect hostile directives, restrict the sub-agent's tools, deny network access, isolate the host filesystem, or remove credentials from the execution environment.

Consequently, a malicious candidate Skill can include instructions that override the intended evaluation task. Depending on the tools and permissions exposed by the agent runtime, such instructions could direct the sub-agent to read unrelated files, disclose sensitive content, invoke external services, modify local resources, or disregard the required output format. Separate sessions protect the baseline arm from contamination but do not secure the with-Skill arm or its surrounding runtime.

The same unsafe behavior is required by SKILL.md, lines 42-48, which specifies that Sub-Agent B receives the full Skill content and applies it.

Attack Path

1. An attacker publishes or supplies a candidate Skill containing hostile instructions.
2. A user asks SkillProbe to evaluate that candidate.
3. The orchestrator copies the ...[truncated 1385 chars]

Remediation

View remediation

Remediation Suggestions

1. Treat all candidate Skill content as untrusted data rather than operative instructions.
2. Perform an initial static review in a no-tools agent session. Require that session to describe the candidate's behavior without applying its directives.
3. Run dynamic evaluations only inside an ephemeral sandbox with:
   • No host filesystem access.
   • No inherited secrets, API tokens, SSH agents, or cloud credentials.
   • Network access denied by default.
   • A minimal, explicit tool allowlist.
   • Read-only test fixtures.
   • CPU, token, time, and storage limits.
4. Add higher-priority dispatch rules stating that embedded Skill content cannot modify the evaluation protocol, request unrelated data, invoke unapproved tools, create persistence, delegate work, or communicate externally.
5. Delimit and encode candidate content as quoted data. Do not represent it as a trusted Skill to Apply section in the same instruction hierarchy.
6. Scan candidate Skills for tool-use directives, credential access, remote communication, persistence, and instruction-override patterns before dynamic execution.
7. Require explicit user approval before evaluating a Skill that requests privileged tools or network access.
8. Record all tool calls and reject an evaluation if the candidate attempts actions outside a narrowly defined test workspace.

T07 · Tool Hijacking and Spoofing

Warning

• Location: scripts/evaluate.sh:74

Finding

Helper Script Executes an Unverified Runtime from the Parent Tree or PATH

Content

View full analysis

Vulnerability Details

File Location: scripts/evaluate.sh, lines 74-83 and 104-140
Vulnerability Type: Executable and Python module resolution hijacking
Risk Level: Medium

Vulnerable Code

bash

SCRIPT_DIR=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)
PACKAGE_ROOT=$(cd "$SCRIPT_DIR/../.." && pwd)
HAS_RUNTIME=0

if [ -f "$PACKAGE_ROOT/apps/cli/main.py" ]; then
    HAS_RUNTIME=1
fi

if command -v skillprobe >/dev/null 2>&1; then
    HAS_RUNTIME=1
fi

bash

if [ -f "$PACKAGE_ROOT/apps/cli/main.py" ]; then
    CMD=(python -m apps.cli.main evaluate "$SKILL_PATH" --tasks "$TASKS" --repeats "$REPEATS" --db "$DB_PATH")
    if [ -n "$MODEL" ]; then
        CMD+=(--model "$MODEL")
    fi
    if [ "$ENABLE_LLM_JUDGE" -eq 1 ]; then
        CMD+=(--llm-judge)
    fi
    if [ -n "$JUDGE_MODEL" ]; then
        CMD+=(--judge-model "$JUDGE_MODEL")
    fi
    (
        cd "$PACKAGE_ROOT"
        "${CMD[@]}"
    )
    exit 0
fi

if command -v skillprobe >/dev/null 2>&1; then
    CMD=(skillprobe evaluate "$SKILL_PATH" --tasks "$TASKS" --repeats "$REPEATS" --db "$DB_PATH")
    if [ -n "$MODEL" ]; then
        CMD+=(--model "$MODEL")
    fi
    if [ "$ENABLE_LLM_JUDGE" -eq 1 ]; then
        CMD+=(--llm-judge)
    fi
    if [ -n "$JUDGE_MODEL" ]; then
        CMD+=(--judge-model "$JUDGE_MODEL")
    fi
    "${CMD[@]}"
    exit 0
fi

Technical Analysis

The helper calculates PACKAGE_ROOT by moving two directories above its own scripts directory. That location is outside the distributed Skill directory shown in the project structure. If an apps/cli/main.py file exists there, the script changes into that directory and runs python -m apps.cli.main without verifying that the module belongs to the legitimate SkillProbe project.

If that file is absent, the script executes the first skillprobe command resolved ...[truncated 2149 chars]

Remediation

View remediation

Remediation Suggestions

1. Do not discover the Python runtime by traversing outside the installed Skill package.
2. Require an explicitly configured, absolute path to the trusted SkillProbe executable or Python environment.
3. Resolve the executable once with command -v, canonicalize its path, and reject locations outside an administrator-approved installation directory.
4. Verify the runtime's package identity and expected version before execution.
5. Prefer a dedicated virtual environment whose interpreter and module paths are fixed, for example an absolute path to the environment's Python binary.
6. Avoid generic python -m apps.cli.main; use a uniquely named installed package and verify its distribution metadata.
7. If PATH fallback must remain, print the resolved executable path and require explicit user confirmation before running it.
8. In higher-assurance deployments, verify a cryptographic digest or signed release manifest for the executable and its Python modules.
9. Reject package roots or executables located in directories writable by untrusted users.

SkillSpector

Vulnerability Patterns

• Prompt Injection: Instruction Override, Hidden Instructions, Exfiltration Commands
• Data Exfiltration: External Transmission, Env Variable Harvesting, File System Enumeration
• Privilege Escalation: Excessive Permissions, Sudo/Root Execution, Credential Access
• Supply Chain: Unpinned Dependencies, External Script Fetching, Obfuscated Code
• Excessive Agency: Unrestricted Tool Access, Autonomous Decision Making, Scope Creep

Static analysis

No suspicious patterns detected.