ClawHub
Back to skill

Security audit

agent-evaluation

Security checks for vulnerabilities and agentic risk

Overview

The skill is a coherent agent-evaluation guide, but users should know its example sends evaluation messages to SkillBoss's external API.

Install only if you are comfortable sending evaluation messages to SkillBoss. Use synthetic or redacted test data unless your organization has approved sending production prompts, traces, user conversations, or confidential benchmarks to that service.

A.I.G

Vulnerability Patterns
  • Insecure Skill Coding PracticesFinds exploitable flaws such as hardcoded secrets or command injection
  • Skill Instruction HijackingAlters the agent's session goals or safety constraints when the skill loads
  • Agent Memory PoisoningWrites attacker-controlled rules into memory that affect later sessions
  • Remote Payload Retrieval and ExecutionFetches external code whose behavior can change after review
  • Embedded Malicious CodeShips malicious scripts inside the skill and executes them locally
Findings (1)

T09 · Insecure Skill Coding Practices

Warning
Location
SKILL.md:36
Finding

Unredacted Evaluation Data Sent to a Fixed Third-Party API

Content
View full analysis

Vulnerability Details

File Location: SKILL.md, lines 36–51
Vulnerability Type: Sensitive information transmitted to an external service without redaction or explicit consent controls
Risk Level: Medium

Vulnerable Code

python
import requests, os

SKILLBOSS_API_KEY = os.environ["SKILLBOSS_API_KEY"]

def call_llm(messages: list) -> str:
    r = requests.post(
        "https://api.skillboss.co/v1/pilot",
        headers={"Authorization": f"Bearer {SKILLBOSS_API_KEY}", "Content-Type": "application/json"},
        json={
            "type": "chat",
            "inputs": {"messages": messages},
            "prefer": "balanced"
        },
        timeout=60,
    )
    return r.json()["result"]["choices"][0]["message"]["content"]

Technical Analysis

The documented function transmits the entire caller-supplied messages collection to the fixed third-party endpoint https://api.skillboss.co/v1/pilot. It does not classify or redact data, detect embedded credentials, request explicit approval, or offer a local or user-selected provider.

Agent-evaluation inputs may include production prompts, model outputs, proprietary test cases, system instructions, personal information, authentication material, and operational traces. Any such content placed in messages is included unchanged in the outbound JSON request. HTTPS protects the data in transit but does not prevent disclosure to or processing by the remote service.

External LLM access can be relevant to the skill's stated functionality, but requiring a single external service is not the minimum privilege necessary for generic agent evaluation. Evaluation can instead operate against a configurable provider or local model. The API credential itself is read from an environment variable and sent only to its apparent intended endpoint; the reviewed files contain no evidence that the credential is exfiltrated to an unrelated destination.

The fu ...[truncated 1731 chars]

Remediation
View remediation

Remediation Suggestions

  1. Require explicit, informed user approval before transmitting evaluation data to an external provider.
  2. Clearly document the destination, categories of data transmitted, retention policy, subprocessors, and applicable privacy terms.
  3. Make the API endpoint and provider configurable rather than requiring a fixed third-party service.
  4. Provide a local or self-hosted backend option for sensitive evaluation workflows.
  5. Add a preprocessing layer that detects and redacts credentials, access tokens, personal information, proprietary identifiers, and other configured sensitive fields.
  6. Default to denying transmission when secret-like content is detected, requiring an explicit override rather than silently sending it.
  7. Minimize payloads by transmitting only the fields necessary for a specific evaluation.
  8. Separate synthetic benchmark workflows from production-trace workflows and apply stricter controls to production data.
  9. Add r.raise_for_status(), validate response content and schema, and handle timeouts, invalid JSON, and unexpected responses safely.
  10. Avoid logging authorization headers, full request bodies, or unredacted response content.
  11. Document secure API-key handling, rotation, and least-privilege configuration.

SkillSpector

Vulnerability Patterns
  • Data ExfiltrationExternal Transmission, Env Variable Harvesting, File System Enumeration
  • Prompt InjectionInstruction Override, Hidden Instructions, Exfiltration Commands
  • Privilege EscalationExcessive Permissions, Sudo/Root Execution, Credential Access
  • Supply ChainUnpinned Dependencies, External Script Fetching, Obfuscated Code
  • Excessive AgencyUnrestricted Tool Access, Autonomous Decision Making, Scope Creep
Findings (4)

Missing User Warnings

Medium
Category
Not specified by scanner
Confidence
92% confidence
Finding

The skill includes example code that sends arbitrary evaluation messages to an external third-party API, but the surrounding documentation does not clearly warn users that prompts, test cases, or potentially sensitive production data will leave the local environment. In an agent-evaluation context, those messages may include proprietary prompts, benchmark data, or user interactions, so omission of a clear disclosure creates a real data-exposure risk.

Content

No source excerpt is available for this finding.

External Transmission

Medium
Category
Data Exfiltration
Confidence
88% confidence
Finding

The finding is substantively the same as the other line-44 E1: the function posts evaluation content to an external API using an authorization token. Even though this appears to be normal product functionality rather than malicious behavior, it still creates a meaningful risk of unintentional disclosure if users assume evaluation runs are local or fail to sanitize inputs.

Content

Scanner excerpt · SKILL.md (reported line 44)May include surrounding context.

md
SKILLBOSS_API_KEY = os.environ["SKILLBOSS_API_KEY"]

def call_llm(messages: list) -> str:
    r = requests.post(
        "https://api.skillboss.co/v1/pilot",
        headers={"Authorization": f"Bearer {SKILLBOSS_API_KEY}", "Content-Type": "application/json"},
        json={

External Transmission

Medium
Category
Data Exfiltration
Confidence
88% confidence
Finding

The finding is substantively the same as the other line-44 E1: the function posts evaluation content to an external API using an authorization token. Even though this appears to be normal product functionality rather than malicious behavior, it still creates a meaningful risk of unintentional disclosure if users assume evaluation runs are local or fail to sanitize inputs.

Content

Scanner excerpt · SKILL.md (reported line 44)May include surrounding context.

md
SKILLBOSS_API_KEY = os.environ["SKILLBOSS_API_KEY"]

def call_llm(messages: list) -> str:
    r = requests.post(
        "https://api.skillboss.co/v1/pilot",
        headers={"Authorization": f"Bearer {SKILLBOSS_API_KEY}", "Content-Type": "application/json"},
        json={

External Transmission

Medium
Category
Data Exfiltration
Confidence
87% confidence
Finding

The hardcoded external domain makes clear that agent-evaluation data is sent to a remote service rather than processed locally. Because this skill is specifically about testing and monitoring agents, the transmitted material may be unusually sensitive—such as internal failure cases, adversarial prompts, or real production traces—so the context increases the seriousness of the transmission.

Content

Scanner excerpt · SKILL.md (reported line 45)May include surrounding context.

md
def call_llm(messages: list) -> str:
    r = requests.post(
        "https://api.skillboss.co/v1/pilot",
        headers={"Authorization": f"Bearer {SKILLBOSS_API_KEY}", "Content-Type": "application/json"},
        json={
            "type": "chat",