Back to skill

Security audit

agent-evaluation

Security checks for vulnerabilities and agentic risk

Overview

The skill is coherent for agent evaluation, but it routes evaluation messages to a fixed third-party API without enough privacy or consent guidance.

Review this skill carefully before installing if your evaluations may contain customer data, production traces, proprietary prompts, benchmark sets, credentials, or regulated information. Use it only with data approved for processing by SkillBoss/heybossai, and prefer redaction or a local/approved endpoint for sensitive evaluations.

Vulnerability Patterns
  • Skill Instruction HijackingAlters the agent's session goals or safety constraints when the skill loads
  • Agent Memory PoisoningWrites attacker-controlled rules into memory that affect later sessions
  • Remote Payload Retrieval and ExecutionFetches external code whose behavior can change after review
  • Embedded Malicious CodeShips malicious scripts inside the skill and executes them locally
  • Unauthorized Access and Privilege EscalationObtains permissions beyond the task's legitimate needs
Findings (1)

other

Warning
Location
skill.md:38
Finding

Unrestricted Transmission of Evaluation Messages to a Third-Party API

Content
View full analysis
str: r = requests.post( "https://api.heybossai.com/v1/pilot", headers={"Authorization": f"Bearer {SKILLBOSS_API_KEY}", "Content-Type": "application/json"}, json={ "type": "chat", "inputs": {"messages": messages}, "prefer": "balanced" }, timeout=60, ) return r.json()["result"]["choices"][0]["message"]["content"] ``` ### Technical Analysis The `call_llm` function sends the complete caller-provided `messages` object to the fixed third-party endpoint `https://api.heybossai.com/v1/pilot`. It does not classify, minimize, filter, or redact the content before transmission. Agent-evaluation messages may contain system prompts, production conversations, customer information, proprietary benchmark material, credentials, access tokens, or other confidential data. Although remote LLM access is relevant to the Skill's declared evaluation functionality and the destination is disclosed, requiring all calls to use this external endpoint exceeds minimum privilege when local or user-selected processing could perform the same function. The Skill also provides no explicit consent control, retention-policy notice, or safeguard preventing production data from being submitted. The API key is obtained from an environment variable and transmitted as a bearer token over HTTPS to its intended endpoint. There is no evidence that the key itself is stolen or sent to an unrelated destination. The finding concerns unrestricted disclosure of the `messages` payload. The function additionally fails to call `raise_for_status()` or validate the response schema before accessing nested fields. This can cause availability or reliability failures, ...[truncated 1912 chars]
Remediation
View remediation
Vulnerability Patterns
  • Data ExfiltrationExternal Transmission, Env Variable Harvesting, File System Enumeration
  • Prompt InjectionInstruction Override, Hidden Instructions, Exfiltration Commands
  • Privilege EscalationExcessive Permissions, Sudo/Root Execution, Credential Access
  • Supply ChainUnpinned Dependencies, External Script Fetching, Obfuscated Code
  • Excessive AgencyUnrestricted Tool Access, Autonomous Decision Making, Scope Creep
Findings (4)

Missing User Warnings

Medium
Category
Not specified by scanner
Confidence
94% confidence
Finding

The example code explicitly reads an API key from the environment and sends arbitrary messages to a third-party HTTPS endpoint, but the skill does not clearly warn users that prompt contents and possibly sensitive evaluation data will leave their environment. In an agent-evaluation context, those messages may contain proprietary prompts, benchmark corpora, or production traces, so the omission can lead to unintended data disclosure even if the code is presented as a normal integration example.

Content

No source excerpt is available for this finding.

External Transmission

Medium
Category
Data Exfiltration
Confidence
90% confidence
Finding

The requests.post(...) invocation transmits both authentication material and prompt content to an external endpoint, which creates a real confidentiality boundary crossing. In this skill's context, evaluation workflows often handle sensitive model inputs, failure logs, or internal datasets, so the lack of guardrails increases the chance of unintentional exposure.

Content

Scanner excerpt · skill.md (reported line 41)May include surrounding context.

md
SKILLBOSS_API_KEY = os.environ["SKILLBOSS_API_KEY"]

def call_llm(messages: list) -> str:
    r = requests.post(
        "https://api.heybossai.com/v1/pilot",
        headers={"Authorization": f"Bearer {SKILLBOSS_API_KEY}", "Content-Type": "application/json"},
        json={

External Transmission

Medium
Category
Data Exfiltration
Confidence
90% confidence
Finding

The requests.post(...) invocation transmits both authentication material and prompt content to an external endpoint, which creates a real confidentiality boundary crossing. In this skill's context, evaluation workflows often handle sensitive model inputs, failure logs, or internal datasets, so the lack of guardrails increases the chance of unintentional exposure.

Content

Scanner excerpt · skill.md (reported line 41)May include surrounding context.

md
SKILLBOSS_API_KEY = os.environ["SKILLBOSS_API_KEY"]

def call_llm(messages: list) -> str:
    r = requests.post(
        "https://api.heybossai.com/v1/pilot",
        headers={"Authorization": f"Bearer {SKILLBOSS_API_KEY}", "Content-Type": "application/json"},
        json={

External Transmission

Medium
Category
Data Exfiltration
Confidence
88% confidence
Finding

The hardcoded remote host https://api.heybossai.com/v1/pilot confirms that the skill routes content to an external service. Hardcoding a third-party endpoint is not inherently malicious, but without accompanying privacy/security disclosure it can mislead users into pasting sensitive evaluation material into a workflow that exports it outside their trust boundary.

Content

Scanner excerpt · skill.md (reported line 42)May include surrounding context.

md
def call_llm(messages: list) -> str:
    r = requests.post(
        "https://api.heybossai.com/v1/pilot",
        headers={"Authorization": f"Bearer {SKILLBOSS_API_KEY}", "Content-Type": "application/json"},
        json={
            "type": "chat",