Back to skill

Security audit

Benchmark Model Provider

Security checks for vulnerabilities and agentic risk

Overview

The skill mostly matches its benchmarking purpose, but its runner can send any configured environment variable as an API bearer token to any configured HTTPS endpoint.

Install only if you will run benchmark specs you trust. Review every spec before execution, keep auth_env set to a dedicated low-privilege benchmark key such as BENCHMARK_API_KEY, avoid sensitive prompts unless the endpoint is approved, and pin dependencies before use in a controlled environment.

Vulnerability Patterns
  • Insecure DependenciesIntroduces malicious components through unsafe dependency sources
  • Insecure Skill Coding PracticesFinds exploitable flaws such as hardcoded secrets or command injection
  • Skill Instruction HijackingAlters the agent's session goals or safety constraints when the skill loads
  • Agent Memory PoisoningWrites attacker-controlled rules into memory that affect later sessions
  • Remote Payload Retrieval and ExecutionFetches external code whose behavior can change after review
Findings (2)

T09 · Insecure Skill Coding Practices

Error
Location
scripts/run_benchmark.py:53
Finding

Specification-Controlled Credential Disclosure to Arbitrary HTTPS Endpoints

Content
View full analysis
bool: """Basic safety checks. - Require https - Block raw IPs - Block localhost This is not a complete security solution, but it prevents the most common foot-guns that trigger security scanners and protects against accidental exfiltration to an IP/short endpoint. """ if not base_url: return False base_url = base_url.strip() if not base_url.startswith('https://'): return False lowered = base_url.lower() if 'localhost' in lowered or '127.0.0.1' in lowered or '0.0.0.0' in lowered: return False # crude IP check (blocks http(s)://...) import re if re.search(r'https://\d{1,3}(?:\.\d{1,3}){3}(?::\d+)?(?:/|$)', lowered): return False return True def run_single_model(spec: dict, model: str, run_dir: Path, run_id: str): base_url = spec.get('base_url') or os.environ.get('BENCHMARK_BASE_URL') or '' if not _is_safe_base_url(base_url): raise Syste ...[truncated 3243 chars]
Remediation
View remediation

T08 · Insecure Dependencies

Note
Location
requirements.txt:1
Finding

Unpinned Third-Party Dependencies

Content
View full analysis
Remediation
View remediation
Vulnerability Patterns
  • Data ExfiltrationExternal Transmission, Env Variable Harvesting, File System Enumeration
  • Excessive AgencyUnrestricted Tool Access, Autonomous Decision Making, Scope Creep
  • System Prompt LeakageDirect Leakage, Indirect Extraction, Tool-Based Exfiltration
  • MCP Least PrivilegeUnderdeclared Capability, Wildcard Permission, Missing Permission Declaration
  • MCP Tool PoisoningHidden Instructions, Unicode Deception, Parameter Description Injection
Findings (34)

Tp4

High
Category
MCP Tool Poisoning
Confidence
95% confidence
Finding

The declared description promises substantive benchmarking and model/provider evaluation features. The supplied code chunk only contains a placeholder frontend script with a console.log statement indicating a landing page template was loaded. This is materially different from the declared purpose, with none of the described core capabilities present in the code shown.

Content

No source excerpt is available for this finding.

Tp4

High
Category
MCP Tool Poisoning
Confidence
99% confidence
Finding

The declared description describes a full benchmarking and model-comparison capability, but the actual code chunk only implements a basic token estimation helper. It does not benchmark models, compare providers, generate prompt suites, rank outputs, or create reports. While token estimation could be a supporting utility in a larger benchmarking system, this code chunk by itself does not substantively perform the declared primary purpose, so the description materially overstates the behavior shown.

Content

No source excerpt is available for this finding.

Tp4

High
Category
MCP Tool Poisoning
Confidence
98% confidence
Finding

The declared description centers on benchmarking and ranking AI providers/models and generating benchmark reports. This code chunk does not implement any benchmarking, scoring, reranking, model comparison, or prompt-suite derivation. Instead, its primary function is post-processing/publication support for reports: copying a PDF to a published filename and emitting suggestions for hosting static HTML via services like Vercel, Netlify, Cloudflare Pages, and GitHub Pages. While report publication could be tangentially related to benchmark reporting, this chunk’s actual purpose is materially different from the declared benchmark-and-rank functionality, so it should be flagged as a mismatch.

Content

No source excerpt is available for this finding.

Tp4

High
Category
MCP Tool Poisoning
Confidence
97% confidence
Finding

The declared description promises a broad benchmark and ranking capability for AI models/providers, driven by user-specific prompts and producing comparative assessments and reports. The supplied code does none of that. It only maps model identifiers to input/output token pricing values using a built-in dictionary and returns fallback estimates when unknown. While pricing could be a supporting component of a benchmarking system, this code chunk by itself has a materially narrower and different primary purpose: pricing resolution. Therefore the description does not accurately represent this code chunk's actual behavior.

Content

No source excerpt is available for this finding.

Direct Prompt Extraction

High
Category
System Prompt Leakage
Confidence
85% confidence
Finding

Skill contains instructions that could directly expose system prompts, internal rules, or hidden instructions to users or external parties.

Content

Scanner excerpt · SKILL.md (reported line 84)May include surrounding context.

md
- Normalize model ids before calling the endpoint when the provider catalog exposes raw model ids but the user/runtime spec may contain provider-prefixed names.
- If the endpoint returns naming/provider mismatch errors, explain the mismatch clearly instead of leaving only a raw 502/unknown-provider error.

### Output rules
- Mark every estimated metric clearly.
- Rewrite reports/landing pages to the newest snapshot.
- Do **not** append patch fragments to stale output.

Undeclared Tool Scope

Medium
Category
MCP Least Privilege
Confidence
92% confidence
Finding

The skill advertises capabilities that imply environment access, file I/O, shell execution, and network use, but it does not declare an explicit tool scope such as permissions or allowed-tools. That creates an unclear trust boundary and increases the chance the skill is executed with broader privileges than intended, especially since it can send prompts to arbitrary configured endpoints and handle local artifacts.

Content

No source excerpt is available for this finding.

Natural-Language Policy Violations

Medium
Category
Not specified by scanner
Confidence
95% confidence
Finding

The template sets language: en directly, which imposes a specific language by default. Because there is no accompanying note that this is optional, user-selectable, or justified by a region-specific purpose, it creates a natural-language policy concern under the language/locale rule.

Content

No source excerpt is available for this finding.

Natural-Language Policy Violations

Medium
Category
Not specified by scanner
Confidence
90% confidence
Finding

The template hard-codes language: zh, which imposes a specific language/locale choice in the skill definition. The policy allows locale constraints only when they are optional, user-selected, or clearly justified as region-specific; no such opt-in or justification appears here.

Content

No source excerpt is available for this finding.

Natural-Language Policy Violations

Medium
Category
Not specified by scanner
Confidence
95% confidence
Finding

The template sets language: vi as a fixed default, which is a natural-language locale constraint. Because the file does not offer an explicit user choice or explain that the locale is optional or region-specific, it can violate the policy against forcing a specific language without opt-in.

Content

No source excerpt is available for this finding.

External Transmission

Medium
Category
Data Exfiltration
Confidence
50% confidence
Finding

Data is being sent to an external URL. This could be legitimate telemetry or data exfiltration. Manual review is recommended.

Content

Scanner excerpt · examples/boss-multidomain-daily-10q.yaml (reported line 6)May include surrounding context.

yaml
mode: agent_context
# IMPORTANT: use a trusted HTTPS OpenAI-compatible endpoint.
# For safety, the runner blocks localhost/raw IP and non-HTTPS URLs by default.
# Example: https://api.openai.com/v1  (or your own trusted gateway)
base_url: https://api.openai.com/v1
auth_env: BENCHMARK_API_KEY
context_profile:

External Transmission

Medium
Category
Data Exfiltration
Confidence
50% confidence
Finding

Data is being sent to an external URL. This could be legitimate telemetry or data exfiltration. Manual review is recommended.

Content

Scanner excerpt · examples/boss-multidomain-daily-10q.yaml (reported line 7)May include surrounding context.

yaml
mode: agent_context
# IMPORTANT: use a trusted HTTPS OpenAI-compatible endpoint.
# For safety, the runner blocks localhost/raw IP and non-HTTPS URLs by default.
# Example: https://api.openai.com/v1  (or your own trusted gateway)
base_url: https://api.openai.com/v1
auth_env: BENCHMARK_API_KEY
context_profile:

Natural-Language Policy Violations

Medium
Category
Not specified by scanner
Confidence
96% confidence
Finding

The manifest sets language: vi, which imposes a fixed language/locale preference in the skill configuration. Under the policy, forcing a specific language without user opt-in or a clearly documented region-specific justification is a natural-language policy violation.

Content

No source excerpt is available for this finding.

Natural-Language Policy Violations

Medium
Category
Not specified by scanner
Confidence
94% confidence
Finding

The titles and prompts are written entirely in Chinese, indicating a language-specific skill behavior, but the manifest does not document user opt-in for Chinese output or explain a justified locale restriction. This can violate language/locale policy when a skill forces a specific language without offering choice or documenting the constraint.

Content

No source excerpt is available for this finding.

External Transmission

Medium
Category
Data Exfiltration
Confidence
50% confidence
Finding

Data is being sent to an external URL. This could be legitimate telemetry or data exfiltration. Manual review is recommended.

Content

Scanner excerpt · examples/vietnam-market-5q.yaml (reported line 3)May include surrounding context.

yaml
name: vietnam-market-5q
version: v1
base_url: https://api.example.com/v1
auth_env: BENCHMARK_API_KEY
models:
  - Qwen3.5-35B

External Transmission

Medium
Category
Data Exfiltration
Confidence
50% confidence
Finding

Data is being sent to an external URL. This could be legitimate telemetry or data exfiltration. Manual review is recommended.

Content

Scanner excerpt · references/benchmark-schema.md (reported line 17)May include surrounding context.

md
name: vietnam-market-5q
version: v1
base_url: https://api.example.com/v1
auth_env: BENCHMARK_API_KEY
models:
  - Qwen3.5-35B

External Transmission

Medium
Category
Data Exfiltration
Confidence
50% confidence
Finding

Data is being sent to an external URL. This could be legitimate telemetry or data exfiltration. Manual review is recommended.

Content

Scanner excerpt · references/runtime-safety.md (reported line 19)May include surrounding context.

md
name: vietnam-market-5q
version: v1
base_url: https://api.example.com/v1
auth_env: BENCHMARK_API_KEY
models:
  - Qwen3.5-35B

Natural-Language Policy Violations

Medium
Category
Not specified by scanner
Confidence
96% confidence
Finding

The titles and prompts are entirely written in Vietnamese and implicitly require Vietnamese-language interaction, but the file provides no user choice, opt-in, or justification for a locale-specific restriction. This is a natural-language policy concern because the skill appears to enforce a specific language across its user-facing content.

Content

No source excerpt is available for this finding.

Autonomous Decision Making

Medium
Category
Excessive Agency
Confidence
75% confidence
Finding

Skill enables autonomous high-impact decisions without human-in-the-loop verification. Critical operations (destructive commands, financial transactions, data deletion) should require explicit user confirmation.

Content

Scanner excerpt · references/initial-project-spec.md (reported line 520)May include surrounding context.

md
- PDF
- both

Do not auto-deploy to Vercel without asking first at the delivery step.

---

Autonomous Decision Making

Medium
Category
Excessive Agency
Confidence
75% confidence
Finding

Skill enables autonomous high-impact decisions without human-in-the-loop verification. Critical operations (destructive commands, financial transactions, data deletion) should require explicit user confirmation.

Content

Scanner excerpt · references/output-modes.md (reported line 23)May include surrounding context.

md
- PDF
- both

Do not auto-deploy to Vercel without asking first at the delivery step.

---

Autonomous Decision Making

Medium
Category
Excessive Agency
Confidence
85% confidence
Finding

Skill enables autonomous high-impact decisions without human-in-the-loop verification. Critical operations (destructive commands, financial transactions, data deletion) should require explicit user confirmation.

Content

Scanner excerpt · references/initial-project-spec.md (reported line 520)May include surrounding context.

md
- PDF
- both

Do not auto-deploy to Vercel without asking first at the delivery step.

---

Autonomous Decision Making

Medium
Category
Excessive Agency
Confidence
85% confidence
Finding

Skill enables autonomous high-impact decisions without human-in-the-loop verification. Critical operations (destructive commands, financial transactions, data deletion) should require explicit user confirmation.

Content

Scanner excerpt · references/output-modes.md (reported line 23)May include surrounding context.

md
- PDF
- both

Do not auto-deploy to Vercel without asking first at the delivery step.

---

Natural-Language Policy Violations

Medium
Category
Not specified by scanner
Confidence
96% confidence
Finding

This Python file contains multiple user-facing natural-language strings for summaries and recommendations that are written only in Vietnamese. The policy allows locale constraints only when the skill offers user choice or clearly documents a justified regional limitation, neither of which is present here.

Content

No source excerpt is available for this finding.

Rp1

Medium
Category
MCP Rug Pull
Confidence
70% confidence
Finding

npx commands without a version suffix (e.g. @1.0.0) create a rug-pull risk if the upstream server is compromised and publishes a malicious update.

Content

No source excerpt is available for this finding.

Rp1

Medium
Category
MCP Rug Pull
Confidence
70% confidence
Finding

npx commands without a version suffix (e.g. @1.0.0) create a rug-pull risk if the upstream server is compromised and publishes a malicious update.

Content

No source excerpt is available for this finding.

Missing User Warnings

Medium
Category
Not specified by scanner
Confidence
95% confidence
Finding

The script sends full user-supplied prompts to external model endpoints via HTTP requests, but this file provides no explicit user-facing consent, warning, or data-classification guard before transmission. In a benchmarking skill, prompts may contain sensitive business context, proprietary test suites, or personal data, so silent exfiltration to third-party providers is a meaningful privacy and compliance risk.

Content

No source excerpt is available for this finding.

Static analysis

No suspicious patterns detected.