EvalScope

• Outcome: Review
• Latest audit: Sep 15, 2026 · 1:09 AM UTC
• Version: 1.0.2

Overview

The skill is a coherent EvalScope command guide, but several examples can expose evaluation data, secrets, or local execution surfaces without enough warning or scoping.

Review this skill before installing. Use a virtual environment or container, pin package versions where possible, avoid the broad all extra unless needed, keep dashboards bound to localhost unless you deliberately expose them, do not paste real API keys into command lines, and use local models or redacted data for confidential RAG, embedding, image, or judge evaluations. Be especially cautious before running Docker sandbox benchmarks on a machine with sensitive files or credentials.

A.I.G

Vulnerability Patterns

• Insecure Dependencies: Introduces malicious components through unsafe dependency sources
• Insecure Skill Coding Practices: Finds exploitable flaws such as hardcoded secrets or command injection
• Skill Instruction Hijacking: Alters the agent's session goals or safety constraints when the skill loads
• Agent Memory Poisoning: Writes attacker-controlled rules into memory that affect later sessions
• Remote Payload Retrieval and Execution: Fetches external code whose behavior can change after review

Findings (2)

T08 · Insecure Dependencies

Warning

• Location: SKILL.md:35

Finding

Unpinned Third-Party Dependency Installation

Content

View full analysis

Vulnerability Details

File Locations:

• SKILL.md:35-39
• SKILL.md:134
• eval-reference.md:74-76
• rag-reference.md:5
• troubleshooting.md:52

Vulnerability Type: Unpinned and unverified third-party dependencies
Risk Level: Medium

Vulnerable Code

bash

pip install evalscope
pip install 'evalscope[all]'
pip install 'evalscope[perf]'
pip install 'evalscope[rag]'
pip install 'evalscope[service]'

Other documented installation commands include:

bash

pip install 'evalscope[opencompass]'
pip install 'evalscope[vlmeval]'
pip install 'evalscope[rag]'

Technical Analysis

The Skill instructs users or an Agent to install EvalScope and several optional dependency sets from the active Python package index without version constraints, package hashes, a lockfile, or an explicitly trusted repository. The broad evalscope[all] extra also expands the number of transitive packages trusted during installation.

Because these package references are mutable, the code installed when the instructions are followed may differ from the code reviewed when the Skill was audited. Python packages and their build systems may execute code during installation or later import. A compromised upstream release, compromised transitive dependency, or malicious package supplied by an untrusted configured index could therefore result in arbitrary code execution.

No evidence establishes that the named packages are currently malicious. The issue is the absence of supply-chain integrity controls in the documented installation process.

Attack Path

1. An attacker compromises an upstream package, one of its transitive dependencies, or a Python package index configured in the execution environment.
2. The attacker publishes a malicious release that satisfies the unconstrained package requirement.
3. A user or Agent follows the Skill instructions and runs an unpinned pip install command.
4. Pip downloads the mutable package and dependency set without va ...[truncated 824 chars]

Remediation

View remediation

Remediation Suggestions

1. Pin EvalScope and every optional package set to reviewed versions.
2. Maintain a lockfile containing resolved transitive versions and cryptographic hashes.
3. Install with hash verification, such as pip install --require-hashes -r requirements.txt.
4. Configure and document an explicitly trusted package index rather than relying on the environment's active index configuration.
5. Install packages inside a dedicated virtual environment or restricted container.
6. Avoid the broad [all] extra unless every included backend is required; install only the minimum necessary feature set.
7. Regularly scan locked dependencies for known vulnerabilities and review version changes before updating.
8. Avoid running package installation as root or from an account holding production credentials.

T09 · Insecure Skill Coding Practices

Warning

• Location: SKILL.md:129

Finding

Evaluation Dashboard Exposed on All Network Interfaces

Content

View full analysis

Vulnerability Details

File Location: SKILL.md:129-132
Vulnerability Type: Insecure network service exposure
Risk Level: Medium

Vulnerable Code

bash

evalscope service --host 0.0.0.0 --port 9000 --outputs ./outputs

The associated option description reinforces this configuration:

text

Options: --host (default 0.0.0.0), --port (default 9000), --outputs PATH (scan dir), --debug.

Technical Analysis

The visualization command binds the EvalScope service to 0.0.0.0, causing it to listen on every available IPv4 network interface rather than only the local loopback interface. The Skill does not document authentication, TLS, firewall restrictions, or a warning that the dashboard may become remotely accessible.

The service scans and presents content from ./outputs. Evaluation outputs may contain model prompts, generated responses, benchmark samples, endpoint information, logs, and other operational artifacts. Exposing the service to an untrusted network can therefore disclose information beyond what is required for local visualization.

The audit did not inspect EvalScope's external service implementation and does not establish that it lacks built-in authentication. The confirmed configuration weakness is that the supplied command unnecessarily exposes the service to all interfaces without documenting compensating controls.

Attack Path

1. A user follows the visualization instructions and starts the service with --host 0.0.0.0.
2. The process listens on port 9000 across all network interfaces.
3. A remote party with network access to the host discovers or directly connects to port 9000.
4. If no external or built-in access control blocks the request, the party accesses the dashboard and the evaluation artifacts exposed from ./outputs.
5. The party can inspect any sensitive prompts, model responses, reports, or metadata made available by the service.

Impact Assessment

The primary impact is unauthorized netw ...[truncated 490 chars]

Remediation

View remediation

Remediation Suggestions

1. Change the default example to bind only to loopback:

bash

evalscope service --host 127.0.0.1 --port 9000 --outputs ./outputs

2. Require explicit user confirmation before binding to 0.0.0.0 or another non-loopback interface.
3. If remote access is necessary, place the service behind an authenticated reverse proxy with TLS.
4. Restrict port 9000 through host and network firewalls to specifically authorized clients.
5. Verify whether the EvalScope service supports authentication and enable it before remote exposure.
6. Use a dedicated output directory containing only artifacts intended for dashboard access.
7. Review and redact prompts, responses, credentials, endpoint details, and sensitive logs before serving the directory.
8. Avoid --debug in remotely reachable deployments because verbose diagnostics may expose additional sensitive information.

SkillSpector

Vulnerability Patterns

• Data Exfiltration: External Transmission, Env Variable Harvesting, File System Enumeration
• Excessive Agency: Unrestricted Tool Access, Autonomous Decision Making, Scope Creep
• Prompt Injection: Instruction Override, Hidden Instructions, Exfiltration Commands
• Privilege Escalation: Excessive Permissions, Sudo/Root Execution, Credential Access
• Supply Chain: Unpinned Dependencies, External Script Fetching, Obfuscated Code

Findings (15)

External Model or Provider Selection

High

• Category: Excessive Agency
• Confidence: 90% confidence

Finding

Skill selects an external model or provider that may use a different account or billing plan than the operator expects. Undisclosed model switches can cause unexpected cost or quota consumption.

Content

Scanner excerpt · SKILL.md (reported line 76)May include surrounding context.

--api-url http://localhost:8000/v1/chat/completions --api-key sk-xxx --limit 10

Anthropic

evalscope eval --model claude-3-5-sonnet --eval-type anthropic_api --datasets mmlu --api-key sk-ant-xxx

text

Key parameters: `--datasets`, `--limit`, `--generation-config`, `--dataset-args`, `--eval-backend`, `--judge-strategy`. For full parameter list → [eval-reference.md](eval-reference.md).

Missing User Warnings

Medium

• Category: Not specified by scanner
• Confidence: 96% confidence

Finding

The visualization workflow explicitly binds the service to 0.0.0.0, exposing the dashboard on all network interfaces. In a skill that translates user requests into runnable commands, omitting a warning or safer localhost default can unintentionally publish evaluation results and any accessible dashboard functionality to other hosts on the network.

Content

No source excerpt is available for this finding.

Missing User Warnings

Medium

• Category: Not specified by scanner
• Confidence: 93% confidence

Finding

The sandbox evaluation section describes Docker-based execution for code benchmarks but does not warn that benchmark tasks may execute untrusted model-generated or benchmark-supplied code and require access to the Docker daemon. Even with container isolation, this materially increases attack surface and can surprise users into enabling powerful local execution capabilities.

Content

No source excerpt is available for this finding.

Missing User Warnings

Medium

• Category: Not specified by scanner
• Confidence: 96% confidence

Finding

The example command includes an inline API key field inside --judge-model-args, which encourages unsafe copy/paste practices. Passing secrets on the command line can expose them to shell history, observability tooling, logs, and local process inspection, making accidental leakage plausible.

Content

No source excerpt is available for this finding.

External Transmission

Medium

• Category: Data Exfiltration
• Confidence: 50% confidence

Finding

Data is being sent to an external URL. This could be legitimate telemetry or data exfiltration. Manual review is recommended.

Content

Scanner excerpt · eval-reference.md (reported line 130)May include surrounding context.

--datasets arena_hard
--api-url http://localhost:8000/v1/chat/completions
--judge-strategy llm
--judge-model-args '{"model": "gpt-4", "api_url": "https://api.openai.com/v1/chat/completions", "api_key": "sk-xxx"}'

text

## Cache and Output

External Transmission

Medium

• Category: Data Exfiltration
• Confidence: 60% confidence

Finding

Data is being sent to an external URL. This could be legitimate telemetry or data exfiltration. Manual review is recommended.

Content

Scanner excerpt · examples.md (reported line 125)May include surrounding context.

Steps:

1. Verify endpoint manually:

bash

curl -s http://localhost:8000/v1/models
curl -X POST http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "qwen-plus", "messages": [{"role": "user", "content": "hi"}]}'

Missing User Warnings

Medium

• Category: Not specified by scanner
• Confidence: 95% confidence

Finding

The RAGAS examples configure API-based LLM endpoints for testset generation and evaluation but do not warn that documents, generated questions, answers, and retrieved context may be transmitted to the configured HTTP service. In a RAG evaluation skill, users are especially likely to test with proprietary corpora, so omission of this disclosure can lead to unintentional exfiltration of sensitive data to local proxies or third-party providers.

Content

No source excerpt is available for this finding.

Missing User Warnings

Medium

• Category: Not specified by scanner
• Confidence: 94% confidence

Finding

The API embedding example sends user text to a third-party embedding endpoint but does not disclose that input content will leave the local environment. Because embedding evaluation commonly uses internal queries and documents, this can expose confidential text to an external provider without informed user consent.

Content

No source excerpt is available for this finding.

Missing User Warnings

Medium

• Category: Not specified by scanner
• Confidence: 93% confidence

Finding

The multimodal API example lacks notice that images and prompts are sent to the configured endpoint. In multimodal retrieval or caption benchmarks, images may contain proprietary visuals or personal data, so silent remote transmission creates a meaningful privacy and data-handling risk.

Content

No source excerpt is available for this finding.

Missing User Warnings

Low

• Category: Not specified by scanner
• Confidence: 84% confidence

Finding

The reference documents an --api-key parameter with a placeholder default and no warning about using real secrets safely. In a CLI context, users commonly copy examples directly, and absent guidance can lead to credential exposure via shell history, process lists, screenshots, or checked-in scripts.

Content

No source excerpt is available for this finding.

Missing User Warnings

Low

• Category: Not specified by scanner
• Confidence: 79% confidence

Finding

This markdown file documents use of --api-key but provides no caution about passing secrets on the command line, which can expose credentials via shell history or process listings. Under the markdown-specific missing-warning rule, this is a privacy-sensitive behavior that should be accompanied by a brief warning or safer alternative.

Content

No source excerpt is available for this finding.

Missing User Warnings

Low

• Category: Not specified by scanner
• Confidence: 73% confidence

Finding

The --headers parameter allows arbitrary HTTP headers, which commonly include authentication tokens or other sensitive metadata, but the documentation does not warn users about transmitting secrets to remote endpoints. For a markdown skill description, this omission matters because it affects user privacy and external data transmission.

Content

No source excerpt is available for this finding.

Missing User Warnings

Low

• Category: Not specified by scanner
• Confidence: 82% confidence

Finding

The documentation advertises wandb, swanlab, and clearml integrations and lists API key parameters, but it does not warn that using these services may upload run metadata or benchmark results to third-party platforms. This is a user-data and privacy-relevant behavior that should be disclosed in markdown documentation.

Content

No source excerpt is available for this finding.

Natural-Language Policy Violations

Low

• Category: Not specified by scanner
• Confidence: 86% confidence

Finding

The examples explicitly set language to chinese, which is a locale-specific choice presented as the default usage pattern in the reference. Because the file does not offer an opt-in choice or explain a region-specific requirement, this is a natural-language locale policy concern under the stated rules.

Content

No source excerpt is available for this finding.

Natural-Language Policy Violations

Low

• Category: Not specified by scanner
• Confidence: 85% confidence

Finding

This example again hard-codes language as chinese, reinforcing a single-language expectation without documenting that it is merely illustrative. Under the policy rule, forcing a specific language or locale without opt-in or justification is reportable.

Content

No source excerpt is available for this finding.