EvalScope

• Outcome: Review
• Latest audit: Sep 15, 2026 · 1:09 AM UTC
• Version: 1.0.2

Overview

The skill is a coherent EvalScope command guide, but several examples can expose evaluation data, secrets, or local execution surfaces without enough warning or scoping.

Review this skill before installing. Use a virtual environment or container, pin package versions where possible, avoid the broad all extra unless needed, keep dashboards bound to localhost unless you deliberately expose them, do not paste real API keys into command lines, and use local models or redacted data for confidential RAG, embedding, image, or judge evaluations. Be especially cautious before running Docker sandbox benchmarks on a machine with sensitive files or credentials.

A.I.G

Vulnerability Patterns

• Insecure Dependencies: Introduces malicious components through unsafe dependency sources
• Insecure Skill Coding Practices: Finds exploitable flaws such as hardcoded secrets or command injection
• Skill Instruction Hijacking: Alters the agent's session goals or safety constraints when the skill loads
• Agent Memory Poisoning: Writes attacker-controlled rules into memory that affect later sessions
• Remote Payload Retrieval and Execution: Fetches external code whose behavior can change after review

Findings (2)

T08 · Insecure Dependencies

Warning

• Location: SKILL.md:35
• Finding: Unpinned Third-Party Dependency Installation## Vulnerability Details **File Locations**: - `SKILL.md:35-39` - `SKILL.md:134` - `eval-reference.md:74-76` - `rag-reference.md:5` - `troubleshooting.md:52` **Vulnerability Type**: Unpinned and unverified third-party dependencies **Risk Level**: Medium ### Vulnerable Code ```bash pip install evalscope pip install 'evalscope[all]' pip install 'evalscope[perf]' pip install 'evalscope[rag]' pip install 'evalscope[service]' ``` Other documented installation commands include: ```bash pip install 'evalscope[opencompass]' pip install 'evalscope[vlmeval]' pip install 'evalscope[rag]' ``` ### Technical Analysis The Skill instructs users or an Agent to install EvalScope and several optional dependency sets from the active Python package index without version constraints, package hashes, a lockfile, or an explicitly trusted repository. The broad `evalscope[all]` extra also expands the number of transitive packages trusted during installation. Because these package references are mutable, the code installed when the instructions are followed may differ from the code reviewed when the Skill was audited. Python packages and their build systems may execute code during installation or later import. A compromised upstream release, compromised transitive dependency, or malicious package supplied by an untrusted configured index could therefore result in arbitrary code execution. No evidence establishes that the named packages are currently malicious. The issue is the absence of supply-chain integrity controls in the documented installation process. ### Attack Path 1. An attacker compromises an upstream package, one of its transitive dependencies, or a Python package index configured in the execution environment. 2. The attacker publishes a malicious release that satisfies the unconstrained package requirement. 3. A user or Agent follows the Skill instructions and runs an unpinned `pip install` command. 4. Pip downloads the mutable package and dependency set without va ...[truncated 824 chars]
• Remediation: ## Remediation Suggestions 1. Pin EvalScope and every optional package set to reviewed versions. 2. Maintain a lockfile containing resolved transitive versions and cryptographic hashes. 3. Install with hash verification, such as `pip install --require-hashes -r requirements.txt`. 4. Configure and document an explicitly trusted package index rather than relying on the environment's active index configuration. 5. Install packages inside a dedicated virtual environment or restricted container. 6. Avoid the broad `[all]` extra unless every included backend is required; install only the minimum necessary feature set. 7. Regularly scan locked dependencies for known vulnerabilities and review version changes before updating. 8. Avoid running package installation as root or from an account holding production credentials.

T09 · Insecure Skill Coding Practices

Warning

• Location: SKILL.md:129
• Finding: Evaluation Dashboard Exposed on All Network Interfaces## Vulnerability Details **File Location**: `SKILL.md:129-132` **Vulnerability Type**: Insecure network service exposure **Risk Level**: Medium ### Vulnerable Code ```bash evalscope service --host 0.0.0.0 --port 9000 --outputs ./outputs ``` The associated option description reinforces this configuration: ```text Options: --host (default 0.0.0.0), --port (default 9000), --outputs PATH (scan dir), --debug. ``` ### Technical Analysis The visualization command binds the EvalScope service to `0.0.0.0`, causing it to listen on every available IPv4 network interface rather than only the local loopback interface. The Skill does not document authentication, TLS, firewall restrictions, or a warning that the dashboard may become remotely accessible. The service scans and presents content from `./outputs`. Evaluation outputs may contain model prompts, generated responses, benchmark samples, endpoint information, logs, and other operational artifacts. Exposing the service to an untrusted network can therefore disclose information beyond what is required for local visualization. The audit did not inspect EvalScope's external service implementation and does not establish that it lacks built-in authentication. The confirmed configuration weakness is that the supplied command unnecessarily exposes the service to all interfaces without documenting compensating controls. ### Attack Path 1. A user follows the visualization instructions and starts the service with `--host 0.0.0.0`. 2. The process listens on port `9000` across all network interfaces. 3. A remote party with network access to the host discovers or directly connects to port `9000`. 4. If no external or built-in access control blocks the request, the party accesses the dashboard and the evaluation artifacts exposed from `./outputs`. 5. The party can inspect any sensitive prompts, model responses, reports, or metadata made available by the service. ### Impact Assessment The primary impact is unauthorized netw ...[truncated 490 chars]
• Remediation: ## Remediation Suggestions 1. Change the default example to bind only to loopback: ```bash evalscope service --host 127.0.0.1 --port 9000 --outputs ./outputs ``` 2. Require explicit user confirmation before binding to `0.0.0.0` or another non-loopback interface. 3. If remote access is necessary, place the service behind an authenticated reverse proxy with TLS. 4. Restrict port `9000` through host and network firewalls to specifically authorized clients. 5. Verify whether the EvalScope service supports authentication and enable it before remote exposure. 6. Use a dedicated output directory containing only artifacts intended for dashboard access. 7. Review and redact prompts, responses, credentials, endpoint details, and sensitive logs before serving the directory. 8. Avoid `--debug` in remotely reachable deployments because verbose diagnostics may expose additional sensitive information.

SkillSpector

Vulnerability Patterns

• Data Exfiltration: External Transmission, Env Variable Harvesting, File System Enumeration
• Excessive Agency: Unrestricted Tool Access, Autonomous Decision Making, Scope Creep
• Prompt Injection: Instruction Override, Hidden Instructions, Exfiltration Commands
• Privilege Escalation: Excessive Permissions, Sudo/Root Execution, Credential Access
• Supply Chain: Unpinned Dependencies, External Script Fetching, Obfuscated Code

Findings (15)

External Model or Provider Selection

High

• Category: Excessive Agency

Content

--api-url http://localhost:8000/v1/chat/completions --api-key sk-xxx --limit 10

# Anthropic
evalscope eval --model claude-3-5-sonnet --eval-type anthropic_api --datasets mmlu --api-key sk-ant-xxx
```

Key parameters: `--datasets`, `--limit`, `--generation-config`, `--dataset-args`, `--eval-backend`, `--judge-strategy`. For full parameter list → [eval-reference.md](eval-reference.md).

• Confidence: 90% confidence
• Finding: Skill selects an external model or provider that may use a different account or billing plan than the operator expects. Undisclosed model switches can cause unexpected cost or quota consumption.

Missing User Warnings

Medium

• Confidence: 96% confidence
• Finding: The visualization workflow explicitly binds the service to 0.0.0.0, exposing the dashboard on all network interfaces. In a skill that translates user requests into runnable commands, omitting a warning or safer localhost default can unintentionally publish evaluation results and any accessible dashboard functionality to other hosts on the network.

Missing User Warnings

Medium

• Confidence: 93% confidence
• Finding: The sandbox evaluation section describes Docker-based execution for code benchmarks but does not warn that benchmark tasks may execute untrusted model-generated or benchmark-supplied code and require access to the Docker daemon. Even with container isolation, this materially increases attack surface and can surprise users into enabling powerful local execution capabilities.

Missing User Warnings

Medium

• Confidence: 96% confidence
• Finding: The example command includes an inline API key field inside `--judge-model-args`, which encourages unsafe copy/paste practices. Passing secrets on the command line can expose them to shell history, observability tooling, logs, and local process inspection, making accidental leakage plausible.

External Transmission

Medium

• Category: Data Exfiltration

Content

--datasets arena_hard \
  --api-url http://localhost:8000/v1/chat/completions \
  --judge-strategy llm \
  --judge-model-args '{"model": "gpt-4", "api_url": "https://api.openai.com/v1/chat/completions", "api_key": "sk-xxx"}'
```

## Cache and Output

• Confidence: 50% confidence
• Finding: Data is being sent to an external URL. This could be legitimate telemetry or data exfiltration. Manual review is recommended.

External Transmission

Medium

• Category: Data Exfiltration

Content

**Steps**:
1. Verify endpoint manually:
```bash
curl -s http://localhost:8000/v1/models
curl -X POST http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "qwen-plus", "messages": [{"role": "user", "content": "hi"}]}'

• Confidence: 60% confidence
• Finding: Data is being sent to an external URL. This could be legitimate telemetry or data exfiltration. Manual review is recommended.

Missing User Warnings

Medium

• Confidence: 95% confidence
• Finding: The RAGAS examples configure API-based LLM endpoints for testset generation and evaluation but do not warn that documents, generated questions, answers, and retrieved context may be transmitted to the configured HTTP service. In a RAG evaluation skill, users are especially likely to test with proprietary corpora, so omission of this disclosure can lead to unintentional exfiltration of sensitive data to local proxies or third-party providers.

Missing User Warnings

Medium

• Confidence: 94% confidence
• Finding: The API embedding example sends user text to a third-party embedding endpoint but does not disclose that input content will leave the local environment. Because embedding evaluation commonly uses internal queries and documents, this can expose confidential text to an external provider without informed user consent.

Missing User Warnings

Medium

• Confidence: 93% confidence
• Finding: The multimodal API example lacks notice that images and prompts are sent to the configured endpoint. In multimodal retrieval or caption benchmarks, images may contain proprietary visuals or personal data, so silent remote transmission creates a meaningful privacy and data-handling risk.

Missing User Warnings

Low

• Confidence: 84% confidence
• Finding: The reference documents an `--api-key` parameter with a placeholder default and no warning about using real secrets safely. In a CLI context, users commonly copy examples directly, and absent guidance can lead to credential exposure via shell history, process lists, screenshots, or checked-in scripts.

Missing User Warnings

Low

• Confidence: 79% confidence
• Finding: This markdown file documents use of `--api-key` but provides no caution about passing secrets on the command line, which can expose credentials via shell history or process listings. Under the markdown-specific missing-warning rule, this is a privacy-sensitive behavior that should be accompanied by a brief warning or safer alternative.

Missing User Warnings

Low

• Confidence: 73% confidence
• Finding: The `--headers` parameter allows arbitrary HTTP headers, which commonly include authentication tokens or other sensitive metadata, but the documentation does not warn users about transmitting secrets to remote endpoints. For a markdown skill description, this omission matters because it affects user privacy and external data transmission.

Missing User Warnings

Low

• Confidence: 82% confidence
• Finding: The documentation advertises `wandb`, `swanlab`, and `clearml` integrations and lists API key parameters, but it does not warn that using these services may upload run metadata or benchmark results to third-party platforms. This is a user-data and privacy-relevant behavior that should be disclosed in markdown documentation.

Natural-Language Policy Violations

Low

• Confidence: 86% confidence
• Finding: The examples explicitly set `language` to `chinese`, which is a locale-specific choice presented as the default usage pattern in the reference. Because the file does not offer an opt-in choice or explain a region-specific requirement, this is a natural-language locale policy concern under the stated rules.

Natural-Language Policy Violations

Low

• Confidence: 85% confidence
• Finding: This example again hard-codes `language` as `chinese`, reinforcing a single-language expectation without documenting that it is merely illustrative. Under the policy rule, forcing a specific language or locale without opt-in or justification is reportable.

Static analysis

No suspicious patterns detected.