Benchmark Store

• Outcome: Review
• Latest audit: Sep 16, 2026 · 4:44 AM UTC
• Version: 1.1.1

Overview

The skill is mostly a benchmark/evaluation utility, but it includes under-disclosed scoring and untrusted execution paths with weak containment.

Install only in a controlled benchmark environment. Do not use this as a trusted hidden-test or untrusted-skill execution runner until it uses vetted authenticated encryption, enforces timeouts and sandboxing, resolves dependencies from a pinned trusted location, and clearly documents its scoring/deletion behavior.

A.I.G

Vulnerability Patterns

• Insecure Dependencies: Introduces malicious components through unsafe dependency sources
• Insecure Skill Coding Practices: Finds exploitable flaws such as hardcoded secrets or command injection
• Skill Instruction Hijacking: Alters the agent's session goals or safety constraints when the skill loads
• Agent Memory Poisoning: Writes attacker-controlled rules into memory that affect later sessions
• Remote Payload Retrieval and Execution: Fetches external code whose behavior can change after review

Findings (4)

T08 · Insecure Dependencies

Error

• Location: scripts/pareto.py:8
• Finding: Unsafe External Module Resolution Enables Dependency Hijacking<![CDATA[ ## Vulnerability Details **File Location**: `scripts/pareto.py`, lines 8–12 **Vulnerability Type**: Unsafe Python import path manipulation **Risk Level**: High ### Vulnerable Code ```python _REPO_ROOT = Path(__file__).resolve().parents[3] if str(_REPO_ROOT) not in sys.path: sys.path.insert(0, str(_REPO_ROOT)) from lib.common import read_json, write_json, utc_now_iso ``` ### Technical Analysis The script calculates a high-level ancestor directory, places it at the beginning of `sys.path`, and imports `lib.common` from that location. In the audited layout, `parents[3]` resolves outside the Skill artifact, potentially as broadly as `/tmp`. The audited artifact does not include `lib.common`. Consequently, the effective implementation of `read_json`, `write_json`, and `utc_now_iso` depends on files available in the surrounding runtime environment. Python executes top-level module code when importing a module. If an attacker can create a `lib/common.py` file in the prepended directory, importing `scripts/pareto.py` can execute attacker-controlled Python code before any Pareto operation occurs. Placing a broad, potentially shared or writable directory first in `sys.path` creates a dependency-hijacking boundary. The tests reinforce this external dependency by similarly modifying `sys.path` and importing from `lib`, rather than verifying a dependency bundled with the artifact. ### Attack Path 1. The attacker obtains write access to the ancestor directory inserted into `sys.path`. 2. The attacker creates a package structure such as: ```text lib/ __init__.py common.py ``` 3. The attacker places executable Python statements in `lib/common.py`. 4. A user, evaluator, or Agent imports or invokes `scripts/pareto.py`. 5. Python resolves `lib.common` from the attacker-controlled location. 6. Top-level code in the malicious module executes with the privileges of the invoking process. ### Impact Assessment Successful exploitation provi ...[truncated 601 chars]
• Remediation: <![CDATA[ ## Remediation Suggestions 1. Bundle `common` inside the audited package and use an explicit package-relative import: ```python from .common import read_json, write_json, utc_now_iso ``` 2. Remove all broad ancestor-directory insertion into `sys.path`. 3. Convert the project into a normal Python package with a defined package root. 4. If `lib.common` must be an external dependency: - Declare it in a locked dependency manifest. - Pin the exact version and verify package hashes. - Install it into an isolated virtual environment. - Do not resolve it from shared temporary or working directories. 5. Add a test that verifies the resolved module path is within an approved package or virtual-environment directory: ```python import lib.common assert Path(lib.common.__file__).resolve().is_relative_to(APPROVED_PACKAGE_ROOT) ``` 6. Run the Skill with a restricted filesystem and environment so that untrusted users cannot place modules in any import-search directory. ]]>

T09 · Insecure Skill Coding Practices

Error

• Location: interfaces/hidden_tests.py:451
• Finding: Hidden Test Data Is Protected with Repeating-Key XOR and Unsalted Password Hashing<![CDATA[ ## Vulnerability Details **File Location**: `interfaces/hidden_tests.py`, lines 451–464 and 704–718 **Vulnerability Type**: Broken cryptographic protection of sensitive hidden-test data **Risk Level**: High ### Vulnerable Code ```python def _derive_key(self, password: str) -> bytes: """派生解密密钥""" # 简化实现：实际应使用 PBKDF2 或 Argon2 return hashlib.sha256(password.encode()).digest() def _decrypt(self, encrypted_data: bytes, salt: bytes) -> Any: """解密数据""" if self._decryption_key is None: raise RuntimeError("Test suite is locked. Call unlock() first.") # 简化实现：实际应使用 AES-GCM 或 ChaCha20-Poly1305 # 这里使用简单的 XOR 作为演示 key = self._decryption_key decrypted = bytes([b ^ key[i % len(key)] for i, b in enumerate(encrypted_data)]) return json.loads(decrypted.decode()) ``` ```python # 生成盐值 salt = secrets.token_bytes(16) # 派生密钥 key = hashlib.sha256(password.encode()).digest() # 加密数据 def encrypt(data: Any) -> bytes: json_data = json.dumps(data, default=str).encode() return bytes([b ^ key[i % len(key)] for i, b in enumerate(json_data)]) encrypted_input = encrypt(input_data) encrypted_expected = encrypt(expected_output) encrypted_validator = encrypt(validator) ``` ### Technical Analysis The implementation hashes the password once with SHA-256 and then uses the resulting bytes as a repeating XOR keystream. This construction does not provide secure encryption. Specific weaknesses include: - The generated salt is not used in key derivation. - SHA-256 is deliberately fast, making offline password guessing inexpensive. - The same key is reused for the input, expected output, and validator. - The same password produces the same keystream across different tests. - Repeating-key XOR leaks relationships between plaintexts because: ```text ciphertext1 XOR ciphertext2 = plaintext1 XOR plaintext2 ``` - JSON has predictable structure and characters, providing known-plaintext material that can help recover keystream byte ...[truncated 2061 chars]
• Remediation: <![CDATA[ ## Remediation Suggestions 1. Replace repeating-key XOR with authenticated encryption: - AES-256-GCM, or - ChaCha20-Poly1305. 2. Derive encryption keys using a password-hardening algorithm: - Argon2id is preferred. - PBKDF2-HMAC-SHA-256 or scrypt may be used with appropriately strong parameters. 3. Use the generated per-test salt in key derivation. 4. Generate a unique random nonce for every encrypted field. Never reuse a nonce with the same key. 5. Authenticate non-secret metadata as associated data, including: - Suite ID - Test ID - Test type - Category - Visibility - Version 6. Store the algorithm identifier, KDF parameters, salt, and nonce alongside each ciphertext. 7. Reject every authentication failure without attempting to parse the resulting bytes. 8. Use a versioned ciphertext format so existing insecure records can be migrated safely. 9. Clear decrypted material and keys as soon as practical, and avoid returning sensitive plaintext in exception messages. 10. Add cryptographic tests for: - Wrong-password rejection - Ciphertext modification rejection - Metadata modification rejection - Nonce uniqueness - Different ciphertext for identical plaintext encrypted twice ]]>

T09 · Insecure Skill Coding Practices

Error

• Location: interfaces/hidden_tests.py:466
• Finding: Untrusted Skill Execution Ignores the Declared Timeout and Lacks Isolation<![CDATA[ ## Vulnerability Details **File Location**: `interfaces/hidden_tests.py`, lines 466–505 **Vulnerability Type**: Missing execution timeout and sandbox boundary **Risk Level**: High ### Vulnerable Code ```python def run_test( self, test_id: str, skill: SkillUnderTest, timeout_ms: float = 30000.0, ) -> TestResult: """ 运行单个隐藏测试 Args: test_id: 测试 ID skill: 被测 Skill timeout_ms: 超时时间 (毫秒) Returns: 测试结果 """ if test_id not in self._tests: return TestResult( test_id=test_id, passed=False, score=0.0, details={"error": f"Test {test_id} not found"}, ) test = self._tests[test_id] try: # 解密测试数据 input_data = self._decrypt(test.encrypted_input, test.salt) expected_output = self._decrypt(test.encrypted_expected, test.salt) validator = self._decrypt(test.encrypted_validator, test.salt) # 验证完整性 if not test.verify_hash(input_data, expected_output): return TestResult( test_id=test_id, passed=False, score=0.0, details={"error": "Test integrity check failed"}, ) # 执行测试 import time start_time = time.time() actual_output = skill.execute(input_data) execution_time_ms = (time.time() - start_time) * 1000 ``` ### Technical Analysis The method accepts a `timeout_ms` value and documents it as an execution timeout, but never uses it to constrain `skill.execute()`. The tested Skill executes synchronously inside the evaluator process. A thread or elapsed-time check performed after `execute()` returns would not be sufficient because it could not stop a blocked or malicious execution safely. There is also no process, filesystem, network, CPU, or memory isolation around the tested Skill. This exceeds the minimum privilege needed to evaluate untrusted code becaus ...[truncated 1463 chars]
• Remediation: <![CDATA[ ## Remediation Suggestions 1. Execute each tested Skill in a separate subprocess or hardened container. 2. Enforce a wall-clock timeout from the parent process: - Wait only for `timeout_ms`. - Send graceful termination on timeout. - Forcefully kill the process if it does not stop. 3. Apply operating-system resource limits: - CPU time - Address-space or memory limit - Process and thread count - Open-file count - Output size 4. Deny network access by default unless a specific benchmark requires it. 5. Mount only required files and use a read-only filesystem where possible. 6. Run the worker under a dedicated, unprivileged account. 7. Do not expose decryption keys or expected outputs to the worker. Send only the test input to the worker and perform validation in the parent process. 8. Capture stdout and stderr with strict size limits. 9. Return a structured timeout or resource-limit result without including sensitive internal exceptions. 10. Add tests using Skills that: - Loop indefinitely - Allocate excessive memory - Spawn child processes - Produce unlimited output - Attempt filesystem and network access ]]>

T09 · Insecure Skill Coding Practices

Note

• Location: scripts/benchmark_db.py:329
• Finding: CLI Initializes an Unrequested Fixed Database Before Processing the Selected Database Path<![CDATA[ ## Vulnerability Details **File Location**: `scripts/benchmark_db.py`, lines 329–339 **Vulnerability Type**: Unintended filesystem write and inconsistent path handling **Risk Level**: Low ### Vulnerable Code ```python if __name__ == "__main__": # 如果是首次运行，加载默认基准测试 db_path = "benchmarks.db" if not os.path.exists(db_path): logger.info("数据库不存在，加载默认基准测试...") init_db(db_path) load_default_benchmarks(db_path) main() ``` ### Technical Analysis The script initializes the fixed relative path `benchmarks.db` before `main()` parses and uses the caller-provided `--db-path`. For example, a caller may invoke: ```bash python3 scripts/benchmark_db.py \ --action list \ --db-path /approved/location/project.db ``` If `./benchmarks.db` does not exist in the current working directory, the script first creates and populates that unrelated file. It then processes the requested path inside `main()`. Creating local SQLite benchmark databases is part of the declared functionality, but creating a second database that the caller did not select is unnecessary and violates least-surprise path handling. ### Attack Path 1. A user runs the CLI from a writable directory without an existing `benchmarks.db`. 2. The user supplies a different path through `--db-path`. 3. Before argument processing, the module checks only the fixed relative path. 4. The script creates and populates `./benchmarks.db`. 5. The CLI subsequently operates on the separately requested database. 6. The unintended file remains in the working directory and may later be mistaken for the authoritative benchmark database. ### Impact Assessment The issue can cause: - Unintended file creation in the current working directory - Conflicting or stale benchmark databases - Accidental use of the wrong benchmark data - Disk consumption and cleanup burden - Confusing audit trails and regression results The write remains constrained by the invoking process’s existing file ...[truncated 109 chars]
• Remediation: <![CDATA[ ## Remediation Suggestions 1. Parse command-line arguments before initializing any database. 2. Initialize only the path explicitly selected by `args.db_path`. 3. Remove import-time and pre-dispatch filesystem side effects. 4. Refactor startup logic as follows: ```python def main(): args = parse_args() init_db(args.db_path) # Dispatch the selected action using args.db_path. if __name__ == "__main__": main() ``` 5. If loading default benchmarks is desired, require an explicit action or option such as: ```text --initialize-defaults ``` 6. Resolve the selected path, display it before writing, and optionally require confirmation before populating a non-temporary database. 7. Add an integration test asserting that invocation with `--db-path /custom/path.db` does not create `./benchmarks.db`. ]]>

SkillSpector

Vulnerability Patterns

• Privilege Escalation: Excessive Permissions, Sudo/Root Execution, Credential Access
• Trigger Abuse: Overly Broad Trigger, Shadow Command Trigger, Keyword Baiting Trigger
• MCP Least Privilege: Underdeclared Capability, Wildcard Permission, Missing Permission Declaration
• MCP Tool Poisoning: Hidden Instructions, Unicode Deception, Parameter Description Injection
• Prompt Injection: Instruction Override, Hidden Instructions, Exfiltration Commands

Findings (33)

Tp4

High

• Category: MCP Tool Poisoning
• Confidence: 98% confidence
• Finding: Yes, this is a mismatch. The description presents a specialized evaluation/benchmark-analysis skill, but the actual code chunk is just a minimal stub that prints a message and exits. There is no evidence of any benchmark handling, score comparison, Pareto analysis, standards retrieval, or any other behavior aligned with the declared purpose. This is a materially different primary purpose: a placeholder/test script rather than the described benchmarking utility.

Tp4

High

• Category: MCP Tool Poisoning
• Confidence: 92% confidence
• Finding: 声明强调该技能适用于基准数据库初始化、历史基线对比、Pareto front 检查和质量标准查阅，并特别说明“不用于给候选打分”。但代码的核心功能恰恰是运行 benchmark：接受 evaluator，逐条执行测试用例，收集 BenchmarkResult，计算 pass_rate、avg_score、weighted_score、时间和 token 使用，并导出评估结果。这属于直接评估/打分能力，与声明中的限制相冲突。反过来，声明中提到的历史基线比较、Pareto front 分析、质量分级标准查询在代码中都没有出现。虽然“冻结基准数据/标准基准套件”这一部分与描述部分吻合，但整体主功能存在明显偏差，且触发场景中多个关键用途未被实现，因此应判定为描述与行为不匹配。

Tp4

High

• Category: MCP Tool Poisoning
• Confidence: 89% confidence
• Finding: 声明描述的是“基准数据库/历史基线/Pareto front/质量标准查询”类能力，偏向基准管理与结果比较分析；而代码并未实现基准数据库初始化、历史评分比较、Pareto front 分析或质量分级标准查询。相反，代码的核心功能是隐藏测试套件管理与执行，对 skill 进行测试并产出分数和报告，这与声明中“ 不用于给候选打分 ”也明显不一致。虽然隐藏测试可作为评估基础设施的一部分，属于相关领域，但这里的主功能和具体能力与声明相比存在实质偏差，因此应判定为描述与行为不匹配。

Tp4

High

• Category: MCP Tool Poisoning
• Confidence: 82% confidence
• Finding: 描述与代码部分吻合：代码确实会初始化基准数据库、维护基准测试数据，并可将 skill 与基准进行比较。但存在明显不一致。第一，描述强调“不用于给候选打分”，而代码中的 compare_with_benchmark 会调用 evaluator、计算 overall_score，并持久化 eval_results，属于实际评分/评估能力。虽然对象是 skill 而非候选，但这仍是评分型行为。第二，代码提供 leaderboard 排行榜能力，这是实际功能，但描述中没有明确提及。第三，描述中提到“查看 Pareto front 是否有维度回退、或查阅质量分级标准”，但代码并没有 Pareto front 分析、维度回退检测，也没有质量分级标准查询逻辑；仅有若干评估字段和默认基准样例。整体来看，主旨接近“基准数据库工具”，但功能边界与声称用途不完全一致，因此应判定为存在描述—行为不匹配。

Credential Access

High

• Category: Privilege Escalation

Content

**测试用例**:
```
../../../etc/passwd
..\\..\\..\\Windows\\System32\\config\\SAM
/etc/shadow
C:\\Windows\\System32\\drivers\\etc\\hosts

• Confidence: 60% confidence
• Finding: Code accesses credential files (SSH keys, AWS credentials, etc.). This could indicate credential theft attempts.

Credential Access

High

• Category: Privilege Escalation

Content

**测试用例**:
```
../../../etc/passwd
..\\..\\..\\Windows\\System32\\config\\SAM
/etc/shadow
C:\\Windows\\System32\\drivers\\etc\\hosts

• Confidence: 60% confidence
• Finding: Code accesses credential files (SSH keys, AWS credentials, etc.). This could indicate credential theft attempts.

Credential Access

High

• Category: Privilege Escalation

Content

```
../../../etc/passwd
..\\..\\..\\Windows\\System32\\config\\SAM
/etc/shadow
C:\\Windows\\System32\\drivers\\etc\\hosts
```

• Confidence: 95% confidence
• Finding: Code accesses credential files (SSH keys, AWS credentials, etc.). This could indicate credential theft attempts.

Description-Behavior Mismatch

High

• Confidence: 97% confidence
• Finding: Manifest 说明该技能用于初始化基准数据库、对比历史基线、查看 Pareto front 回退和查阅质量分级标准，并明确不用于给候选打分或自动改进。但该文件实际维护的是覆盖 tool/process/analysis/creation/evaluation/red-team 等广泛类别的通用测试用例库，内容更像通用 skill-evaluator 的评测语料，而非仅限于 benchmark-store 的基准存储或基线对比数据。

Description-Behavior Mismatch

High

• Confidence: 98% confidence
• Finding: L230-L260 定义了“Skill 评估测试”，要求输出“等级、指标、建议的完整报告”，这已经是在构造面向技能能力评估的测试案例。Manifest 明确声明本技能不用于给候选打分，而应改用 improvement-discriminator，因此这里的测试内容与声明边界直接冲突。

Intent-Code Divergence

High

• Confidence: 99% confidence
• Finding: The implementation claims encrypted storage but uses a SHA-256 hash of the password directly as a repeating XOR keystream, which provides no authenticated encryption and is trivially reversible once plaintext structure is known. The salt is included in the object and hash computation but ignored during encryption/decryption, so it provides no cryptographic protection; this makes offline recovery and tampering of hidden tests much easier.

Lp3

Medium

• Category: MCP Least Privilege
• Confidence: 70% confidence
• Finding: Without declared permissions the skill's intent is opaque and cannot be validated.

Natural-Language Policy Violations

Medium

• Confidence: 95% confidence
• Finding: The description is written entirely in Chinese, and the file continues to present primary usage guidance in Chinese without any opt-in or indication that the skill is region-specific. This creates a language/locale policy issue because the skill implicitly enforces a specific language rather than offering a user choice.

Natural-Language Policy Violations

Medium

• Confidence: 83% confidence
• Finding: The document title and content are written as a normative standard in Chinese, and there is no indication that language selection is optional or that the standard is intended only for a Chinese-speaking audience. Under the policy, forcing a specific language without user opt-in or clear region-specific justification is a natural-language policy violation.

Vague Triggers

Medium

• Confidence: 95% confidence
• Finding: The usage guidance only says the skill is for a '测试场景' ('test scenario'), which is too broad to distinguish when the skill should or should not be invoked. The file does not provide specific trigger phrases, scope constraints, or negative examples to prevent unintended activation.

Natural-Language Policy Violations

Medium

• Confidence: 91% confidence
• Finding: The description and title are written in Chinese, and the file provides no indication that language choice is optional or that the skill is intentionally region-specific. Under the policy, forcing a specific language or locale without user opt-in can be a natural-language policy violation.

Natural-Language Policy Violations

Medium

• Confidence: 91% confidence
• Finding: The file’s user-facing instructions and descriptions are entirely in Chinese, and there is no indication that the skill is region-specific or that users may choose another language. Under the stated policy, forcing a specific language without opt-in is a natural-language policy violation.

Natural-Language Policy Violations

Medium

• Confidence: 95% confidence
• Finding: The description and title explicitly present the skill in Chinese ("测试样本", "生产就绪", "测试 Skill") with no indication that language is configurable or limited for a documented regional purpose. This creates a natural-language policy concern because the skill appears to enforce a specific language/locale without user opt-in.

Natural-Language Policy Violations

Medium

• Confidence: 94% confidence
• Finding: The entire guide is written in Chinese and includes no indication that other languages are supported or that Chinese is a required locale for a region-specific purpose. Per the policy, forcing a specific language without user opt-in or documented justification is a natural-language policy concern.

Natural-Language Policy Violations

Medium

• Confidence: 95% confidence
• Finding: This manifest file includes a test case whose input explicitly requires translating English into Chinese, and the assertion validates success by checking for Chinese characters in the output. Because the file is a general-purpose test case library rather than a clearly region-specific tool, this creates a natural-language locale constraint without opt-in or justification.

Intent-Code Divergence

Medium

• Confidence: 91% confidence
• Finding: 尾注将该测试用例库明确归因于“skill-evaluator”生成，而当前审计上下文中的技能是 benchmark-store。虽然这可能表示数据来源，但按当前文件归属来看，该说明会让读者理解为此处内容属于评测器职责，与 benchmark-store 的声明用途相矛盾，形成文档层面的意图混淆。

Natural-Language Policy Violations

Medium

• Confidence: 95% confidence
• Finding: The top-level docstring presents the skill/module description entirely in Chinese, which can impose a specific language on users or maintainers without opt-in. The file does not indicate that the locale is optional, configurable, or justified by a region-specific requirement.

Intent-Code Divergence

Medium

• Confidence: 88% confidence
• Finding: The code and documentation present these tests as 'hidden' and encrypted until execution, but the API can return full HiddenTest objects, including ciphertext and metadata, to any role permitted by the visibility boundary. In this benchmark/evaluation context, exposing the complete encrypted corpus materially increases the chance of offline analysis, test leakage, and overfitting against supposedly hidden tests, undermining evaluation integrity.

Natural-Language Policy Violations

Medium

• Confidence: 97% confidence
• Finding: The module docstring, argparse description, and help text are written exclusively in Chinese, which imposes a specific language on users of this script. Under the policy, locale or language restrictions should either provide user choice or be clearly documented as a justified region-specific constraint, neither of which appears here.

Description-Behavior Mismatch

Medium

• Confidence: 89% confidence
• Finding: The manifest description emphasizes initializing benchmark databases, comparing scores to historical baselines, checking Pareto-front regressions, and consulting grading standards. Adding a CLI 'delete' action introduces destructive database mutation that is not mentioned in that stated scope, expanding behavior beyond a store/reference utility into record removal.

Natural-Language Policy Violations

Low

• Confidence: 80% confidence
• Finding: The file presents all instructions and examples in Chinese and does not state that the skill is China/Chinese-specific or offer any language choice. Per SQP-3, forcing a specific language without user opt-in can be a natural-language policy violation when no justification is provided.

Static analysis

No suspicious patterns detected.