Security audit

Benchmark Store

Security checks for vulnerabilities and agentic risk

Overview

The skill is mostly a benchmark/evaluation utility, but it includes under-disclosed scoring and untrusted execution paths with weak containment.

Install only in a controlled benchmark environment. Do not use this as a trusted hidden-test or untrusted-skill execution runner until it uses vetted authenticated encryption, enforces timeouts and sandboxing, resolves dependencies from a pinned trusted location, and clearly documents its scoring/deletion behavior.

A.I.G

Vulnerability Patterns

Insecure DependenciesIntroduces malicious components through unsafe dependency sources
Insecure Skill Coding PracticesFinds exploitable flaws such as hardcoded secrets or command injection
Skill Instruction HijackingAlters the agent's session goals or safety constraints when the skill loads
Agent Memory PoisoningWrites attacker-controlled rules into memory that affect later sessions
Remote Payload Retrieval and ExecutionFetches external code whose behavior can change after review

Findings (4)

T08 · Insecure Dependencies

Error

Location: scripts/pareto.py:8
Finding: Unsafe External Module Resolution Enables Dependency Hijacking
Content: View full analysis
Remediation: View remediation

T09 · Insecure Skill Coding Practices

Error

Location: interfaces/hidden_tests.py:451
Finding: Hidden Test Data Is Protected with Repeating-Key XOR and Unsalted Password Hashing
Content: View full analysis
bytes: """派生解密密钥""" # 简化实现：实际应使用 PBKDF2 或 Argon2 return hashlib.sha256(password.encode()).digest() def _decrypt(self, encrypted_data: bytes, salt: bytes) -> Any: """解密数据""" if self._decryption_key is None: raise RuntimeError("Test suite is locked. Call unlock() first.") # 简化实现：实际应使用 AES-GCM 或 ChaCha20-Poly1305 # 这里使用简单的 XOR 作为演示 key = self._decryption_key decrypted = bytes([b ^ key[i % len(key)] for i, b in enumerate(encrypted_data)]) return json.loads(decrypted.decode()) ``` ```python # 生成盐值 salt = secrets.token_bytes(16) # 派生密钥 key = hashlib.sha256(password.encode()).digest() # 加密数据 def encrypt(data: Any) -> bytes: json_data = json.dumps(data, default=str).encode() return bytes([b ^ key[i % len(key)] for i, b in enumerate(json_data)]) encrypted_input = encrypt(input_data) encrypted_expected = encrypt(expected_output) encrypted_validator = encrypt(validator) ``` ### Technical Analysis The implementation hashes the password once with SHA-256 and then uses the resulting bytes as a repeating XOR keystream. This construction does not provide secure encryption. Specific weaknesses include: - The generated salt is not used in key derivation. - SHA-256 is deliberately fast, making offline password guessing inexpensive. - The same key is reused for the input, expected output, and validator. - The same password produces the same keystream across different tests. - Repeating-key XOR leaks relationships between plaintexts because: ```text ciphertext1 XOR ciphertext2 = plaintext1 XOR plaintext2 ``` - JSON has predictable structure and characters, providing known-plaintext material that can help recover keystream byte ...[truncated 2061 chars]
Remediation: View remediation

T09 · Insecure Skill Coding Practices

Error

Location: interfaces/hidden_tests.py:466
Finding: Untrusted Skill Execution Ignores the Declared Timeout and Lacks Isolation
Content: View full analysis
TestResult: """ 运行单个隐藏测试 Args: test_id: 测试 ID skill: 被测 Skill timeout_ms: 超时时间 (毫秒) Returns: 测试结果 """ if test_id not in self._tests: return TestResult( test_id=test_id, passed=False, score=0.0, details={"error": f"Test {test_id} not found"}, ) test = self._tests[test_id] try: # 解密测试数据 input_data = self._decrypt(test.encrypted_input, test.salt) expected_output = self._decrypt(test.encrypted_expected, test.salt) validator = self._decrypt(test.encrypted_validator, test.salt) # 验证完整性 if not test.verify_hash(input_data, expected_output): return TestResult( test_id=test_id, passed=False, score=0.0, details={"error": "Test integrity check failed"}, ) # 执行测试 import time start_time = time.time() actual_output = skill.execute(input_data) execution_time_ms = (time.time() - start_time) * 1000 ``` ### Technical Analysis The method accepts a `timeout_ms` value and documents it as an execution timeout, but never uses it to constrain `skill.execute()`. The tested Skill executes synchronously inside the evaluator process. A thread or elapsed-time check performed after `execute()` returns would not be sufficient because it could not stop a blocked or malicious execution safely. There is also no process, filesystem, network, CPU, or memory isolation around the tested Skill. This exceeds the minimum privilege needed to evaluate untrusted code becaus ...[truncated 1463 chars]
Remediation: View remediation

T09 · Insecure Skill Coding Practices

Note

Location: scripts/benchmark_db.py:329
Finding: CLI Initializes an Unrequested Fixed Database Before Processing the Selected Database Path
Content: View full analysis
Remediation: View remediation

SkillSpector

Vulnerability Patterns

Privilege EscalationExcessive Permissions, Sudo/Root Execution, Credential Access
Trigger AbuseOverly Broad Trigger, Shadow Command Trigger, Keyword Baiting Trigger
MCP Least PrivilegeUnderdeclared Capability, Wildcard Permission, Missing Permission Declaration
MCP Tool PoisoningHidden Instructions, Unicode Deception, Parameter Description Injection
Prompt InjectionInstruction Override, Hidden Instructions, Exfiltration Commands

Findings (33)

Tp4

High

Category: MCP Tool Poisoning
Confidence: 98% confidence
Finding: Yes, this is a mismatch. The description presents a specialized evaluation/benchmark-analysis skill, but the actual code chunk is just a minimal stub that prints a message and exits. There is no evidence of any benchmark handling, score comparison, Pareto analysis, standards retrieval, or any other behavior aligned with the declared purpose. This is a materially different primary purpose: a placeholder/test script rather than the described benchmarking utility.
Content: No source excerpt is available for this finding.

Tp4

High

Category: MCP Tool Poisoning
Confidence: 92% confidence
Finding: 声明强调该技能适用于基准数据库初始化、历史基线对比、Pareto front 检查和质量标准查阅，并特别说明“不用于给候选打分”。但代码的核心功能恰恰是运行 benchmark：接受 evaluator，逐条执行测试用例，收集 BenchmarkResult，计算 pass_rate、avg_score、weighted_score、时间和 token 使用，并导出评估结果。这属于直接评估/打分能力，与声明中的限制相冲突。反过来，声明中提到的历史基线比较、Pareto front 分析、质量分级标准查询在代码中都没有出现。虽然“冻结基准数据/标准基准套件”这一部分与描述部分吻合，但整体主功能存在明显偏差，且触发场景中多个关键用途未被实现，因此应判定为描述与行为不匹配。
Content: No source excerpt is available for this finding.

Tp4

High

Category: MCP Tool Poisoning
Confidence: 89% confidence
Finding: 声明描述的是“基准数据库/历史基线/Pareto front/质量标准查询”类能力，偏向基准管理与结果比较分析；而代码并未实现基准数据库初始化、历史评分比较、Pareto front 分析或质量分级标准查询。相反，代码的核心功能是隐藏测试套件管理与执行，对 skill 进行测试并产出分数和报告，这与声明中“ 不用于给候选打分 ”也明显不一致。虽然隐藏测试可作为评估基础设施的一部分，属于相关领域，但这里的主功能和具体能力与声明相比存在实质偏差，因此应判定为描述与行为不匹配。
Content: No source excerpt is available for this finding.

Tp4

High

Category: MCP Tool Poisoning
Confidence: 82% confidence
Finding: 描述与代码部分吻合：代码确实会初始化基准数据库、维护基准测试数据，并可将 skill 与基准进行比较。但存在明显不一致。第一，描述强调“不用于给候选打分”，而代码中的 compare_with_benchmark 会调用 evaluator、计算 overall_score，并持久化 eval_results，属于实际评分/评估能力。虽然对象是 skill 而非候选，但这仍是评分型行为。第二，代码提供 leaderboard 排行榜能力，这是实际功能，但描述中没有明确提及。第三，描述中提到“查看 Pareto front 是否有维度回退、或查阅质量分级标准”，但代码并没有 Pareto front 分析、维度回退检测，也没有质量分级标准查询逻辑；仅有若干评估字段和默认基准样例。整体来看，主旨接近“基准数据库工具”，但功能边界与声称用途不完全一致，因此应判定为存在描述—行为不匹配。
Content: No source excerpt is available for this finding.

Credential Access

High

Category: Privilege Escalation
Confidence: 60% confidence
Finding: Code accesses credential files (SSH keys, AWS credentials, etc.). This could indicate credential theft attempts.
Content: Scanner excerpt · data/red-team-guide.md (reported line 131)May include surrounding context.
测试用例:

text
../../../etc/passwd ..\\..\\..\\Windows\\System32\\config\\SAM /etc/shadow C:\\Windows\\System32\\drivers\\etc\\hosts

Credential Access

High

Category: Privilege Escalation
Confidence: 60% confidence
Finding: Code accesses credential files (SSH keys, AWS credentials, etc.). This could indicate credential theft attempts.
Content: Scanner excerpt · data/test-cases.yaml (reported line 321)May include surrounding context.
yaml
**测试用例**: ``` ../../../etc/passwd ..\\..\\..\\Windows\\System32\\config\\SAM /etc/shadow C:\\Windows\\System32\\drivers\\etc\\hosts

Credential Access

High

Category: Privilege Escalation
Confidence: 95% confidence
Finding: Code accesses credential files (SSH keys, AWS credentials, etc.). This could indicate credential theft attempts.
Content: Scanner excerpt · data/red-team-guide.md (reported line 133)May include surrounding context.
text
../../../etc/passwd ..\\..\\..\\Windows\\System32\\config\\SAM /etc/shadow C:\\Windows\\System32\\drivers\\etc\\hosts

Description-Behavior Mismatch

High

Category: Not specified by scanner
Confidence: 97% confidence
Finding: Manifest 说明该技能用于初始化基准数据库、对比历史基线、查看 Pareto front 回退和查阅质量分级标准，并明确不用于给候选打分或自动改进。但该文件实际维护的是覆盖 tool/process/analysis/creation/evaluation/red-team 等广泛类别的通用测试用例库，内容更像通用 skill-evaluator 的评测语料，而非仅限于 benchmark-store 的基准存储或基线对比数据。
Content: No source excerpt is available for this finding.

Description-Behavior Mismatch

High

Category: Not specified by scanner
Confidence: 98% confidence
Finding: L230-L260 定义了“Skill 评估测试”，要求输出“等级、指标、建议的完整报告”，这已经是在构造面向技能能力评估的测试案例。Manifest 明确声明本技能不用于给候选打分，而应改用 improvement-discriminator，因此这里的测试内容与声明边界直接冲突。
Content: No source excerpt is available for this finding.

Intent-Code Divergence

High

Category: Not specified by scanner
Confidence: 99% confidence
Finding: The implementation claims encrypted storage but uses a SHA-256 hash of the password directly as a repeating XOR keystream, which provides no authenticated encryption and is trivially reversible once plaintext structure is known. The salt is included in the object and hash computation but ignored during encryption/decryption, so it provides no cryptographic protection; this makes offline recovery and tampering of hidden tests much easier.
Content: No source excerpt is available for this finding.

Undeclared Tool Scope

Medium

Category: MCP Least Privilege
Confidence: 70% confidence
Finding: Without declared permissions the skill's intent is opaque and cannot be validated.
Content: No source excerpt is available for this finding.

Natural-Language Policy Violations

Medium

Category: Not specified by scanner
Confidence: 95% confidence
Finding: The description is written entirely in Chinese, and the file continues to present primary usage guidance in Chinese without any opt-in or indication that the skill is region-specific. This creates a language/locale policy issue because the skill implicitly enforces a specific language rather than offering a user choice.
Content: No source excerpt is available for this finding.

Natural-Language Policy Violations

Medium

Category: Not specified by scanner
Confidence: 83% confidence
Finding: The document title and content are written as a normative standard in Chinese, and there is no indication that language selection is optional or that the standard is intended only for a Chinese-speaking audience. Under the policy, forcing a specific language without user opt-in or clear region-specific justification is a natural-language policy violation.
Content: No source excerpt is available for this finding.

Vague Triggers

Medium

Category: Not specified by scanner
Confidence: 95% confidence
Finding: The usage guidance only says the skill is for a '测试场景' ('test scenario'), which is too broad to distinguish when the skill should or should not be invoked. The file does not provide specific trigger phrases, scope constraints, or negative examples to prevent unintended activation.
Content: No source excerpt is available for this finding.

Natural-Language Policy Violations

Medium

Category: Not specified by scanner
Confidence: 91% confidence
Finding: The description and title are written in Chinese, and the file provides no indication that language choice is optional or that the skill is intentionally region-specific. Under the policy, forcing a specific language or locale without user opt-in can be a natural-language policy violation.
Content: No source excerpt is available for this finding.

Natural-Language Policy Violations

Medium

Category: Not specified by scanner
Confidence: 91% confidence
Finding: The file’s user-facing instructions and descriptions are entirely in Chinese, and there is no indication that the skill is region-specific or that users may choose another language. Under the stated policy, forcing a specific language without opt-in is a natural-language policy violation.
Content: No source excerpt is available for this finding.

Natural-Language Policy Violations

Medium

Category: Not specified by scanner
Confidence: 95% confidence
Finding: The description and title explicitly present the skill in Chinese ("测试样本", "生产就绪", "测试 Skill") with no indication that language is configurable or limited for a documented regional purpose. This creates a natural-language policy concern because the skill appears to enforce a specific language/locale without user opt-in.
Content: No source excerpt is available for this finding.

Natural-Language Policy Violations

Medium

Category: Not specified by scanner
Confidence: 94% confidence
Finding: The entire guide is written in Chinese and includes no indication that other languages are supported or that Chinese is a required locale for a region-specific purpose. Per the policy, forcing a specific language without user opt-in or documented justification is a natural-language policy concern.
Content: No source excerpt is available for this finding.

Natural-Language Policy Violations

Medium

Category: Not specified by scanner
Confidence: 95% confidence
Finding: This manifest file includes a test case whose input explicitly requires translating English into Chinese, and the assertion validates success by checking for Chinese characters in the output. Because the file is a general-purpose test case library rather than a clearly region-specific tool, this creates a natural-language locale constraint without opt-in or justification.
Content: No source excerpt is available for this finding.

Intent-Code Divergence

Medium

Category: Not specified by scanner
Confidence: 91% confidence
Finding: 尾注将该测试用例库明确归因于“skill-evaluator”生成，而当前审计上下文中的技能是 benchmark-store。虽然这可能表示数据来源，但按当前文件归属来看，该说明会让读者理解为此处内容属于评测器职责，与 benchmark-store 的声明用途相矛盾，形成文档层面的意图混淆。
Content: No source excerpt is available for this finding.

Natural-Language Policy Violations

Medium

Category: Not specified by scanner
Confidence: 95% confidence
Finding: The top-level docstring presents the skill/module description entirely in Chinese, which can impose a specific language on users or maintainers without opt-in. The file does not indicate that the locale is optional, configurable, or justified by a region-specific requirement.
Content: No source excerpt is available for this finding.

Intent-Code Divergence

Medium

Category: Not specified by scanner
Confidence: 88% confidence
Finding: The code and documentation present these tests as 'hidden' and encrypted until execution, but the API can return full HiddenTest objects, including ciphertext and metadata, to any role permitted by the visibility boundary. In this benchmark/evaluation context, exposing the complete encrypted corpus materially increases the chance of offline analysis, test leakage, and overfitting against supposedly hidden tests, undermining evaluation integrity.
Content: No source excerpt is available for this finding.

Natural-Language Policy Violations

Medium

Category: Not specified by scanner
Confidence: 97% confidence
Finding: The module docstring, argparse description, and help text are written exclusively in Chinese, which imposes a specific language on users of this script. Under the policy, locale or language restrictions should either provide user choice or be clearly documented as a justified region-specific constraint, neither of which appears here.
Content: No source excerpt is available for this finding.

Description-Behavior Mismatch

Medium

Category: Not specified by scanner
Confidence: 89% confidence
Finding: The manifest description emphasizes initializing benchmark databases, comparing scores to historical baselines, checking Pareto-front regressions, and consulting grading standards. Adding a CLI 'delete' action introduces destructive database mutation that is not mentioned in that stated scope, expanding behavior beyond a store/reference utility into record removal.
Content: No source excerpt is available for this finding.

Natural-Language Policy Violations

Low

Category: Not specified by scanner
Confidence: 80% confidence
Finding: The file presents all instructions and examples in Chinese and does not state that the skill is China/Chinese-specific or offer any language choice. Per SQP-3, forcing a specific language without user opt-in can be a natural-language policy violation when no justification is provided.
Content: No source excerpt is available for this finding.