Security audit

Improvement Discriminator

Security checks for vulnerabilities and agentic risk

Overview

The skill broadly matches a candidate-review purpose, but it includes gate-adjacent execution routing and unsafe in-process loading of local Python code that users should review before installing.

Install only for a controlled improvement pipeline. Do not run the runtime evaluator on untrusted skill paths unless it is sandboxed, do not let accept_for_execution automatically apply changes without a separate gate, and treat human-review receipts as advisory unless you add real signer verification.

A.I.G

Vulnerability Patterns

Insecure Skill Coding PracticesFinds exploitable flaws such as hardcoded secrets or command injection
Skill Instruction HijackingAlters the agent's session goals or safety constraints when the skill loads
Agent Memory PoisoningWrites attacker-controlled rules into memory that affect later sessions
Remote Payload Retrieval and ExecutionFetches external code whose behavior can change after review
Embedded Malicious CodeShips malicious scripts inside the skill and executes them locally

Findings (2)

T09 · Insecure Skill Coding Practices

Error

Location: interfaces/critic_engine.py:353
Finding: Untrusted Skill Modules Are Executed In-Process Without Isolation
Content: View full analysis
BenchmarkResult: """评估单个基准测试用例""" start_time = time.time() try: # 调用 Skill 函数 if self.skill_func: actual_output = self.skill_func(case.input_data) ``` The loader is reached through the following caller-controlled path: ```python if skill_path: # 优先使用真实 Skill 评估器 try: ...[truncated 2895 chars]
Remediation: View remediation

T09 · Insecure Skill Coding Practices

Warning

Location: interfaces/human_review.py:132
Finding: Human-Review Receipts Use Forgeable Checksums and Are Accepted Without Verification
Content: View full analysis
str: """计算回执的数字签名 (用于验证完整性)""" content = json.dumps({ "receipt_id": self.receipt_id, "skill_name": self.skill_name, "skill_version": self.skill_version, "reviewer_id": self.reviewer_id, "decision": self.decision.value, "final_score": self.final_score, "created_at": self.created_at, "findings_count": len(self.findings), }, sort_keys=True) return hashlib.sha256(content.encode()).hexdigest()[:16] def verify_signature(self) -> bool: """验证签名是否有效""" return self.signature == self._compute_signature() ``` Receipt ingestion does not enforce the available verification method: ```python def add_receipt(self, receipt: HumanReviewReceipt) -> None: """添加已有的审查回执""" self._receipts[receipt.receipt_id] = receipt def load_receipt(self, path: Union[str, Path]) -> HumanReviewReceipt: """ 从文件加载审查回执 Args: path: 文件路径 Returns: 加载的审查回执 """ receipt = HumanReviewReceipt.load(path) self._receipts[receipt.receipt_id] = receipt return receipt ``` Unverified attacker-controlled scores are then included in aggregate scoring: ```python if use_final_scores: # 使用人工指定的 final_score final_scores = [] for r in self._receipts.values(): if hasattr(r, 'final_score') and r.final_score is not None: final_scores.append(r.final_score) elif isinstance(r, dict) and r.get('final_score') is not None: final_scores.append(r['final_score']) if final_scores: review_score = ...[truncated 2538 chars]
Remediation: View remediation

SkillSpector

Vulnerability Patterns

Excessive AgencyUnrestricted Tool Access, Autonomous Decision Making, Scope Creep
Trigger AbuseOverly Broad Trigger, Shadow Command Trigger, Keyword Baiting Trigger
MCP Least PrivilegeUnderdeclared Capability, Wildcard Permission, Missing Permission Declaration
MCP Tool PoisoningHidden Instructions, Unicode Deception, Parameter Description Injection
Prompt InjectionInstruction Override, Hidden Instructions, Exfiltration Commands

Findings (47)

Tp4

High

Category: MCP Tool Poisoning
Confidence: 98% confidence
Finding: 声明描述的是一个针对“改进候选评审”的评分与裁决工具，重点能力应包括多人盲审、LLM 语义评估、候选接受判断及原因分析。实际代码却只是一个最小断言系统，围绕规则校验、阈值检查、正则匹配、成本/延迟检查、自定义断言和加权评分进行通用结果验证。它没有任何与候选评审、panel 审稿人、盲审流程、LLM 调用、语义判断或 hold 原因诊断直接相关的实现。因此这不是支持性细节层面的差异，而是主要目的和能力上的明显不一致。
Content: No source excerpt is available for this finding.

Tp4

High

Category: MCP Tool Poisoning
Confidence: 92% confidence
Finding: 声明描述强调的是“多人盲审 + LLM judge”的候选评审工具，且特别说不用于门禁决策。但代码中没有看到任何 LLM 调用、多人盲审编排、panel 机制或盲审匿名处理逻辑；相反，核心实现是测试驱动的评估框架，侧重 benchmark、hidden tests、assertions、external regression、human review receipt 的汇总评分。它还会给出 level/verdict（如 production_ready、stable、basic、needs_improvement），这与‘不用于门禁决策’存在明显张力。虽然‘人工审查’与声明中的‘多人盲审’有表面相近之处，但这里更像接收已有人工审查回执，而不是组织多审阅者盲审流程。因此，代码的主要用途与声明存在实质性偏差。
Content: No source excerpt is available for this finding.

Tp4

High

Category: MCP Tool Poisoning
Confidence: 98% confidence
Finding: 声明描述的是“候选改进评审/盲审/LLM 评判”类能力，重点在多人评审、语义判断和候选接受性分析；而代码没有任何与 panel 审阅者、盲审流程、LLM 调用、语义评估、hold 原因分析或候选接受决策相关的实现。相反，代码的主功能是一个外部回归测试结果接入层，负责从文件或字典读取测试结果、标准化为统一结构、计算测试分数、汇总并生成报告。这是与声明明显不同的主要用途，且触发场景也更接近“导入回归测试/生成测试报告/合并测试分数”，并非“score candidate / multi-reviewer panel / LLM judge”。因此属于显著不匹配。
Content: No source excerpt is available for this finding.

Tp4

High

Category: MCP Tool Poisoning
Confidence: 97% confidence
Finding: 代码的核心是“人工审查结果接入接口”和“审查管理器”，用于记录 reviewer 身份、决策、findings、confidence、manual override、final score，并保存为 JSON、加载、汇总、导出报告，还能把人工审查结果融合进基础评分。这与声明的“改进候选打分/多人盲审/LLM 语义评估”并不一致。虽然两者都与评审有关，但代码明显侧重人工 review receipt artifact 与审查结果管理，而不是候选评估技能本身。特别是声明强调的 --llm-judge、语义评估、多人盲审均未在代码中出现；相反，代码引入了人工覆盖、审查签名、报告导出、审查工件持久化等未声明的实质能力。因此应判定为描述与实际行为不匹配。
Content: No source excerpt is available for this finding.

Tp4

High

Category: MCP Tool Poisoning
Confidence: 97% confidence
Finding: 代码的主功能与描述明显不符。描述强调的是候选改进的多人盲审和 LLM 语义评估流程；而代码完全没有 panel、reviewer aggregation、盲审、LLM 调用或文本语义判断逻辑。相反，它读取候选元数据和文件结构，映射评估类别，按固定权重和阈值计算 accuracy/reliability/efficiency/cost/coverage/security 等分数，给出 approve/conditional/reject verdict，并生成 gate/review 建议。这不仅缺少声明中的核心能力，还执行了描述中特别排除的结构评估和门禁式判断，因此应判定为实质性不匹配。
Content: No source excerpt is available for this finding.

Tp4

High

Category: MCP Tool Poisoning
Confidence: 90% confidence
Finding: 描述前半部分与代码大体一致：确实支持候选打分、多人 panel 和 LLM judge，也能帮助解释为何结果为 hold。但存在关键不一致。描述明确说“不用于结构评估（用 improvement-learner）或门禁决策（用 improvement-gate）”，而代码实际会：1) 直接生成 recommendation（accept_for_execution / hold / reject），这是明显的门禁/准入建议；2) 在 build_blockers 中依据 skill_level_insufficient_for_structural_change、类别、风险、protected_target 等规则对结构性变更进行阻断判断；3) 产出 ranking artifact 时写入 next_step='execute_candidate'、next_owner='executor'，表明结果被用于后续执行流转，而非仅做分析评分。因此描述低估了该脚本的实际职责，属于实质性能力不匹配。
Content: No source excerpt is available for this finding.

Tp4

High

Category: MCP Tool Poisoning
Confidence: 96% confidence
Finding: 声明中明确写了“不用于门禁决策（用 improvement-gate）”，但该代码块实际不仅做评分和 LLM 评估测试，还直接测试了与 gate/ReviewGate 的集成：例如 llm_reject_adds_blocker、ReviewGate.validate 中因 LLM reject 而 fail gate、conditional 时 pass 等。这说明该技能实际行为会影响门禁决策流程，而这正是声明中排除的用途。虽然代码是测试代码而非完整实现，但其覆盖范围清楚表明该模块预期参与 gate 判定，因此描述与实际行为存在实质性不一致。
Content: No source excerpt is available for this finding.

Tp4

High

Category: MCP Tool Poisoning
Confidence: 94% confidence
Finding: 声明描述的是一个面向‘改进候选评审与打分’的技能，核心能力应包括多审阅者盲审、LLM 语义裁判和候选接受相关分析。给出的代码却是测试代码，主要验证底层评估组件和集成是否正常工作：断言检查、评估引擎、真实 skill 加载、报告导出、JSON 导出、以及 mock 配置。虽然其中存在“score/evaluate”相关字样和评估引擎调用，可能属于同一大领域，但这段代码的直接用途是内部集成测试，不是声明中的最终用户功能实现。因此主用途明显不同，且多个关键宣称能力在代码中缺失，应判定为描述与行为不匹配。
Content: No source excerpt is available for this finding.

Tp4

High

Category: MCP Tool Poisoning
Confidence: 95% confidence
Finding: 声明强调的是“候选改进打分/多人盲审/LLM judge”这一用户可用评审能力，但代码并未实现或暴露这些核心能力。相反，它是一个测试文件，用于验证评估系统的 P2-a 组件：hidden tests、external regression、human review receipt、以及 critic engine 的集成。代码中没有看到 --panel、--llm-judge、多人盲审流程、LLM 语义评估调用，触发词与代码行为也不一致。虽然代码确实涉及‘评分/评估’，但其主要目的明显是测试评估框架本身，而非执行声明中的候选评审技能，因此构成实质性描述-行为不匹配。
Content: No source excerpt is available for this finding.

Description-Behavior Mismatch

High

Category: Not specified by scanner
Confidence: 95% confidence
Finding: This file implements capabilities far beyond the declared purpose of an improvement discriminator, including benchmark execution, hidden tests, runtime skill loading, regression ingestion, and human-review processing. That scope expansion increases attack surface and makes it easier for a caller to use this component in unintended, security-sensitive workflows under misleading skill metadata.
Content: No source excerpt is available for this finding.

Description-Behavior Mismatch

High

Category: Not specified by scanner
Confidence: 98% confidence
Finding: The code assigns production-style verdicts such as production_ready and stable, which turns a supposed non-gating scoring tool into a gate-decision engine. In context, this is dangerous because users may rely on these outputs for deployment or trust decisions that the manifest explicitly says this skill should not make.
Content: No source excerpt is available for this finding.

Context-Inappropriate Capability

High

Category: Not specified by scanner
Confidence: 99% confidence
Finding: The RealSkillEvaluator loads a Python file from a user-provided filesystem path and executes it via importlib, which runs top-level module code immediately. This is effectively arbitrary code execution in the current Python process and can lead to full compromise of the host environment, data exfiltration, filesystem tampering, or persistence if an attacker can influence the path or module contents.
Content: No source excerpt is available for this finding.

Description-Behavior Mismatch

High

Category: Not specified by scanner
Confidence: 94% confidence
Finding: The manifest describes this skill as for blind-review scoring, LLM semantic evaluation, and understanding hold outcomes, and explicitly says it is not for gate decisions. However, merge_scores() and _decision_to_score() convert review decisions such as approved/rejected into numeric outcomes and blend them into a final score, which is directly usable as a gating mechanism rather than just review intake or analysis.
Content: No source excerpt is available for this finding.

Description-Behavior Mismatch

High

Category: Not specified by scanner
Confidence: 97% confidence
Finding: The module computes a final verdict (approve/conditional/reject) and directly incorporates boundary checks into that decision, which is a gating function. This conflicts with the skill’s declared scope of evaluation-only use, creating a capability/intent mismatch that can cause downstream systems or operators to treat heuristic evidence as an authorization decision.
Content: No source excerpt is available for this finding.

Description-Behavior Mismatch

High

Category: Not specified by scanner
Confidence: 95% confidence
Finding: The returned boundary object exposes auto_promote_eligible and the verdict is derived partly from that flag, which extends the adapter from assessment into promotion/execution policy. In this skill context, that is dangerous because users may rely on these fields as de facto gate signals despite the manifest stating it is not for gate decisions.
Content: No source excerpt is available for this finding.

Description-Behavior Mismatch

High

Category: Not specified by scanner
Confidence: 98% confidence
Finding: The artifact explicitly sets "next_step" to "execute_candidate" and "next_owner" to "executor", directly advancing outputs into the execution workflow. In this context, that materially increases the danger beyond mere labeling because the scorer is not only making gate-like judgments but wiring its results into an automated execution path, which can cause unreviewed or mis-scored changes to be applied.
Content: No source excerpt is available for this finding.

Undeclared Tool Scope

Medium

Category: MCP Least Privilege
Confidence: 90% confidence
Finding: The skill advertises operational commands involving reading input files, invoking Python scripts, and producing output artifacts, but it declares no explicit tool scope or permissions boundary. In an agent environment, that omission can cause the runtime to grant broader file, shell, or environment access than users expect, increasing the risk of unsafe execution or data exposure if the skill is invoked on untrusted inputs.
Content: No source excerpt is available for this finding.

Description-Behavior Mismatch

Medium

Category: Not specified by scanner
Confidence: 95% confidence
Finding: The documentation says the skill is not for gate or accept decisions, yet elsewhere it describes determining whether candidates should be accepted and emitting accept/conditional/reject decisions. In agent pipelines, this kind of contradictory interface contract can cause operators or orchestrators to route the skill into higher-trust decision points than intended, creating unsafe automation and authorization confusion.
Content: No source excerpt is available for this finding.

Natural-Language Policy Violations

Medium

Category: Not specified by scanner
Confidence: 97% confidence
Finding: The manifest description forces a specific language context for users by presenting the activation/use guidance only in Chinese. This is a natural-language policy concern because no opt-in, alternative language, or justification for the locale constraint is provided.
Content: No source excerpt is available for this finding.

Natural-Language Policy Violations

Medium

Category: Not specified by scanner
Confidence: 91% confidence
Finding: Most of the 'When to Use', 'When NOT to Use', CLI guidance, and output semantics are written in Chinese, which effectively enforces a language requirement on users. The file does not offer an English alternative throughout nor state that the skill is intentionally region- or locale-specific.
Content: No source excerpt is available for this finding.

Intent-Code Divergence

Medium

Category: Not specified by scanner
Confidence: 94% confidence
Finding: The skill documentation contradicts itself about whether it only scores candidates or also makes accept/reject decisions. Ambiguous semantics are dangerous in multi-agent systems because downstream components may treat advisory outputs as authoritative, leading to unintended execution, promotion, or rejection of changes.
Content: No source excerpt is available for this finding.

Intent-Code Divergence

Medium

Category: Not specified by scanner
Confidence: 97% confidence
Finding: The module-level documentation lists json_schema as a core assertion type, and AssertionType defines JSON_SCHEMA, implying this capability exists. However, AssertionExecutor.execute has no branch for AssertionType.JSON_SCHEMA, and create_assertion also omits any json_schema mapping, so such assertions cannot actually run.
Content: No source excerpt is available for this finding.

Intent-Code Divergence

Medium

Category: Not specified by scanner
Confidence: 88% confidence
Finding: The demo constructs an assertion described as Score >= 0.8 and provides actual_output = {"output": ..., "score": 0.85}, which implies the threshold assertion should inspect the score field. In reality, AssertionRunner.run passes the entire dict to every assertion, and _check_threshold attempts float(actual_output) for non-numeric inputs, causing failure rather than evaluating the embedded score.
Content: No source excerpt is available for this finding.

Missing User Warnings

Medium

Category: Not specified by scanner
Confidence: 94% confidence
Finding: The evaluator loads a Python file from a provided path and executes it via importlib, which can run arbitrary module-level code and later invoke exported functions. Although the code comments describe the mechanism, there is no user-facing confirmation, warning, or visible disclosure at the execution point to alert users that untrusted code may be executed.
Content: No source excerpt is available for this finding.

Intent-Code Divergence

Medium

Category: Not specified by scanner
Confidence: 93% confidence
Finding: The documentation claims mock evaluation is limited, but the engine silently falls back to MockSkillEvaluator when real loading fails or no evaluator is available. This can mask failures, produce misleadingly positive or deterministic scores, and create integrity issues where users believe a real skill was tested when it was not.
Content: No source excerpt is available for this finding.

Static analysis

Detected: suspicious.dynamic_code_execution, suspicious.exposed_secret_literal

Dynamic code execution detected.

Critical

Code: suspicious.dynamic_code_execution
Location: interfaces/critic_engine.py:379

Dynamic code execution detected.

Critical

Code: suspicious.dynamic_code_execution
Location: scripts/score.py:402

File appears to expose a hardcoded API secret or token.

Critical

Code: suspicious.exposed_secret_literal
Location: interfaces/critic_engine.py:685

File appears to expose a hardcoded API secret or token.

Critical

Code: suspicious.exposed_secret_literal
Location: tests/test_p1_integration.py:115

File appears to expose a hardcoded API secret or token.

Critical

Code: suspicious.exposed_secret_literal
Location: tests/test_p2a_integration.py:79