ClawHub
Back to skill

Security audit

Improvement Discriminator

Security checks for vulnerabilities and agentic risk

Overview

The skill is a review scorer, but it includes under-scoped execution and routing behavior that could let untrusted skill code or manipulated review data influence downstream actions.

Use this only in an isolated review environment, not as an automatic approval or execution gate. Do not point its critic engine at untrusted skill paths unless they run in a separate sandbox with no inherited secrets. Use the mock LLM backend for confidential artifacts unless external disclosure is acceptable, and treat human-review receipts as unauthenticated unless additional signature and reviewer controls are added.

A.I.G

Vulnerability Patterns
  • Skill Instruction HijackingAlters the agent's session goals or safety constraints when the skill loads
  • Insecure Skill Coding PracticesFinds exploitable flaws such as hardcoded secrets or command injection
  • Agent Memory PoisoningWrites attacker-controlled rules into memory that affect later sessions
  • Remote Payload Retrieval and ExecutionFetches external code whose behavior can change after review
  • Embedded Malicious CodeShips malicious scripts inside the skill and executes them locally
Findings (3)

T09 · Insecure Skill Coding Practices

Error
Location
interfaces/critic_engine.py:356
Finding

Unsandboxed Execution of Evaluated Python Skills

Content
View full analysis

Vulnerability Details

File Location: interfaces/critic_engine.py:356-379
Vulnerability Type: Arbitrary code execution through dynamic module loading
Risk Level: High

Vulnerable Code

python
def _load_skill(self):
    """加载 Skill 模块"""
    # 尝试加载 Python 文件
    py_files = list(self.skill_path.glob("*.py")) if self.skill_path.is_dir() else [self.skill_path]

    # 优先查找 main.py 或 evaluate.py
    priority_files = ["main.py", "evaluate.py", "executor.py"]
    target_file = None

    for pf in priority_files:
        candidate = self.skill_path / pf if self.skill_path.is_dir() else self.skill_path
        if candidate.exists() and candidate.name == pf:
            target_file = candidate
            break

    if not target_file and py_files:
        target_file = py_files[0]

    if not target_file:
        raise ValueError(f"No Python files found in {self.skill_path}")

    # 加载模块
    spec = importlib.util.spec_from_file_location("skill_module", target_file)
    if spec and spec.loader:
        self.skill_module = importlib.util.module_from_spec(spec)
        sys.modules["skill_module"] = self.skill_module
        spec.loader.exec_module(self.skill_module)

Technical Analysis

RealSkillEvaluator dynamically imports a Python file selected from the caller-supplied skill_path. Calling spec.loader.exec_module() executes all module-level code immediately, before the implementation verifies that the module exposes an expected evaluate, execute, run, or main function.

The evaluated module runs in the same process and under the same operating-system identity as the critic engine. No sandbox, subprocess boundary, filesystem restriction, network restriction, resource limit, or environment-variable filtering is applied. Consequently, evaluating an untrusted skill is equivalent to granting that skill arbitrary Python code execution.

This capability exceeds the minimum ...[truncated 1107 chars]

Remediation
View remediation

Remediation Suggestions

  • Do not import reviewed or candidate-controlled Python modules into the critic process.
  • Execute dynamic skills in a disposable sandbox or isolated worker with:
    • No inherited secrets or API credentials.
    • Network access disabled by default.
    • A read-only, allowlisted filesystem.
    • A dedicated unprivileged operating-system identity.
    • CPU, memory, process-count, and execution-time limits.
    • A strict structured input and output protocol.
  • Resolve and validate the candidate path against an explicitly approved skill root.
  • Reject symlinks, special files, and paths escaping the approved root.
  • Require an explicit manifest that identifies the entry point instead of selecting the first Python file.
  • Perform static validation before execution, while recognizing that static validation alone is not an adequate sandbox.
  • Remove the global sys.modules["skill_module"] mutation or assign a unique isolated module name if trusted in-process loading remains necessary.

T01 · Skill Instruction Hijacking

Warning
Location
interfaces/llm_judge.py:58
Finding

Prompt Injection Through Untrusted Skill and Candidate Content

Content
View full analysis

Vulnerability Details

File Location: interfaces/llm_judge.py:58-82, 109-119; scripts/score.py:539-545
Vulnerability Type: LLM judge instruction hijacking and external disclosure of artifact content
Risk Level: Medium

Vulnerable Code

python
JUDGE_PROMPT_TEMPLATE = """You are an expert skill evaluator. Assess this proposed improvement to a skill.

## Target Skill
{target_content}

## Proposed Change
- Category: {category}
- Risk Level: {risk_level}
- Description: {description}
- Proposed Content:
{proposed_content}

## Evaluation Dimensions
Rate each dimension 0.0-1.0:

1. **Clarity**: Is the proposed change clear and unambiguous? Will an AI agent understand exactly what to do?
2. **Specificity**: Does it use concrete examples/values instead of vague language ("various", "etc.")?
3. **Consistency**: Does it match the existing skill's style, terminology, and structure?
4. **Safety**: Could this change cause unintended behavior or break existing functionality?

## Output Format
Respond with ONLY a JSON object:
{{
  "clarity": 0.X,
  "specificity": 0.X,
  "consistency": 0.X,
  "safety": 0.X,
  "overall": 0.X,
  "decision": "approve|conditional|reject",
  "reasoning": "one paragraph explaining your assessment",
  "suggestions": ["improvement suggestion 1", "..."]
}}"""
python
def _build_prompt(self, candidate: dict, target_content: str) -> str:
    plan = candidate.get("execution_plan", {})
    proposed_content = "\n".join(plan.get("content_lines", ["(no content specified)"]))
    return JUDGE_PROMPT_TEMPLATE.format(
        target_content=target_content[:2000] if target_content else "(not available)",
        category=candidate.get("category", "unknown"),
        risk_level=candidate.get("risk_level", "unknown"),
        description=candidate.get("proposed_change_summary", candidate.get("title", "")),
        proposed_content=proposed_content[:100
...[truncated 2607 chars]
Remediation
View remediation

Remediation Suggestions

  • Place immutable judge policy in a system message and explicitly state that all target and candidate content is untrusted evidence that must never be followed as instructions.
  • Pass artifact content in structured fields or strongly delimited data sections rather than interpolating it into free-form instructions.
  • Use provider-supported structured-output or JSON-schema enforcement.
  • Validate all returned dimensions, decisions, reasoning, and suggestions against strict types, ranges, and length limits.
  • Treat LLM results as advisory evidence only; do not permit an LLM verdict by itself to authorize execution.
  • Add independent deterministic safety checks and require human review for sensitive or disputed changes.
  • Require explicit operator consent before transmitting artifact text to a remote model.
  • Document exactly which fields and maximum content lengths are sent externally.
  • Restrict custom API endpoints to an approved allowlist, or disable ANTHROPIC_BASE_URL in environments processing confidential artifacts.
  • Add adversarial tests containing prompt-injection instructions in every candidate-controlled field.

T09 · Insecure Skill Coding Practices

Warning
Location
interfaces/human_review.py:132
Finding

Forgeable Human-Review Receipts Influence Security-Relevant Scores

Content
View full analysis

Vulnerability Details

File Location: interfaces/human_review.py:132-148, 359-374, 423-459
Vulnerability Type: Missing authentication and integrity enforcement for human-review decisions
Risk Level: Medium

Vulnerable Code

python
def _compute_signature(self) -> str:
    """计算回执的数字签名 (用于验证完整性)"""
    content = json.dumps({
        "receipt_id": self.receipt_id,
        "skill_name": self.skill_name,
        "skill_version": self.skill_version,
        "reviewer_id": self.reviewer_id,
        "decision": self.decision.value,
        "final_score": self.final_score,
        "created_at": self.created_at,
        "findings_count": len(self.findings),
    }, sort_keys=True)
    return hashlib.sha256(content.encode()).hexdigest()[:16]

def verify_signature(self) -> bool:
    """验证签名是否有效"""
    return self.signature == self._compute_signature()
python
def add_receipt(self, receipt: HumanReviewReceipt) -> None:
    """添加已有的审查回执"""
    self._receipts[receipt.receipt_id] = receipt

def load_receipt(self, path: Union[str, Path]) -> HumanReviewReceipt:
    """
    从文件加载审查回执

    Args:
        path: 文件路径

    Returns:
        加载的审查回执
    """
    receipt = HumanReviewReceipt.load(path)
    self._receipts[receipt.receipt_id] = receipt
    return receipt
python
def merge_scores(
    self,
    base_score: float,
    review_weight: float = 0.3,
    use_final_scores: bool = True,
) -> float:
    """
    将人工审查结果合并到基础评分中
    """
    if not self._receipts:
        return base_score

    if use_final_scores:
        # 使用人工指定的 final_score
        final_scores = []
        for r in self._receipts.values():
            if hasattr(r, 'final_score') and r.final_score is not None:
                final_scores.append(r.final_score)
            elif isinstance(r, dict) and r.get('final_score') is not None:
                final_scores.appe
...[truncated 2230 chars]
Remediation
View remediation

Remediation Suggestions

  • Replace the unkeyed truncated digest with authenticated signatures:
    • Use Ed25519 signatures with each trusted reviewer bound to an approved public key; or
    • Use HMAC-SHA-256 where a suitable secure shared-key model exists.
  • Sign a canonical representation of every security-relevant receipt field.
  • Do not truncate authentication values to 16 hexadecimal characters.
  • Verify signatures before accepting receipts through either load_receipt() or add_receipt().
  • Reject unsigned, invalid, expired, revoked, duplicate, or unauthorized-reviewer receipts.
  • Bind reviewer identities and roles to trusted keys rather than trusting identifiers inside the receipt.
  • Validate that the receipt applies to the expected skill name, version, run identifier, and candidate identifier.
  • Record verification status separately from attacker-controlled receipt data.
  • Prevent unverified receipts from contributing to merge_scores().
  • Maintain an append-only audit log for receipt creation, verification, revocation, and score aggregation.

SkillSpector

Vulnerability Patterns
  • Excessive AgencyUnrestricted Tool Access, Autonomous Decision Making, Scope Creep
  • Trigger AbuseOverly Broad Trigger, Shadow Command Trigger, Keyword Baiting Trigger
  • MCP Least PrivilegeUnderdeclared Capability, Wildcard Permission, Missing Permission Declaration
  • MCP Tool PoisoningHidden Instructions, Unicode Deception, Parameter Description Injection
  • Prompt InjectionInstruction Override, Hidden Instructions, Exfiltration Commands
Findings (45)

Context-Inappropriate Capability

Critical
Category
Not specified by scanner
Confidence
99% confidence
Finding

The evaluator accepts a supplied skill path, dynamically loads a Python module from that path, and executes its top-level code plus a discovered function. In the context of a scoring/review skill, this creates arbitrary code execution if an attacker can influence the path or submitted skill contents, allowing filesystem access, network calls, credential theft, or lateral movement under the evaluator's privileges.

Content

No source excerpt is available for this finding.

Tp4

High
Category
MCP Tool Poisoning
Confidence
98% confidence
Finding

The skill explicitly says it is 'not for gate decisions,' yet it also documents recommendations such as accept_for_execution, reject, hold, and blockers that drive acceptance logic. This mismatch can mislead operators about the authority of the tool and cause it to be used as a de facto gate, especially in automated workflows.

Content

No source excerpt is available for this finding.

Tp4

High
Category
MCP Tool Poisoning
Confidence
95% confidence
Finding

The skill explicitly says it is 'not for gate decisions,' yet it also documents recommendations such as accept_for_execution, reject, hold, and blockers that drive acceptance logic. This mismatch can mislead operators about the authority of the tool and cause it to be used as a de facto gate, especially in automated workflows.

Content

No source excerpt is available for this finding.

Tp4

High
Category
MCP Tool Poisoning
Confidence
99% confidence
Finding

The skill explicitly says it is 'not for gate decisions,' yet it also documents recommendations such as accept_for_execution, reject, hold, and blockers that drive acceptance logic. This mismatch can mislead operators about the authority of the tool and cause it to be used as a de facto gate, especially in automated workflows.

Content

No source excerpt is available for this finding.

Tp4

High
Category
MCP Tool Poisoning
Confidence
97% confidence
Finding

The skill explicitly says it is 'not for gate decisions,' yet it also documents recommendations such as accept_for_execution, reject, hold, and blockers that drive acceptance logic. This mismatch can mislead operators about the authority of the tool and cause it to be used as a de facto gate, especially in automated workflows.

Content

No source excerpt is available for this finding.

Tp4

High
Category
MCP Tool Poisoning
Confidence
95% confidence
Finding

The skill explicitly says it is 'not for gate decisions,' yet it also documents recommendations such as accept_for_execution, reject, hold, and blockers that drive acceptance logic. This mismatch can mislead operators about the authority of the tool and cause it to be used as a de facto gate, especially in automated workflows.

Content

No source excerpt is available for this finding.

Tp4

High
Category
MCP Tool Poisoning
Confidence
94% confidence
Finding

The skill explicitly says it is 'not for gate decisions,' yet it also documents recommendations such as accept_for_execution, reject, hold, and blockers that drive acceptance logic. This mismatch can mislead operators about the authority of the tool and cause it to be used as a de facto gate, especially in automated workflows.

Content

No source excerpt is available for this finding.

Tp4

High
Category
MCP Tool Poisoning
Confidence
93% confidence
Finding

The skill explicitly says it is 'not for gate decisions,' yet it also documents recommendations such as accept_for_execution, reject, hold, and blockers that drive acceptance logic. This mismatch can mislead operators about the authority of the tool and cause it to be used as a de facto gate, especially in automated workflows.

Content

No source excerpt is available for this finding.

Tp4

High
Category
MCP Tool Poisoning
Confidence
96% confidence
Finding

The skill explicitly says it is 'not for gate decisions,' yet it also documents recommendations such as accept_for_execution, reject, hold, and blockers that drive acceptance logic. This mismatch can mislead operators about the authority of the tool and cause it to be used as a de facto gate, especially in automated workflows.

Content

No source excerpt is available for this finding.

Tp4

High
Category
MCP Tool Poisoning
Confidence
98% confidence
Finding

The skill explicitly says it is 'not for gate decisions,' yet it also documents recommendations such as accept_for_execution, reject, hold, and blockers that drive acceptance logic. This mismatch can mislead operators about the authority of the tool and cause it to be used as a de facto gate, especially in automated workflows.

Content

No source excerpt is available for this finding.

Description-Behavior Mismatch

High
Category
Not specified by scanner
Confidence
95% confidence
Finding

The manifest describes an improvement discriminator focused on blind review scoring, LLM judging, and explaining hold decisions, explicitly saying it is not for structural evaluation or gate decisions. In contrast, this code builds a generic critic engine that runs frozen benchmarks and hidden tests, loads external regression data, and aggregates human review into a final verdict such as 'production_ready', which is a materially different evaluation workflow.

Content

No source excerpt is available for this finding.

Description-Behavior Mismatch

High
Category
Not specified by scanner
Confidence
97% confidence
Finding

The manifest says the skill is not for gate decisions, but the scoring logic assigns readiness levels and verdicts including 'production_ready', 'stable', and 'needs_improvement'. Those labels are operational acceptance decisions rather than just blind-review scoring or explanatory analysis.

Content

No source excerpt is available for this finding.

Description-Behavior Mismatch

High
Category
Not specified by scanner
Confidence
97% confidence
Finding

This code implements explicit gate/acceptance decision logic (approve, conditional, reject) even though the skill is declared as not intended for gate decisions. In practice, downstream components may treat this output as authoritative and use it to approve or reject candidates, creating a scope-confusion vulnerability where an advisory scoring tool silently becomes a policy enforcement mechanism.

Content

No source excerpt is available for this finding.

Description-Behavior Mismatch

High
Category
Not specified by scanner
Confidence
95% confidence
Finding

The recommendations instruct operators or automation to auto-keep, execute conservatively, or route to human review, which are decisioning behaviors rather than neutral evaluation. In this skill's context, that is especially risky because the skill description says it is not for gate decisions, so these recommendations can mislead integrators into granting it de facto approval authority.

Content

No source excerpt is available for this finding.

Description-Behavior Mismatch

High
Category
Not specified by scanner
Confidence
97% confidence
Finding

The evidence builder computes auto_promote_eligible and returns gate-oriented outputs, which goes beyond evidence generation into operational approval logic. That is dangerous because automated pipelines often key off fields like this to bypass review, enabling unintended promotion of changes based on a simplified heuristic model that explicitly does not run frozen benchmarks or hidden tests.

Content

No source excerpt is available for this finding.

Description-Behavior Mismatch

High
Category
Not specified by scanner
Confidence
96% confidence
Finding

The scoring component is making policy decisions by returning accept_for_execution, hold, or reject rather than limiting itself to evaluation output. In this skill’s stated context, that violates separation of duties and can let a ranking tool become an implicit gatekeeper, causing unsafe or unreviewed candidates to advance based on heuristics or LLM output.

Content

No source excerpt is available for this finding.

Description-Behavior Mismatch

High
Category
Not specified by scanner
Confidence
98% confidence
Finding

The artifact explicitly sets next_step to execute_candidate and next_owner to executor, which operationalizes the scorer’s output into workflow progression. That makes the component an active controller of downstream execution, so any scoring error, prompt manipulation, or evaluator misjudgment can directly trigger actions the skill metadata says it should not control.

Content

No source excerpt is available for this finding.

Undeclared Tool Scope

Medium
Category
MCP Least Privilege
Confidence
84% confidence
Finding

The skill advertises CLI behavior involving environment-variable API keys plus reading input JSON and writing output files, yet it declares no explicit tool scope or permissions. In an agent system, this can lead to over-broad execution authority and make it easier for the skill to access files or secrets beyond what users expect.

Content

No source excerpt is available for this finding.

Vague Triggers

Medium
Category
Not specified by scanner
Confidence
91% confidence
Finding

The trigger phrase 'evaluate improvement' is broad enough to match many ordinary review requests, increasing the chance this skill is invoked outside its intended niche. In an agent environment, overly broad routing can expose file, environment, or model-judging capabilities in contexts where users did not intend them.

Content

No source excerpt is available for this finding.

Vague Triggers

Medium
Category
Not specified by scanner
Confidence
88% confidence
Finding

The trigger 'LLM judge' is ambiguous and could activate on generic discussions of model evaluation, not specifically this candidate-scoring workflow. That raises the risk of accidental invocation and unintended access to external APIs or local artifacts during unrelated tasks.

Content

No source excerpt is available for this finding.

Vague Triggers

Medium
Category
Not specified by scanner
Confidence
86% confidence
Finding

The standalone trigger '盲审' is extremely broad and likely to match many unrelated blind-review scenarios. Because this skill supports scoring, recommendations, and optional external-model judging, accidental activation could route sensitive review tasks through the wrong workflow.

Content

No source excerpt is available for this finding.

Intent-Code Divergence

Medium
Category
Not specified by scanner
Confidence
98% confidence
Finding

The module-level documentation and AssertionType enum advertise a "json_schema" assertion type as part of the core design, but AssertionExecutor.execute has no branch for JSON_SCHEMA and create_assertion does not map "json_schema" at all. This is an active contradiction between documented capability and implemented behavior, not merely an omitted detail.

Content

No source excerpt is available for this finding.

Intent-Code Divergence

Medium
Category
Not specified by scanner
Confidence
87% confidence
Finding

The demo sets up a threshold assertion described as "Score >= 0.8" and passes an output dict containing a score field, implying the assertion system can evaluate that score. In reality, AssertionRunner passes the whole dict into _check_threshold, which attempts float(actual_output) and will fail, so the demonstrated/documented usage contradicts actual behavior.

Content

No source excerpt is available for this finding.

Context-Inappropriate Capability

Medium
Category
Not specified by scanner
Confidence
88% confidence
Finding

The file prepends a sibling directory to sys.path and imports code from it, creating cross-skill code-loading behavior that can change import resolution globally for the process. If that sibling path is writable or replaceable by an attacker, this can enable import hijacking or execution of unintended code, especially combined with the broader runtime harness design.

Content

No source excerpt is available for this finding.

Missing User Warnings

Medium
Category
Not specified by scanner
Confidence
93% confidence
Finding

The code dynamically imports a Python file from a provided path and immediately executes it via the module loader, which can run arbitrary module-level code. Although comments describe the mechanism, there is no user-facing confirmation, warning, or visible disclosure at the point of execution about running untrusted code from disk.

Content

No source excerpt is available for this finding.