LLM Quality Evaluation（LLM质量评测）

• Outcome: Pass
• Latest audit: Sep 16, 2026 · 11:25 AM UTC
• Version: 1.0.0

Overview

This skill is a local LLM quality-evaluation guide and toolkit with no evidence of hidden network access, credential access, persistence, or destructive behavior.

Installers should treat this as a Chinese-language local quality-evaluation aid. If using real customer logs or production conversations to build evaluation sets, sanitize and de-identify the data first and confirm authorization. Do not rely solely on the bundled SECURITY_AUDIT.md because it contains stale references, but the inspected code itself is local and low-privilege.

A.I.G

Vulnerability Patterns

• Skill Instruction Hijacking: Alters the agent's session goals or safety constraints when the skill loads
• Agent Memory Poisoning: Writes attacker-controlled rules into memory that affect later sessions
• Remote Payload Retrieval and Execution: Fetches external code whose behavior can change after review
• Embedded Malicious Code: Ships malicious scripts inside the skill and executes them locally
• Unauthorized Access and Privilege Escalation: Obtains permissions beyond the task's legitimate needs

SkillSpector

Vulnerability Patterns

• Excessive Agency: Unrestricted Tool Access, Autonomous Decision Making, Scope Creep
• Behavioral AST: exec() Call, eval() Call, Dynamic Import
• MCP Least Privilege: Underdeclared Capability, Wildcard Permission, Missing Permission Declaration
• MCP Tool Poisoning: Hidden Instructions, Unicode Deception, Parameter Description Injection
• Prompt Injection: Instruction Override, Hidden Instructions, Exfiltration Commands

Findings (21)

Tp4

High

• Category: MCP Tool Poisoning
• Confidence: 99% confidence

Finding

The declared purpose is LLM quality evaluation, but the finding indicates the packaged behavior instead processes security/compliance data and emits artifacts like security_results.json and security-radar.svg. This mismatch is dangerous because users and reviewers may grant trust, permissions, or run workflows under false assumptions, enabling unintended handling of sensitive security outputs and bypassing review controls tied to the stated purpose.

Content

No source excerpt is available for this finding.

Natural-Language Policy Violations

Medium

• Category: Not specified by scanner
• Confidence: 94% confidence

Finding

This markdown file contains user-facing natural language exclusively in Chinese, including headings, findings, and usage guidance. Under the policy rule for language/locale, forcing a single language without user opt-in or an explicit documented justification can violate organizational language-choice requirements.

Content

No source excerpt is available for this finding.

Intent-Code Divergence

Medium

• Category: Not specified by scanner
• Confidence: 97% confidence

Finding

The audit report claims to assess the ai-llm-evaluation package, but the detailed evidence references different ai_governance files. This mismatch undermines the integrity and traceability of the security review, creating a real risk that consumers rely on a report that does not actually cover the artifact being distributed.

Content

No source excerpt is available for this finding.

Intent-Code Divergence

Medium

• Category: Not specified by scanner
• Confidence: 98% confidence

Finding

The document asserts zero findings and no informational notes, yet later admits command-execution occurrences and gives follow-up recommendations. These internal contradictions make the audit unreliable and can mask risk by presenting a false sense of safety to users or reviewers.

Content

No source excerpt is available for this finding.

Undeclared Tool Scope

Medium

• Category: MCP Least Privilege
• Confidence: 70% confidence

Finding

Without declared permissions the skill's intent is opaque and cannot be validated.

Content

No source excerpt is available for this finding.

Missing User Warnings

Medium

• Category: Not specified by scanner
• Confidence: 92% confidence

Finding

The document explicitly recommends building normal test cases by sampling from online logs or customer-service records, but it provides no privacy, consent, minimization, anonymization, or access-control guidance. In an evaluation-data-construction skill, this omission can lead users to ingest production conversations containing personal, confidential, or regulated data into test datasets, spreading sensitive data into annotation, storage, and reporting workflows.

Content

No source excerpt is available for this finding.

Natural-Language Policy Violations

Medium

• Category: Not specified by scanner
• Confidence: 97% confidence

Finding

This code file contains natural-language strings that force a specific language/locale for the interface, including the module docstring, help text, and printed output. The policy requires flagging language constraints unless the skill offers user opt-in or clearly documents a justified region-specific limitation, neither of which appears here.

Content

No source excerpt is available for this finding.

Description-Behavior Mismatch

Medium

• Category: Not specified by scanner
• Confidence: 92% confidence

Finding

The manifest describes a skill for LLM quality evaluation, regression testing, RAG metrics, hallucination detection, and evaluation reporting, explicitly contrasting it with security red-team testing. This file instead reads security_results.json and generates a multi-dimensional 'security radar' report with security-specific dimensions and benchmarks, which is materially outside the stated quality-only scope.

Content

No source excerpt is available for this finding.

Natural-Language Policy Violations

Medium

• Category: Not specified by scanner
• Confidence: 95% confidence

Finding

The module docstring and all user-facing output are written in Chinese, including labels and status messages, with no option to select another language or locale. This is a natural-language policy concern because the skill hard-codes one language rather than offering user opt-in or documenting a justified locale restriction.

Content

No source excerpt is available for this finding.

Natural-Language Policy Violations

Medium

• Category: Not specified by scanner
• Confidence: 85% confidence

Finding

The file-level docstring presents the skill entirely in Chinese and frames the evaluation dimensions only in that locale, with no indication that users can choose another language. Under the policy, forcing a specific language without explicit user opt-in is a natural-language locale violation unless the constraint is documented and justified.

Content

No source excerpt is available for this finding.

Context-Inappropriate Capability

Medium

• Category: Not specified by scanner
• Confidence: 87% confidence

Finding

The manifest describes a local toolkit for checking metrics, building evaluation-set checklists, viewing RAG indicators, comparing models, and generating report templates. Spawning a subprocess to invoke another script is a more powerful execution capability than those user-facing evaluation functions require, especially in a verification script framed as local validation.

Content

No source excerpt is available for this finding.

subprocess module call

Medium

• Category: Dangerous Code Execution
• Confidence: 70% confidence

Finding

subprocess module calls execute external commands. Without careful input validation, this enables command injection.

Content

Scanner excerpt · verify/llm_eval_quality_test.py (reported line 23)May include surrounding context.

python

def run_tool(args, timeout=30):
    try:
        r = subprocess.run([PY, os.path.join(TOOLS, "llm_eval_toolkit.py")] + args,
                           capture_output=True, text=True, timeout=timeout)
        return r.returncode, r.stdout or ""
    except Exception as e:

Natural-Language Policy Violations

Low

• Category: Not specified by scanner
• Confidence: 88% confidence

Finding

Line L04 requires branding to uniformly use the Chinese phrase "注册老炮" and forbids alternatives, which is a natural-language locale constraint. Because the file provides no user opt-in, language choice, or region-specific justification, this can conflict with language/locale policy expectations.

Content

No source excerpt is available for this finding.

Natural-Language Policy Violations

Low

• Category: Not specified by scanner
• Confidence: 84% confidence

Finding

The manifest uses Chinese-only natural-language values for the package and author fields, which signals a language-specific presentation without any accompanying opt-in, multilingual alternative, or documented regional scope. Under the stated policy, locale-specific language constraints should either offer user choice or be clearly justified as region-specific.

Content

No source excerpt is available for this finding.

Natural-Language Policy Violations

Low

• Category: Not specified by scanner
• Confidence: 84% confidence

Finding

This markdown file uses Chinese throughout and does not provide any opt-in, alternative language, or justification that the content is intended only for a Chinese-language audience. Under the policy rule for language or locale constraints, forcing a specific language without user choice can be a natural-language policy violation.

Content

No source excerpt is available for this finding.

Natural-Language Policy Violations

Low

• Category: Not specified by scanner
• Confidence: 92% confidence

Finding

This markdown file contains natural-language guidance but presents it only in Chinese, despite the title's inclusion of 'FAQ' suggesting broader audience use. Under the policy, forcing a specific language without user opt-in can be a language/locale policy violation when no choice or justification is provided.

Content

No source excerpt is available for this finding.

Intent-Code Divergence

Low

• Category: Not specified by scanner
• Confidence: 67% confidence

Finding

The generated SVG footer states the artifact is a local closed-loop verification with no network or collection, but the broader intent signaled by this file is a security verification/reporting workflow. In the context of the manifest, which says this skill measures quality and is complementary to AI security red-team testing rather than performing it, this documentation contributes to an intent mismatch by presenting security validation output from a skill not described as doing security assessment.

Content

No source excerpt is available for this finding.

Description-Behavior Mismatch

Low

• Category: Not specified by scanner
• Confidence: 78% confidence

Finding

The module docstring at L002 describes '本地闭环，零网络' quality/security validation, which suggests an offline verifier; however, the script also persists a JSON report to verify/security_results.json. While local file output is not dangerous, it is additional behavior beyond pure validation/checking and is not mentioned in the stated purpose.

Content

No source excerpt is available for this finding.

Natural-Language Policy Violations

Low

• Category: Not specified by scanner
• Confidence: 89% confidence

Finding

This SVG contains user-facing labels entirely in Chinese, such as the axis titles and chart annotations, but the file provides no indication that Chinese is optional or that the asset is intended only for a Chinese-specific audience. Under the policy, forcing a specific language without user opt-in can be a natural-language policy violation.

Content

No source excerpt is available for this finding.

Natural-Language Policy Violations

Low

• Category: Not specified by scanner
• Confidence: 93% confidence

Finding

The main title, legend, and footer are all rendered in Chinese and there is no accompanying notice that the graphic is region-specific or that alternate locales are available. Because this is user-visible natural language, the single-language presentation may violate the language/locale policy when no opt-in or justification is provided.

Content

No source excerpt is available for this finding.

Natural-Language Policy Violations

Low

• Category: Not specified by scanner
• Confidence: 87% confidence

Finding

This file contains user-facing natural-language content exclusively in Chinese (for example, the note field), which can indicate a fixed language/locale assumption. Under the policy, forced language without user opt-in or clear justification is a natural-language policy concern.

Content

No source excerpt is available for this finding.