rag-eval-harness

• Outcome: Pass
• Latest audit: Sep 28, 2026 · 7:36 PM UTC
• Version: 1.1.5

Overview

This is a coherent local RAG evaluation skill with purpose-aligned scripts and no evidence of hidden exfiltration, persistence, or destructive behavior.

Before installing or running, review the knowledge-base path you provide because its contents will be embedded through your local Ollama service and written into local evaluation reports. Treat suite_registry.json and referenced scripts as trusted code, and avoid running the suite after editing the registry unless you intend those scripts to execute.

A.I.G

Vulnerability Patterns

• Skill Instruction Hijacking: Alters the agent's session goals or safety constraints when the skill loads
• Agent Memory Poisoning: Writes attacker-controlled rules into memory that affect later sessions
• Remote Payload Retrieval and Execution: Fetches external code whose behavior can change after review
• Embedded Malicious Code: Ships malicious scripts inside the skill and executes them locally
• Unauthorized Access and Privilege Escalation: Obtains permissions beyond the task's legitimate needs

SkillSpector

Vulnerability Patterns

• Excessive Agency: Unrestricted Tool Access, Autonomous Decision Making, Scope Creep
• Behavioral AST: exec() Call, eval() Call, Dynamic Import
• Taint Tracking: Direct Taint Flow, Variable-Mediated Taint Flow, Credential Exfiltration Chain
• MCP Least Privilege: Underdeclared Capability, Wildcard Permission, Missing Permission Declaration
• Prompt Injection: Instruction Override, Hidden Instructions, Exfiltration Commands

Findings (16)

Ae1

High

• Category: analysis-evasion
• Confidence: 100% confidence

Finding

Referenced artifact was not completely inspected

Content

Scanner excerpt · SKILL.md (reported line 104)May include surrounding context.

md

| 文档与理论文本 | 本 `SKILL.md`、`README.md`、其中「理论依据」段与一切理论表述 | **不在 MIT 覆盖范围内**：保留所有权利（All rights reserved） |

Ae1

High

• Category: analysis-evasion
• Confidence: 100% confidence

Finding

Referenced artifact was not completely inspected

Content

Scanner excerpt · SKILL.md (reported line 121)May include surrounding context.

md

| 文档与理论文本 | 本 `SKILL.md`、`README.md`、其中「理论依据」段与一切理论表述 | **不在 MIT 覆盖范围内**：保留所有权利（All rights reserved） |

Lp1

High

• Category: MCP Least Privilege
• Confidence: 75% confidence

Finding

The skill uses 'shell' capability that is not listed in its permissions. This may indicate deceptive intent or missing permission declarations.

Content

No source excerpt is available for this finding.

Lp1

High

• Category: MCP Least Privilege
• Confidence: 75% confidence

Finding

The skill uses 'env' capability that is not listed in its permissions. This may indicate deceptive intent or missing permission declarations.

Content

No source excerpt is available for this finding.

Tainted flow: 'req' from open (line 23, file read) → urllib.request.urlopen (network output)

High

• Category: Data Flow
• Confidence: 80% confidence

Finding

File contents flow to a network sink. This may indicate data exfiltration of sensitive files.

Content

Scanner excerpt · references/local_rag_eval.py (reported line 27)May include surrounding context.

python

OLLAMA + "/api/embed",
        data=json.dumps({"model": EMBED_MODEL, "input": text}).encode(),
        headers={"Content-Type": "application/json"})
    with urllib.request.urlopen(req, timeout=60) as r:
        return json.loads(r.read())["embeddings"][0]

def cosine(a, b):

Natural-Language Policy Violations

Medium

• Category: Not specified by scanner
• Confidence: 89% confidence

Finding

The file begins with a Chinese-only scope notice that defines important licensing limitations, while the rest of the license text is in English. This creates a language/locale policy concern because essential usage terms are conveyed in a specific language without offering an explicit language choice or stating a justified region-specific constraint.

Content

No source excerpt is available for this finding.

Natural-Language Policy Violations

Medium

• Category: Not specified by scanner
• Confidence: 93% confidence

Finding

The natural-language instructions and user-facing descriptions are entirely presented in Chinese, including usage and output descriptions, with no indication that language is configurable or user-selectable. Under the policy, forcing a specific language without user opt-in can be a locale-policy violation unless clearly justified as region-specific.

Content

No source excerpt is available for this finding.

Dynamic import via __import__()

Medium

• Category: Dangerous Code Execution
• Confidence: 75% confidence

Finding

Dynamic import() can load arbitrary modules at runtime, bypassing static analysis and potentially importing malicious code.

Content

Scanner excerpt · rag_eval_harness_security_test.py (reported line 131)May include surrounding context.

python

result = {
        "skill": "rag-eval-harness",
        "version": "1.1.0",
        "timestamp_local": __import__("datetime").datetime.now().isoformat(timespec="seconds"),
        "scope": "本地闭环 · 零真实凭据 · 可重复",
        "dimensions": dims,
        "overall": round(overall, 2),

Context-Inappropriate Capability

Medium

• Category: Not specified by scanner
• Confidence: 88% confidence

Finding

The harness executes module scripts defined in suite_registry.json with the current Python interpreter, so anyone who can alter the registry or referenced script files can cause arbitrary code execution when the suite runs. In an evaluation framework this may be expected functionality, but it still creates a real trust-boundary risk if registry contents are not tightly controlled because the harness treats configuration as executable authority.

Content

No source excerpt is available for this finding.

subprocess module call

Medium

• Category: Dangerous Code Execution
• Confidence: 70% confidence

Finding

subprocess module calls execute external commands. Without careful input validation, this enables command injection.

Content

Scanner excerpt · references/capability_eval_suite/run_suite.py (reported line 34)May include surrounding context.

python

return {"module_id": mod["id"], "status": "error", "error": f"脚本不存在: {script}"}
    print(f"\n>>> 运行模块 [{mod['id']}] {mod['name']}")
    try:
        r = subprocess.run([PY, script], cwd=script_dir,
                           capture_output=True, text=True, timeout=300)
    except Exception as e:
        return {"module_id": mod["id"], "status": "error", "error": str(e)}

Natural-Language Policy Violations

Medium

• Category: Not specified by scanner
• Confidence: 95% confidence

Finding

The suite name and description are entirely specified in Chinese, with no indication that language choice is configurable or that the registry is intentionally limited to a Chinese-language environment. The policy for this audit flags language or locale constraints when a skill appears to force a specific language without user opt-in or clear justification.

Content

No source excerpt is available for this finding.

Natural-Language Policy Violations

Medium

• Category: Not specified by scanner
• Confidence: 96% confidence

Finding

This file hardcodes its natural-language instructions and status messages in Chinese, starting with the top-level docstring. Because the script does not offer any locale selection or explain a justified region-specific constraint, it enforces a specific language without user opt-in.

Content

No source excerpt is available for this finding.

Missing User Warnings

Low

• Category: Not specified by scanner
• Confidence: 85% confidence

Finding

The script writes security_results.json to disk automatically in the target directory without prior confirmation, which can overwrite an existing file or create artifacts the user did not expect. In this skill's local, non-privileged context the risk is limited, but silent file writes are still a real safety issue because they modify the filesystem without explicit consent.

Content

No source excerpt is available for this finding.

Natural-Language Policy Violations

Low

• Category: Not specified by scanner
• Confidence: 95% confidence

Finding

The module docstring is entirely in Chinese and presents the skill behavior in a single fixed language. Under the policy, forcing a specific language without user opt-in or a documented regional justification is a natural-language policy concern.

Content

No source excerpt is available for this finding.

Natural-Language Policy Violations

Low

• Category: Not specified by scanner
• Confidence: 91% confidence

Finding

This file contains multiple user-visible labels entirely in Chinese, including the title and legend text, with no indication that the skill is region-specific or that another language is available. Under the policy, forcing a specific language without user opt-in is a natural-language locale violation.

Content

No source excerpt is available for this finding.

Natural-Language Policy Violations

Low

• Category: Not specified by scanner
• Confidence: 91% confidence

Finding

This JSON file contains user-facing natural-language values such as scope, dimension names, and notes exclusively in Chinese. Under the policy rule for language/locale, forcing a specific language without opt-in or justification is a natural-language policy concern.

Content

No source excerpt is available for this finding.