ClawHub
Back to skill

Security audit

rag-eval-harness

Security checks for vulnerabilities and agentic risk

Overview

The skill is a local RAG evaluation toolkit with disclosed scripts for reading a chosen knowledge base and writing local reports, and I found no hidden data sending or destructive behavior.

Install and run this only if you are comfortable with local Python evaluation scripts reading the knowledge-base folder you configure. Check RAG_EVAL_BASE before running, keep suite_registry.json trusted, and remember that text is sent to your local Ollama service and reports/history are written locally.

SkillSpector

Vulnerability Patterns
  • Behavioral ASTexec() Call, eval() Call, Dynamic Import
  • MCP Least PrivilegeUnderdeclared Capability, Wildcard Permission, Missing Permission Declaration
  • MCP Tool PoisoningHidden Instructions, Unicode Deception, Parameter Description Injection
  • Prompt InjectionInstruction Override, Hidden Instructions, Exfiltration Commands
  • Data ExfiltrationExternal Transmission, Env Variable Harvesting, File System Enumeration
Findings (3)

subprocess module call

Medium
Category
Dangerous Code Execution
Content
return {"module_id": mod["id"], "status": "error", "error": f"脚本不存在: {script}"}
    print(f"\n>>> 运行模块 [{mod['id']}] {mod['name']}")
    try:
        r = subprocess.run([PY, script], cwd=script_dir,
                           capture_output=True, text=True, timeout=300)
    except Exception as e:
        return {"module_id": mod["id"], "status": "error", "error": str(e)}
Confidence
91% confidence
Finding
The code executes a Python script path taken from the registry without any trust boundary checks, allowlisting, or path containment validation. Although it avoids shell injection by using an argument list, a malicious or tampered suite_registry.json can cause arbitrary local Python code execution, which is a real security risk in an evaluation harness that runs modules automatically.

Lp3

Medium
Category
MCP Least Privilege
Confidence
84% confidence
Finding
The skill metadata declares no permissions, yet the package reportedly has capabilities for environment access, file read/write, network, and shell execution. This breaks least-privilege expectations and can mislead users or hosts about what the skill is able to do, increasing the risk of unintended file access, local command execution, or data exposure if the skill is invoked.

Tp4

High
Category
MCP Tool Poisoning
Confidence
92% confidence
Finding
The declared purpose is RAG evaluation methodology, but the analyzed behavior reportedly includes integrity/security auditing, writing unrelated security reports, subprocess execution, history logging, and HTTP connections to a local Ollama service. This description-behavior mismatch is dangerous because users may authorize a seemingly benign evaluation skill without realizing it can execute commands, write files, and communicate over the network, which materially broadens the attack surface.

Static analysis

No suspicious patterns detected.