ExpertPack Eval

• Outcome: Pass
• Latest audit: Sep 12, 2026 · 3:12 AM UTC
• Version: 1.1.0

Overview

This skill runs disclosed ExpertPack evaluations using OpenRouter and user-supplied agent endpoints, with privacy considerations but no evidence of hidden or destructive behavior.

Install only if you are comfortable sending pack propositions, eval questions, expected answers, and agent responses to OpenRouter and to the endpoint you configure. Use a local or trusted endpoint for sensitive packs, prefer an isolated virtual environment with pinned dependencies, and provide credentials intentionally rather than relying on implicit .env discovery.

A.I.G

Vulnerability Patterns

• Insecure Dependencies: Introduces malicious components through unsafe dependency sources
• Skill Instruction Hijacking: Alters the agent's session goals or safety constraints when the skill loads
• Agent Memory Poisoning: Writes attacker-controlled rules into memory that affect later sessions
• Remote Payload Retrieval and Execution: Fetches external code whose behavior can change after review
• Embedded Malicious Code: Ships malicious scripts inside the skill and executes them locally

Findings (1)

T08 · Insecure Dependencies

Warning

• Location: scripts/run-eval.py:25

Finding

Unpinned Third-Party Dependency Installation Guidance

Content

View full analysis

Vulnerability Details

File Location: scripts/run-eval.py:25-37
Vulnerability Type: Supply-chain exposure through unpinned dependencies
Risk Level: Medium

Vulnerable Code

python

try:
    import yaml
except ImportError:
    print("Error: pyyaml required. Install with: pip install pyyaml")
    sys.exit(1)

try:
    import httpx
except ImportError:
    httpx = None

try:
    import websockets
except ImportError:
    websockets = None

Additional runtime messages instruct users to install the other dependencies without version or integrity constraints:

python

print("Error: httpx required for http:// endpoints. Install with: pip install httpx")
print("Error: websockets required for ws:// endpoints. Install with: pip install websockets")

Technical Analysis

The Skill depends on pyyaml, httpx, and websockets, but the project does not include a version-locked dependency manifest, cryptographic hashes, or constraints specifying reviewed releases. The error messages direct users to install the latest package version resolved by the default pip package index.

Package installation can execute package build or installation logic and introduces executable third-party code into the Skill's runtime. Because package names alone do not identify immutable artifacts, the code ultimately executed can change after this Skill has been audited. This creates exposure to compromised upstream releases, malicious transitive dependencies, package-index compromise, and future dependency versions containing exploitable or malicious behavior.

This issue does not prove that the named packages are currently malicious. The vulnerability is the absence of reproducible and integrity-verified dependency resolution.

Attack Path

1. A user invokes scripts/run-eval.py in an environment where one or more dependencies are unavailable.
2. The script displays an instruction such as `pip i ...[truncated 1264 chars]

Remediation

View remediation

Remediation Suggestions

1. Add a reviewed dependency manifest containing exact versions for all direct and transitive dependencies.

2. Generate and retain cryptographic hashes for every approved distribution artifact.

3. Install dependencies with hash enforcement, for example:

   bash

   python3 -m pip install --require-hashes -r requirements.txt
   

4. Use an isolated virtual environment rather than installing packages globally or as root.

5. Replace generic installation messages with instructions referencing the repository's locked dependency file.

6. Configure trusted package indexes explicitly and protect dependency-resolution configuration from untrusted overrides.

7. Periodically scan and update locked dependencies through a controlled review process, including vulnerability and provenance checks.

8. Consider publishing a reproducible environment definition or signed container image for evaluation runs.

SkillSpector

Vulnerability Patterns

• Data Exfiltration: External Transmission, Env Variable Harvesting, File System Enumeration
• Privilege Escalation: Excessive Permissions, Sudo/Root Execution, Credential Access
• Excessive Agency: Unrestricted Tool Access, Autonomous Decision Making, Scope Creep
• System Prompt Leakage: Direct Leakage, Indirect Extraction, Tool-Based Exfiltration
• MCP Least Privilege: Underdeclared Capability, Wildcard Permission, Missing Permission Declaration

Findings (15)

Tp4

High

• Category: MCP Tool Poisoning
• Confidence: 92% confidence

Finding

The skill description and documented behavior are inconsistent, including claims about EK-ratio measurement while the analyzed material does not substantiate that implementation and references arbitrary endpoint evaluation over HTTP/WebSocket. Description-behavior mismatches are dangerous because operators may trust the declared purpose while the skill can initiate undeclared remote interactions, causing data to be sent to unintended services or enabling SSRF-style access to internal endpoints.

Content

No source excerpt is available for this finding.

Direct Prompt Extraction

High

• Category: System Prompt Leakage
• Confidence: 85% confidence

Finding

Skill contains instructions that could directly expose system prompts, internal rules, or hidden instructions to users or external parties.

Content

Scanner excerpt · scripts/run-eval.py (reported line 205)May include surrounding context.

python

Be strict but fair. A fact counts as present if the response conveys the same meaning, even if phrased differently. A hallucination violation counts if the response contains the specific wrong information described."""

    return prompt


async def judge_response(question: dict, response: str, judge_model: str, api_key: str) -> dict:

Missing User Warnings

High

• Category: Not specified by scanner
• Confidence: 98% confidence

Finding

The script sends the full judging prompt to OpenRouter, including the original question, expected answer, required facts, anti-hallucination checks, and the agent's response. This can expose proprietary eval datasets, internal ground truth answers, and possibly sensitive model outputs to a third-party provider without a strong disclosure or consent mechanism.

Content

No source excerpt is available for this finding.

Credential Access

High

• Category: Privilege Escalation
• Confidence: 60% confidence

Finding

Code accesses credential files (SSH keys, AWS credentials, etc.). This could indicate credential theft attempts.

Content

Scanner excerpt · scripts/run-eval.py (reported line 288)May include surrounding context.

python

# Load API key for judge
    api_key = os.environ.get('OPENROUTER_API_KEY', '')
    if not api_key and not args.dry_run:
        # Try loading from .env files
        for env_path in ['/root/.openclaw/.env', os.path.expanduser('~/.openclaw/.env')]:
            if os.path.exists(env_path):
                with open(env_path) as f:

Credential Access

High

• Category: Privilege Escalation
• Confidence: 60% confidence

Finding

Code accesses credential files (SSH keys, AWS credentials, etc.). This could indicate credential theft attempts.

Content

Scanner excerpt · scripts/run-eval.py (reported line 289)May include surrounding context.

python

api_key = os.environ.get('OPENROUTER_API_KEY', '')
    if not api_key and not args.dry_run:
        # Try loading from .env files
        for env_path in ['/root/.openclaw/.env', os.path.expanduser('~/.openclaw/.env')]:
            if os.path.exists(env_path):
                with open(env_path) as f:
                    for line in f:

Undeclared Tool Scope

Medium

• Category: MCP Least Privilege
• Confidence: 92% confidence

Finding

The skill advertises executable commands that read environment variables, access local files, write output files, and make external network calls, but it does not declare any explicit tool scope such as permissions or allowed-tools. This creates a least-privilege failure: a caller or platform cannot reliably constrain what the skill may access, increasing the risk of unintended secret exposure, filesystem access, or outbound requests when the skill is invoked.

Content

No source excerpt is available for this finding.

Missing User Warnings

Medium

• Category: Not specified by scanner
• Confidence: 95% confidence

Finding

The function sends raw proposition text to OpenRouter to generate targeted questions, which can disclose potentially sensitive or proprietary pack contents to a third-party API. In this skill's context, packs may contain private expert knowledge, so transmitting that content without an explicit warning, consent gate, or redaction path creates a real data-exposure risk.

Content

No source excerpt is available for this finding.

External Transmission

Medium

• Category: Data Exfiltration
• Confidence: 91% confidence

Finding

This outbound request is the concrete transmission point for proposition content used during question generation. While network access is core to the tool's design, it still constitutes a real confidentiality risk because pack data is sent to an external provider and may contain sensitive or proprietary information.

Content

Scanner excerpt · scripts/eval-ek.py (reported line 176)May include surrounding context.

python

}
    
    try:
        resp = requests.post(OPENROUTER_BASE, headers=headers, json=payload, timeout=15)
        resp.raise_for_status()
        question = resp.json().get("choices", [{}])[0].get("message", {}).get("content", "").strip()
        if question:

External Transmission

Medium

• Category: Data Exfiltration
• Confidence: 80% confidence

Finding

Data is being sent to an external URL. This could be legitimate telemetry or data exfiltration. Manual review is recommended.

Content

Scanner excerpt · scripts/eval-ek.py (reported line 219)May include surrounding context.

python

for attempt in range(max_retries):
        try:
            resp = requests.post(OPENROUTER_BASE, headers=headers, json=payload, timeout=30)
            
            if resp.status_code == 429:
                retry_after = int(resp.headers.get("Retry-After", 5))

Missing User Warnings

Medium

• Category: Not specified by scanner
• Confidence: 95% confidence

Finding

The scorer sends both the ground-truth proposition and the probed model's response to an external judge model, creating another channel for sensitive pack content to leave the environment. Because the judge receives the exact fact being evaluated, this may leak confidential knowledge more directly than the probing step, and users are not clearly warned at runtime.

Content

No source excerpt is available for this finding.

External Transmission

Medium

• Category: Data Exfiltration
• Confidence: 95% confidence

Finding

This outbound request is the concrete transmission point for the judge prompt containing the ground-truth proposition and model response. It directly exposes pack facts to a third-party model service, which is especially risky for an expert-knowledge evaluation tool where the underlying content may be confidential or commercially valuable.

Content

Scanner excerpt · scripts/eval-ek.py (reported line 311)May include surrounding context.

python

}
    
    try:
        resp = requests.post(OPENROUTER_BASE, headers=headers, json=payload, timeout=15)
        resp.raise_for_status()
        
        judge_answer = resp.json().get("choices", [{}])[0].get("message", {}).get("content", "").strip().upper()

Missing User Warnings

Medium

• Category: Not specified by scanner
• Confidence: 95% confidence

Finding

The tool sends evaluation questions to arbitrary HTTP/WebSocket endpoints, which can disclose proprietary eval content or sensitive prompts to third-party services. Because the endpoint is user-supplied and there is no explicit warning, users may unintentionally transmit confidential pack data off-host.

Content

No source excerpt is available for this finding.

Description-Behavior Mismatch

Low

• Category: Not specified by scanner
• Confidence: 81% confidence

Finding

The manifest emphasizes measuring EK ratio and running automated evaluations, but this script also creates an eval/ directory in the target pack and writes a YAML report there by default. Persisting new files inside the pack is a state-changing behavior not clearly conveyed by the measurement-focused description, even though results storage is mentioned at a high level.

Content

No source excerpt is available for this finding.

Context-Inappropriate Capability

Low

• Category: Not specified by scanner
• Confidence: 88% confidence

Finding

The script silently falls back to reading OpenRouter credentials from local OpenClaw .env files instead of only using the declared OPENROUTER_API_KEY environment variable. This expands the credential access surface and may surprise users or operators who did not intend this tool to read local secret stores, especially in shared or automated environments.

Content

No source excerpt is available for this finding.

Missing User Warnings

Low

• Category: Not specified by scanner
• Confidence: 87% confidence

Finding

The script accesses credentials from environment variables and local .env files without clearly informing the user at runtime. While it does not exfiltrate the key directly, undisclosed secret access behavior is risky because it violates least surprise and can create unsafe assumptions about what the tool reads from the host.

Content

No source excerpt is available for this finding.

Static analysis

No suspicious patterns detected.