Back to skill

Security audit

Skill Eval

Security checks across malware telemetry and agentic risk

Overview

This evaluation skill is mostly coherent, but it needs review because it stores full evaluation conversations and includes a viewer that can automatically stop a local process on its port.

Install only if you are comfortable with evaluation transcripts and session histories being saved in the workspace. Avoid running it on sensitive or production conversations unless you control the output directory and retention. Be cautious with the HTML viewer: choose an unused port, and do not open untrusted spreadsheet/report artifacts in it.

SkillSpector

By NVIDIA
Vulnerability Patterns
  • Data ExfiltrationExternal Transmission, Env Variable Harvesting, File System Enumeration
  • Excessive AgencyUnrestricted Tool Access, Autonomous Decision Making, Scope Creep
  • Trigger AbuseOverly Broad Trigger, Shadow Command Trigger, Keyword Baiting Trigger
  • Behavioral ASTexec() Call, eval() Call, Dynamic Import
  • Prompt InjectionInstruction Override, Hidden Instructions, Exfiltration Commands
Findings (20)

Context-Inappropriate Capability

Medium
Confidence
98% confidence
Finding
The helper unconditionally identifies and sends SIGTERM to any process listening on the requested port, regardless of ownership or purpose. That creates an unsafe local denial-of-service capability unrelated to core review-page generation, and can disrupt developer tools or other local services if the chosen/default port is already in use.

Vague Triggers

Medium
Confidence
87% confidence
Finding
The example trigger phrase "evaluate weather trigger" is short and generic enough that it can overlap with ordinary user requests about weather evaluation or trigger testing, causing the skill to activate when the user did not intend to run this evaluation framework. In a skill system where descriptions and example phrases influence routing, overly broad invocation language can lead to accidental skill execution, workspace writes, and spawning subagents for unrelated conversations.

Vague Triggers

Medium
Confidence
82% confidence
Finding
The invocation block describes running evaluations using natural-language instructions like "Evaluate the <skill-name> skill" without clearly constraining when the skill should match, which makes the trigger scope ambiguous. Because this repository is specifically about an evaluation skill that can spawn subagents and execute analysis steps, ambiguous routing increases the chance of unintended activation and unnecessary execution of higher-impact workflows.

Missing User Warnings

Medium
Confidence
91% confidence
Finding
The workflow explicitly instructs the agent to fetch full session histories with tools included and write them to disk in a workspace directory. Session histories can contain sensitive user prompts, model outputs, tool arguments, and possibly secrets or personal data, yet the guide provides no minimization, redaction, retention, or operator warning; this creates a real data-exposure risk if the workspace is shared, backed up, or reused.

Missing User Warnings

Medium
Confidence
89% confidence
Finding
This workflow stores assistant transcripts derived from evaluation prompts to plaintext files without any warning that the content may include sensitive user-derived material, proprietary prompts, or confidential outputs. Even though it extracts only the last assistant message rather than the entire history, persistence to disk still increases disclosure risk and can leak data through shared filesystems, logs, or later processing.

Natural-Language Policy Violations

Medium
Confidence
89% confidence
Finding
The instruction '在所有 eval grading 完成后运行' and the overall Chinese-only content can cause the subagent to respond in Chinese regardless of the user's language, which may reduce usability and lead to misunderstood results or operator error. In a security or evaluation workflow, forcing a single language without negotiation is risky because it can hide important findings from users who expect another language, though it is not directly exploitable in the same way as code execution or prompt injection.

Missing User Warnings

Medium
Confidence
90% confidence
Finding
The script persists merged conversation histories directly to a JSON file, which can contain sensitive session-derived prompts, responses, or identifiers. In a skill-evaluation framework, this creates a real data-handling risk because context gathered from prior sessions may be retained longer than intended, copied into repos or artifacts, and exposed to users who did not need access.

Missing User Warnings

Medium
Confidence
95% confidence
Finding
The script intentionally fetches complete session histories with tool calls and results, then writes both raw histories and extracted transcripts to disk. Those artifacts can contain prompts, credentials, API responses, file contents, or other sensitive data from evaluated sessions, so storing them unredacted creates a real confidentiality risk if the workspace is shared, backed up, or later consumed by other tools.

Missing User Warnings

Medium
Confidence
91% confidence
Finding
The script persists full model transcripts to disk under the output directory without any consent prompt, redaction, or warning. In a skill-evaluation context, transcripts can easily contain sensitive prompts, embedded skill contents, secrets echoed by the model, or regulated data from eval sets, so silent persistence increases the risk of unintended retention and later disclosure.

Missing User Warnings

Medium
Confidence
90% confidence
Finding
The script sends eval prompts and a task instructing the model to read the specified skill path to an external model invocation API, but gives no explicit notice that local data may be transmitted outward. Because this tool is designed to evaluate arbitrary skills and prompts, the outbound content may include proprietary instructions, confidential eval material, or sensitive user data, making undisclosed transmission a meaningful privacy and data-governance risk.

Missing User Warnings

Medium
Confidence
95% confidence
Finding
The script saves full `sessions_history(includeTools=True)` payloads to disk, which can include sensitive conversation content, tool inputs, file paths, and other operational metadata. In a skill-evaluation context this is particularly risky because benchmark runs may involve real prompts or internal data, and the histories are persisted automatically without minimization, redaction, or an explicit warning/consent boundary.

Vague Triggers

Medium
Confidence
92% confidence
Finding
The template uses broad placeholders such as <common-task> and <command> without requiring scope boundaries, safety constraints, or negative examples. In an evaluation framework, this can produce tests that accidentally reward unsafe tool invocation, over-triggering, or execution of destructive commands because the prompt template does not distinguish safe from out-of-scope requests.

Vague Triggers

Medium
Confidence
87% confidence
Finding
The follow-up prompt 'Now <abbreviated-task>' encourages intent inference from prior context without defining when inference is appropriate or when clarification is required. This can cause the agent to over-assume user intent and perform unintended actions, especially in a CLI wrapper setting where inferred commands may have side effects.

Vague Triggers

Medium
Confidence
89% confidence
Finding
The trigger set includes very broad natural-language prompts such as goal-oriented help requests that do not mention the tool, which can cause the skill to activate for unrelated user intents. In a skill-evaluation framework, this is especially risky because it can inflate trigger-rate metrics, produce misleading benchmark results, and cause unintended invocation in real workflows.

Missing User Warnings

Medium
Confidence
99% confidence
Finding
The main flow automatically calls _kill_port on the requested port before starting the server, with no user confirmation or warning. In practice this can terminate unrelated local applications using that port, causing avoidable service interruption and making the tool surprisingly destructive for a viewer utility.

Missing User Warnings

Medium
Confidence
97% confidence
Finding
Spreadsheet sheets are converted to HTML with SheetJS and then inserted with wrapper.innerHTML, which can allow attacker-controlled spreadsheet content to become active DOM. If an untrusted XLSX contains HTML or script-bearing constructs that survive conversion, this can lead to cross-site scripting in a viewer that handles uploaded or generated artifacts, exposing reviewer data and potentially allowing arbitrary actions in the app context.

Context Leakage

High
Category
Data Exfiltration
Content
#!/usr/bin/env python3
"""
Extract conversation history from a completed session.

Phase 3.3 tool: extract conversation history from a real session for context testing.
Confidence
92% confidence
Finding
Extract conversation

Context Leakage

High
Category
Data Exfiltration
Content
"""
Extract conversation history from a completed session.

Phase 3.3 tool: extract conversation history from a real session for context testing.

Usage:
    python extract_session_history.py \
Confidence
92% confidence
Finding
extract conversation

Context Leakage

High
Category
Data Exfiltration
Content
def extract_history_from_session(session_key: str) -> list:
    """
    Extract conversation history from a session.
    
    Returns list of {"role": "user"|"assistant", "content": "..."}
    """
Confidence
96% confidence
Finding
Extract conversation

Context Leakage

High
Category
Data Exfiltration
Content
def main():
    parser = argparse.ArgumentParser(
        description="Extract conversation history from OpenClaw session"
    )
    parser.add_argument(
        "--session-key",
Confidence
90% confidence
Finding
Extract conversation

VirusTotal

64/64 vendors flagged this skill as clean.

View on VirusTotal

Static analysis

Detected: suspicious.dynamic_code_execution

Dynamic code execution detected.

Critical
Code
suspicious.dynamic_code_execution
Location
scripts/analyze_latency.py:219

Dynamic code execution detected.

Critical
Code
suspicious.dynamic_code_execution
Location
scripts/analyze_model_compare.py:330

Dynamic code execution detected.

Critical
Code
suspicious.dynamic_code_execution
Location
scripts/analyze_quality.py:210

Dynamic code execution detected.

Critical
Code
suspicious.dynamic_code_execution
Location
scripts/analyze_triggers.py:243

Dynamic code execution detected.

Critical
Code
suspicious.dynamic_code_execution
Location
scripts/build_evals_with_context.py:89

Dynamic code execution detected.

Critical
Code
suspicious.dynamic_code_execution
Location
scripts/legacy/run_compare.py:91

Dynamic code execution detected.

Critical
Code
suspicious.dynamic_code_execution
Location
scripts/legacy/run_diagnostics.py:605

Dynamic code execution detected.

Critical
Code
suspicious.dynamic_code_execution
Location
scripts/legacy/run_latency_profile.py:495