Back to skill

Security audit

Skill Eval

Security checks for vulnerabilities and agentic risk

Overview

This skill is an evaluation tool, but it retains full session data and includes a local review viewer with unsafe browser/server behavior that should be reviewed before use.

Install only if you are comfortable with an eval tool that stores raw conversation histories and tool outputs. Run it in an isolated workspace, avoid using real secrets or proprietary prompts in evals, review/delete eval-workspace artifacts after use, and patch or avoid the local viewer until the script embedding, benchmark rendering, feedback endpoint, and automatic port termination are fixed. Do not run the viewer as a privileged user.

Vulnerability Patterns
  • Unauthorized Access and Privilege EscalationObtains permissions beyond the task's legitimate needs
  • Insecure Skill Coding PracticesFinds exploitable flaws such as hardcoded secrets or command injection
  • Skill Instruction HijackingAlters the agent's session goals or safety constraints when the skill loads
  • Agent Memory PoisoningWrites attacker-controlled rules into memory that affect later sessions
  • Remote Payload Retrieval and ExecutionFetches external code whose behavior can change after review
Findings (4)

T09 · Insecure Skill Coding Practices

Error
Location
viewer/generate_review.py:271
Finding

Stored Script Injection Through Unsafe Embedding of Evaluation Data

Content
View full analysis
// ---- Embedded data (injected by generate_review.py) ---- /*__EMBEDDED_DATA__*/ ``` ### Technical Analysis The generator serializes workspace-controlled data with `json.dumps()` and inserts the resulting JSON directly into an HTML ` ``` can terminate the original script element before the JavaScript parser processes the JSON string. The browser then interprets the injected markup as a new executable script. Potentially attacker-controlled embedded fields include: - Evaluation prompts - Generated output files - Grading records - Previous feedback and outputs - Benchmark content - Skill names Evaluation output is especially untrusted because an evaluated Skill or model can deliberately produce the breakout sequence. ### Attack Path 1. An attacker creates or influences an evaluated Skill, prompt, output, grading record, or benchmark. 2. The attacker causes one of the embedded text fields to contain a `` breakout followed by an attacker-controlled script element. 3. A reviewer runs `viewer/generate_review.py` against the affected evaluation workspace. 4. `generate_html()` inserts the serialized data into the executable script block without HTML-context escaping. 5. ...[truncated 1196 chars]
Remediation
View remediation
SAFE_SERIALIZED_DATA ``` 3. Before insertion, escape all characters that can affect the HTML parser, including `<`, `>`, `&`, U+2028, and U+2029. At minimum, replace `<` with `\u003c`. 4. Parse the value from text rather than executing it: ```javascript const EMBEDDED_DATA = JSON.parse( document.getElementById("embedded-data").textContent ); ``` 5. Prefer a vetted serializer designed for embedding JSON in HTML. 6. Add a restrictive Content Security Policy, such as a nonce-based `script-src`, and remove inline event handlers and inline scripts. 7. Add regression tests containing ``, HTML tags, quotes, Unicode separators, and script payloads in every embedded field. 8. Treat all evaluation artifacts as untrusted, even when they were generated locally. ]]>

T09 · Insecure Skill Coding Practices

Error
Location
viewer/viewer.html:1146
Finding

DOM-Based Cross-Site Scripting in Benchmark Rendering

Content
View full analysis
Benchmark Results"; html += "

"; if (metadata.skill_name) html += "" + escapeHtml(metadata.skill_name) + " — "; if (metadata.timestamp) html += metadata.timestamp + " — "; if (metadata.evals_run) html += "Evals: " + metadata.evals_run.join(", ") + " — "; html += (metadata.runs_per_configuration || "?") + " runs per configuration"; html += "

"; ``` Configuration names are also inserted as raw HTML: ```javascript const configGroups = [...new Set(evalRuns.map(r => r.configuration))]; for (let ci = 0; ci < configGroups.length; ci++) { const config = configGroups[ci]; const configRuns = evalRuns.filter(r => r.configuration === config); if (configRuns.length === 0) continue; const rowClass = ci === 0 ? "benchmark-row-with" : "benchmark-row-without"; const configLabel = config.replace(/_/g, " ").replace(/\b\w/g, c => c.toUpperCase()); for (const run of configRuns) { const r = run.result || {}; const prClass = r.pass_rate >= 0.8 ? "benchmark-delta-positive" : r.pass_rate < 0.5 ? "benchmark-delta-negative" : ""; html += ''; html += "" + configLabel + ""; ``` The constructed string is finally interpreted as HTML: ```javascript container.innerHTML = html; ``` ### Technical Analysis `renderBenchmark()` builds a large HTML string from benchmark JSON. Although some fields are passed through `escapeHtml()`, several benchmark-controlled fields are not, inc ...[truncated 1880 chars]
Remediation
View remediation

T09 · Insecure Skill Coding Practices

Warning
Location
viewer/generate_review.py:358
Finding

Unauthenticated Cross-Origin Overwrite of Local Evaluation Feedback

Content
View full analysis
None: if self.path == "/api/feedback": length = int(self.headers.get("Content-Length", 0)) body = self.rfile.read(length) try: data = json.loads(body) if not isinstance(data, dict) or "reviews" not in data: raise ValueError("Expected JSON object with 'reviews' key") self.feedback_path.write_text(json.dumps(data, indent=2) + "\n") resp = b'{"ok":true}' self.send_response(200) except (json.JSONDecodeError, OSError, ValueError) as e: resp = json.dumps({"error": str(e)}).encode() self.send_response(500) self.send_header("Content-Type", "application/json") self.send_header("Content-Length", str(len(resp))) self.end_headers() self.wfile.write(resp) else: self.send_error(404) ``` ### Technical Analysis The local feedback endpoint accepts state-changing POST requests without: - Authentication - A per-launch capability token - CSRF protection - `Origin` or `Referer` validation - `Host` validation - Content-Type enforcement - A request-body size limit - Strict validation of individual review records Binding the server to `127.0.0.1` prevents direct remote TCP connections but does not prevent a malicious website loaded in the user's browser from sending requests to localhost. The server does not require `application/json`. Therefore, an attacker can use a cross-origin simple request with `Content-Type: text/plain`, avoiding a CORS preflight in applicable browser configurations. CORS normally prevents the attacker from reading the response, but it does not inherently stop the state-changing request from reaching the server. The u ...[truncated 1530 chars]
Remediation
View remediation
MAX_BODY: self.send_error(413) return ``` 7. Strictly validate the schema: - `reviews` must be a list. - Each item must contain only expected fields. - `run_id`, `feedback`, and `timestamp` must have expected types and length limits. - `status` must be from a fixed allowlist. 8. Write updates atomically through a temporary file followed by `os.replace()`. 9. Consider avoiding an HTTP write endpoint entirely in static mode and using an explicit user-initiated download. ]]>

T05 · Unauthorized Access and Privilege Escalation

Warning
Location
viewer/generate_review.py:286
Finding

Viewer Silently Terminates Unrelated Processes Using the Selected Port

Content
View full analysis
None: """Kill any process listening on the given port.""" try: result = subprocess.run( ["lsof", "-ti", f":{port}"], capture_output=True, text=True, timeout=5, ) for pid_str in result.stdout.strip().split("\n"): if pid_str.strip(): try: os.kill(int(pid_str.strip()), signal.SIGTERM) except (ProcessLookupError, ValueError): pass if result.stdout.strip(): time.sleep(0.5) except subprocess.TimeoutExpired: pass except FileNotFoundError: print("Note: lsof not found, cannot check if port is in use", file=sys.stderr) ``` The function is invoked automatically before binding: ```python # Kill any existing process on the target port port = args.port _kill_port(port) ``` ### Technical Analysis Starting a report viewer does not require terminating existing processes. The code enumerates every process listening on the user-selected port and sends `SIGTERM` without: - User confirmation - Verification that the process belongs to an earlier viewer instance - Verification of executable identity - Ownership metadata recorded by this application - A requirement for an explicit replacement option - An attempt to select another port before terminating processes The port is user-controlled through `--port`, making it possible to target any known listening service that the current user has permission to terminate. The code already contains a safer fallback that binds to an ephemeral free port if binding fails. Consequently, process termination exceeds the minimum privileges necessary for the declared viewer f ...[truncated 1256 chars]
Remediation
View remediation
Vulnerability Patterns
  • Prompt InjectionInstruction Override, Hidden Instructions, Exfiltration Commands
  • Data ExfiltrationExternal Transmission, Env Variable Harvesting, File System Enumeration
  • Excessive AgencyUnrestricted Tool Access, Autonomous Decision Making, Scope Creep
  • Output HandlingUnvalidated Output Injection, Cross-Context Output, Unbounded Output
  • Trigger AbuseOverly Broad Trigger, Shadow Command Trigger, Keyword Baiting Trigger
Findings (45)

Tp4

High
Category
MCP Tool Poisoning
Confidence
93% confidence
Finding

The declared description presents a broader skill evaluation framework centered on running evaluations through session spawning/history to test trigger rate, quality, and model comparisons. The supplied code does not execute evaluations, inspect trigger behavior, or compare response quality. Instead, it only analyzes already-recorded timing data and produces latency/stability reports. While latency benchmarking could be tangentially related to evaluation, the primary purpose and operational behavior here are materially different from the declared framework, so this is a mismatch.

Content

No source excerpt is available for this finding.

Tp4

High
Category
MCP Tool Poisoning
Confidence
93% confidence
Finding

The declared description presents a broader evaluation framework for testing skill performance and comparisons. The supplied code chunk does not evaluate anything; it only retrieves and reformats prior conversation history from a completed session. While this may support evaluation workflows, its primary purpose is extraction/export of session transcripts, not running evaluations. There are no undeclared dangerous permissions, but the primary purpose is materially narrower and different from the declared skill behavior.

Content

No source excerpt is available for this finding.

Tp4

High
Category
MCP Tool Poisoning
Confidence
94% confidence
Finding

The declared description presents a general skill-evaluation framework focused on running evaluations such as trigger-rate tests, A/B comparisons, and model comparisons through sessions-based execution. The supplied code instead implements a narrow diagnostics utility for analyzing skill-description quality after trigger results already exist. Its primary behavior is heuristic scoring, failure categorization, and recommendation/report generation. While related to trigger-rate evaluation workflows, it is materially different in primary purpose and omitted declared capabilities: it neither runs the evaluations itself nor uses the stated sessions_spawn/sessions_history mechanism, and it does not support A/B or model comparison. Therefore the description does not accurately represent this code chunk.

Content

No source excerpt is available for this finding.

Tp4

High
Category
MCP Tool Poisoning
Confidence
95% confidence
Finding

The declared description says this skill is a general skill-evaluation framework for trigger rate, A/B quality comparison, and model comparison, specifically via sessions_spawn + sessions_history. The supplied code instead has a narrower and materially different primary purpose: profiling latency/performance of skill execution. It measures elapsed wall-clock time over repeated runs, computes p50/p90/stability metrics, supports sequential versus parallel timing modes, compares model speed, and writes latency reports to disk. While there is some overlap with 'benchmark' and 'model comparison,' the benchmark here is specifically speed/latency, not trigger rate or response quality. Additionally, the implementation does not call sessions_history despite the description explicitly naming it; step-level analysis is only a heuristic regex parse over returned transcript text. These differences are substantial enough to count as a description/behavior mismatch.

Content

No source excerpt is available for this finding.

Tp4

High
Category
MCP Tool Poisoning
Confidence
96% confidence
Finding

The declared description says this skill is an evaluation framework that runs via sessions_spawn + sessions_history for benchmarking and trigger-rate/model comparisons. The supplied code does not implement spawning sessions, collecting conversation histories, or performing evaluation orchestration. Instead, it is a post-processing/review utility: it discovers existing run directories, reads prompts/outputs/grading, generates a self-contained HTML viewer, serves it over HTTP, optionally writes a static HTML file, and saves human feedback. These are materially different primary functions and include undeclared capabilities such as hosting a server, launching a browser, persisting feedback, and terminating processes on a port. While the code is related to evaluation workflows, it is specifically an eval-results viewer/review tool rather than the declared evaluation execution framework.

Content

No source excerpt is available for this finding.

Context Leakage

High
Category
Data Exfiltration
Confidence
91% confidence
Finding

The skill explicitly collects session histories and preserves sessions with cleanup='keep', which can retain full conversation transcripts, tool outputs, and possibly secrets from evaluated runs. In an evaluation context this may be intentional, but it still creates a real data-exposure risk if prompts contain credentials, proprietary data, or sensitive user content and those artifacts are stored in workspace files or accessible histories.

Content

Scanner excerpt · SKILL.md (reported line 24)May include surrounding context.

md
| Read `~/.openclaw/openclaw.json` | Find skill directories (extraDirs) | Path resolution |
| Write to `eval-workspace/` | Store evaluation results | Every eval run |
| Call `sessions_spawn` | Run test queries in isolated sessions | Trigger & quality tests |
| Call `sessions_history` | Collect conversation data for analysis | After each spawn |
| Persist `cleanup="keep"` sessions | Required for trigger detection | Trigger rate tests |

**NOT performed automatically**: Gateway restart, config modification, skill installation. These require manual user action (see "Bundled Test Skill" section).

Context Leakage

High
Category
Data Exfiltration
Confidence
85% confidence
Finding

Code or instructions that leak agent conversation context to external services, potentially exposing sensitive user interactions.

Content

Scanner excerpt · scripts/build_evals_with_context.py (reported line 8)May include surrounding context.

python
Phase 3.3a: integrate real session histories into evals.json

Usage flow:
1. Run evaluations with run_orchestrator.py, record session keys
2. Extract history from sessions using extract_session_history.py
3. Merge histories into evals.json using this script

Context Leakage

High
Category
Data Exfiltration
Confidence
85% confidence
Finding

Code or instructions that leak agent conversation context to external services, potentially exposing sensitive user interactions.

Content

Scanner excerpt · scripts/build_evals_with_context.py (reported line 13)May include surrounding context.

python
Phase 3.3a: integrate real session histories into evals.json

Usage flow:
1. Run evaluations with run_orchestrator.py, record session keys
2. Extract history from sessions using extract_session_history.py
3. Merge histories into evals.json using this script

Context Leakage

High
Category
Data Exfiltration
Confidence
75% confidence
Finding

Code or instructions that leak agent conversation context to external services, potentially exposing sensitive user interactions.

Content

Scanner excerpt · scripts/extract_session_history.py (reported line 3)May include surrounding context.

python
#!/usr/bin/env python3
"""
Extract conversation history from a completed session.

Phase 3.3 tool: extract conversation history from a real session for context testing.

Context Leakage

High
Category
Data Exfiltration
Confidence
75% confidence
Finding

Code or instructions that leak agent conversation context to external services, potentially exposing sensitive user interactions.

Content

Scanner excerpt · scripts/extract_session_history.py (reported line 5)May include surrounding context.

python
"""
Extract conversation history from a completed session.

Phase 3.3 tool: extract conversation history from a real session for context testing.

Usage:
    python extract_session_history.py \

Context Leakage

High
Category
Data Exfiltration
Confidence
90% confidence
Finding

The function retrieves full session message history via sessions_history and returns user/assistant content, then later writes it to disk. If session keys are accessible to unintended users or if extracted files are stored insecurely, this enables leakage of potentially sensitive prompts, secrets, or personal data from prior conversations.

Content

Scanner excerpt · scripts/extract_session_history.py (reported line 22)May include surrounding context.

python
def extract_history_from_session(session_key: str) -> list:
    """
    Extract conversation history from a session.
    
    Returns list of {"role": "user"|"assistant", "content": "..."}
    """

Context Leakage

High
Category
Data Exfiltration
Confidence
88% confidence
Finding

The CLI exposes a straightforward path to export conversation history from a supplied session key and optionally print it to stdout. In a skill-evaluation context this increases risk because real sessions may contain confidential model inputs, tool outputs, or user data, and the script operationalizes bulk disclosure without visible safeguards.

Content

Scanner excerpt · scripts/extract_session_history.py (reported line 95)May include surrounding context.

python
def main():
    parser = argparse.ArgumentParser(
        description="Extract conversation history from OpenClaw session"
    )
    parser.add_argument(
        "--session-key",

Unvalidated Output Injection

High
Category
Output Handling
Confidence
100% confidence
Finding

Model output is used without validation or sanitization. Unvalidated output injected into downstream contexts (SQL, shell, HTML) enables injection attacks and arbitrary code execution.

Content

Scanner excerpt · scripts/legacy/run_compare.py (reported line 90)May include surrounding context.

python
return "\n".join(lines)


def process_eval(result: dict, evals_by_id: dict, out_dir: Path) -> tuple:
    """Process a single eval (fetch histories, extract transcripts, save files).
    
    Returns: (eval_id, eval_name, success, error_msg)

Vague Triggers

Medium
Category
Not specified by scanner
Confidence
92% confidence
Finding

The example trigger "evaluate weather trigger" is a short natural-language phrase without clear scoping or exclusion conditions. In a README, this can encourage activation language that overlaps with ordinary user requests and does not specify when this skill should or should not be invoked.

Content

No source excerpt is available for this finding.

Vague Triggers

Medium
Category
Not specified by scanner
Confidence
88% confidence
Finding

The recommended prompt starts with "Evaluate the skill" and then includes workflow instructions, but it does not define a precise trigger set or state when similar requests should not activate the skill. This ambiguity can lead to unintended invocation for generic evaluation-related requests.

Content

No source excerpt is available for this finding.

Undeclared Tool Scope

Medium
Category
MCP Least Privilege
Confidence
93% confidence
Finding

The skill advertises operational behavior that includes reading user configuration, writing files, and executing Python/shell commands, but it does not declare an explicit tool scope such as allowed-tools or permissions. That creates an authority ambiguity where an evaluator or host may grant broader capabilities than users expect, increasing the blast radius if the skill is triggered in the wrong context.

Content

No source excerpt is available for this finding.

Intent-Code Divergence

Medium
Category
Not specified by scanner
Confidence
94% confidence
Finding

The documentation says all agent execution runs through sessions APIs, but later sections instruct running analysis scripts via exec. This inconsistency can mislead users and reviewers about the skill's actual authority, causing them to approve or trigger a skill that performs more direct local execution than advertised.

Content

No source excerpt is available for this finding.

Context-Inappropriate Capability

Medium
Category
Not specified by scanner
Confidence
91% confidence
Finding

The skill instructs the agent to run local Python and shell commands such as path resolution and file-copy/restart steps, which expands execution beyond the narrowly described sessions-based evaluation model. Even if some steps are marked manual, normalizing arbitrary local command execution inside a skill increases the chance of unsafe command invocation, path abuse, or execution on sensitive hosts.

Content

No source excerpt is available for this finding.

Missing User Warnings

Medium
Category
Not specified by scanner
Confidence
95% confidence
Finding

The workflow explicitly instructs the agent to persist full session histories, including tool usage and message content, to disk in a workspace path. Because evaluation transcripts can contain sensitive prompts, model outputs, file paths, and potentially secrets surfaced during testing, storing them without minimization, redaction, retention limits, or operator warning creates a real data exposure risk.

Content

No source excerpt is available for this finding.

Missing User Warnings

Medium
Category
Not specified by scanner
Confidence
92% confidence
Finding

This workflow persists raw transcripts and timing artifacts for multiple models and runs, increasing the volume and lifetime of potentially sensitive conversation data on disk. In a skill-evaluation context, repeated benchmark prompts and outputs may include proprietary inputs, internal paths, or confidential test cases, so silent persistence expands the attack surface for accidental disclosure or later compromise.

Content

No source excerpt is available for this finding.

Natural-Language Policy Violations

Medium
Category
Not specified by scanner
Confidence
96% confidence
Finding

The instruction text is written as a direct operating requirement in Chinese and does not indicate that the agent should match the user's preferred language or offer a locale choice. This creates a natural-language policy concern because it implicitly constrains operation to a specific language without opt-in or justification.

Content

No source excerpt is available for this finding.

Natural-Language Policy Violations

Medium
Category
Not specified by scanner
Confidence
92% confidence
Finding

The file’s operational instructions and output guidance are written entirely in Chinese, which imposes a specific language on the skill’s behavior. There is no indication that the user can opt into another language or that the comparator is intentionally limited to a Chinese-language context.

Content

No source excerpt is available for this finding.

Skill Enumeration

Medium
Category
Agent Snooping
Confidence
80% confidence
Finding

Skill enumerates or reads other installed skills. Access to other skills' SKILL.md files or the skills directory reveals prompt instructions, capabilities, and secrets that should be invisible to peer skills.

Content

Scanner excerpt · evals/weather/quality.json (reported line 3)May include surrounding context.

json
{
  "skill_name": "weather",
  "skill_path": "/opt/homebrew/lib/node_modules/openclaw/skills/weather/SKILL.md",
  "description": "Trigger rate tests for weather skill description",
  "evals": [
    {

Skill Enumeration

Medium
Category
Agent Snooping
Confidence
80% confidence
Finding

Skill enumerates or reads other installed skills. Access to other skills' SKILL.md files or the skills directory reveals prompt instructions, capabilities, and secrets that should be invisible to peer skills.

Content

Scanner excerpt · evals/weather/triggers.json (reported line 3)May include surrounding context.

json
{
  "skill_name": "weather",
  "skill_path": "/opt/homebrew/lib/node_modules/openclaw/skills/weather/SKILL.md",
  "description": "Trigger rate tests for weather skill description",
  "evals": [
    {

Skill Enumeration

Medium
Category
Agent Snooping
Confidence
80% confidence
Finding

Skill enumerates or reads other installed skills. Access to other skills' SKILL.md files or the skills directory reveals prompt instructions, capabilities, and secrets that should be invisible to peer skills.

Content

Scanner excerpt · tests/test_analyze_triggers.py (reported line 42)May include surrounding context.

python
{
  "skill_name": "weather",
  "skill_path": "/opt/homebrew/lib/node_modules/openclaw/skills/weather/SKILL.md",
  "description": "Trigger rate tests for weather skill description",
  "evals": [
    {

Static analysis

Detected: suspicious.dynamic_code_execution

Dynamic code execution detected.

Critical
Code
suspicious.dynamic_code_execution
Location
scripts/analyze_latency.py:219

Dynamic code execution detected.

Critical
Code
suspicious.dynamic_code_execution
Location
scripts/analyze_model_compare.py:330

Dynamic code execution detected.

Critical
Code
suspicious.dynamic_code_execution
Location
scripts/analyze_quality.py:210

Dynamic code execution detected.

Critical
Code
suspicious.dynamic_code_execution
Location
scripts/analyze_triggers.py:243

Dynamic code execution detected.

Critical
Code
suspicious.dynamic_code_execution
Location
scripts/build_evals_with_context.py:89

Dynamic code execution detected.

Critical
Code
suspicious.dynamic_code_execution
Location
scripts/legacy/run_compare.py:91

Dynamic code execution detected.

Critical
Code
suspicious.dynamic_code_execution
Location
scripts/legacy/run_diagnostics.py:605

Dynamic code execution detected.

Critical
Code
suspicious.dynamic_code_execution
Location
scripts/legacy/run_latency_profile.py:495