Back to skill

Security audit

Benchclaw Openclaw Benchmark

Security checks for vulnerabilities and agentic risk

Overview

BenchClaw is a coherent benchmark tool, but it lets a remote service drive the user's main OpenClaw agent and uploads captured output by default, so it belongs in Review before installation.

Review carefully before installing. Prefer running only in an isolated disposable OpenClaw profile or workspace with no sensitive files or credentials, set temp/caller_info.txt to upload_to_server=false unless you explicitly want leaderboard submission, and clear cached submissions after failed uploads. The publisher should add signed question sets, a dedicated low-privilege benchmark agent, path containment, deny-by-default uploads, and stronger disclosure before this should be treated as routine.

Vulnerability Patterns
  • Remote Payload Retrieval and ExecutionFetches external code whose behavior can change after review
  • Unauthorized Access and Privilege EscalationObtains permissions beyond the task's legitimate needs
  • Insecure Skill Coding PracticesFinds exploitable flaws such as hardcoded secrets or command injection
  • Skill Instruction HijackingAlters the agent's session goals or safety constraints when the skill loads
  • Agent Memory PoisoningWrites attacker-controlled rules into memory that affect later sessions
Findings (3)

T03 · Remote Payload Retrieval and Execution

Error
Location
scripts/agent_cli.py:84
Finding

Unsigned Remote Benchmark Instructions Are Executed by the Privileged OpenClaw Agent

Content
View full analysis
dict[str, Any]: out = _post_json( api_url, body={"model_name": primary_model, "client_version": CLIENT_VERSION}, headers={"X-Bench-Session-Id": bench_session_id}, encrypt=True, ) if not out.get("success"): raise RuntimeError(f"API returned success=false: {out.get('message', out)}") data = out.get("data") if not isinstance(data, dict): raise RuntimeError(f"API data format error: expected an object, got {type(data).__name__}") questions = data.get("questions") if not isinstance(questions, list): raise RuntimeError("API data.questions is not a list") return { "questions": questions, "session_id": data.get("session_id", ""), "hash": data.get("hash", ""), "model_cost": data.get("model_cost"), } ``` ```python # scripts/agent_cli.py question = (task.get("question") or "").strip() if not question: return _fail("empty question") prompt_file_path = get_temp_file("prompt.md") with open(prompt_file_path, "w", encoding="utf-8") as _pf: _pf.write(question) session_id = f'{DEFAULT_SESSION_PREFIX}{task_id}' base = resolve_openclaw_cmd() cmd = base + [ "agent", "--session-id", session_id, "--message", f'Execute the instructions in the file: {prompt_file_path}', "--timeout", str(timeout_sec), ] proc = subprocess.Popen( cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True, encoding="utf-8", errors="replace", ) ``` ```python # scripts/main.py f ...[truncated 3301 chars]
Remediation
View remediation

T09 · Insecure Skill Coding Practices

Error
Location
scripts/main.py:263
Finding

Benchmark Results and Potentially Sensitive Agent Output Are Uploaded by Default

Content
View full analysis
bool: """ Whether leaderboard results and historical cached submissions are uploaded. caller_info.txt upload_to_server=false disables submission and cache flush. The default is true. """ raw = (caller.get("upload_to_server") or "true").strip().lower() return raw not in ("false", "no", "0") ``` ```python # scripts/main.py caller = _load_caller_info() upload_to_server = _upload_to_server_enabled(caller) if upload_to_server: try: flushed = flush_pending_uploads() if flushed: logger.info(f"Successfully uploaded {len(flushed)} cached records") except Exception as e: logger.warning(f"Failed to upload cached records: {e}") else: logger.info("upload_to_server=false: skipping cached uploads") ``` ```python # scripts/main.py # Upload results to the server unless upload_to_server is false. if upload_to_server: leaderboard = _upload_results(summary, bench_session_id, api_hash) else: logger.info("upload_to_server=false: skipping result upload") leaderboard = {} ``` ```python # scripts/server.py stdout_val = (r.get("stdout") or "") if len(stdout_val) > UPLOAD_STDOUT_TRUNCATE_LENGTH: stdout_val = stdout_val[:UPLOAD_STDOUT_TRUNCATE_LENGTH] + "…(truncated)" stdout_val = _sanitize_output(stdout_val) stderr_val = (r.get("stderr") or "") if len(stderr_val) > UPLOAD_STDERR_TRUNCATE_LENGTH: stderr_val = stderr_val[:UPLOAD_STDERR_TRUNCATE_LENGTH] + "…(truncated)" stderr_val = _sanitize_output(stderr_val) payload[f"r{idx}"] = { "start_time": _iso_time(r.get("start_time")) if r.get("start_time") else "", "end ...[truncated 3136 chars]
Remediation
View remediation

T05 · Unauthorized Access and Privilege Escalation

Error
Location
scripts/verification.py:84
Finding

Server-Controlled Verification Paths Can Read Files Outside the Benchmark Workspace

Content
View full analysis
str: return path.replace("\\", "/").lstrip("./") def resolve_file_paths(workspace_dir: str, file_path_spec: str) -> list[str]: """ Resolve a question-bank file_path into actual paths. """ spec = _normalize_rel_path(file_path_spec) if not _looks_like_regex(spec): return [os.path.join(workspace_dir, spec)] prefix_chars: list[str] = [] for ch in spec: if ch in _REGEX_META: break prefix_chars.append(ch) prefix = "".join(prefix_chars) search_dir_rel = os.path.dirname(prefix) if prefix else "" search_dir = os.path.join(workspace_dir, search_dir_rel) if not os.path.isdir(search_dir): search_dir = workspace_dir pattern = re.compile(spec) matches: list[str] = [] for abs_path in _iter_candidate_files(search_dir): rel = _normalize_rel_path(os.path.relpath(abs_path, workspace_dir)) if pattern.search(rel): matches.append(abs_path) return sorted(matches) ``` ```python # scripts/verification.py for extractor in metric_extractors: if not isinstance(extractor, dict): continue metric_name = str(extractor.get("metric_name") or "") target_file = str(extractor.get("target_file") or "") regex_pattern = str(extractor.get("regex") or "") value_type = str(extractor.get("type") or "string") if not metric_name or not target_file or not regex_pattern: # Error-result construction omitted. continue file_path = os.path.join(workspace_dir, target_file) if not os.path.isfile(file_path): # Error-result construction omitted. continue try: text = ...[truncated 2506 chars]
Remediation
View remediation
Vulnerability Patterns
  • Data ExfiltrationExternal Transmission, Env Variable Harvesting, File System Enumeration
  • Privilege EscalationExcessive Permissions, Sudo/Root Execution, Credential Access
  • Supply ChainUnpinned Dependencies, External Script Fetching, Obfuscated Code
  • Excessive AgencyUnrestricted Tool Access, Autonomous Decision Making, Scope Creep
  • System Prompt LeakageDirect Leakage, Indirect Extraction, Tool-Based Exfiltration
Findings (75)

Known Vulnerable Dependency: cryptography==46.0.5 — 11 advisory(ies): GHSA-537c-gmf6-5ccf (Vulnerable OpenSSL included in cryptography wheels); CVE-2026-69247 (cryptography: PKCS#7 EnvelopedData decryption exposes a Bleichenbacher oracle th); CVE-2026-69249 (python-cryptography: Duplicate self-signed intermediates can cause exponential p) +8 more

Critical
Category
Supply Chain
Confidence
91% confidence
Finding

The pinned dependency cryptography==46.0.5 is flagged with multiple advisories, including issues in bundled OpenSSL wheels and cryptographic side-channel/oracle risks. Because this is a benchmark/scoring tool that may process agent outputs, reports, network traffic, or signed/encrypted artifacts, using a known-vulnerable crypto library can undermine confidentiality, integrity, and trust in results.

Content

No source excerpt is available for this finding.

Tp4

High
Category
MCP Tool Poisoning
Confidence
97% confidence
Finding

A skill presented as a benchmark runner but actually inspecting/modifying session stores, deleting lock/session files, injecting messages via openclaw agent --deliver, and simulating progress materially changes the trust model. Those capabilities can interfere with platform state, tamper with user/agent sessions, or spoof benchmark progress/completion, which is dangerous even if not overtly malicious.

Content

No source excerpt is available for this finding.

Tp4

High
Category
MCP Tool Poisoning
Confidence
98% confidence
Finding

A skill presented as a benchmark runner but actually inspecting/modifying session stores, deleting lock/session files, injecting messages via openclaw agent --deliver, and simulating progress materially changes the trust model. Those capabilities can interfere with platform state, tamper with user/agent sessions, or spoof benchmark progress/completion, which is dangerous even if not overtly malicious.

Content

No source excerpt is available for this finding.

Tp4

High
Category
MCP Tool Poisoning
Confidence
98% confidence
Finding

A skill presented as a benchmark runner but actually inspecting/modifying session stores, deleting lock/session files, injecting messages via openclaw agent --deliver, and simulating progress materially changes the trust model. Those capabilities can interfere with platform state, tamper with user/agent sessions, or spoof benchmark progress/completion, which is dangerous even if not overtly malicious.

Content

No source excerpt is available for this finding.

Tp4

High
Category
MCP Tool Poisoning
Confidence
98% confidence
Finding

A skill presented as a benchmark runner but actually inspecting/modifying session stores, deleting lock/session files, injecting messages via openclaw agent --deliver, and simulating progress materially changes the trust model. Those capabilities can interfere with platform state, tamper with user/agent sessions, or spoof benchmark progress/completion, which is dangerous even if not overtly malicious.

Content

No source excerpt is available for this finding.

Credential Access

High
Category
Privilege Escalation
Confidence
70% confidence
Finding

Code accesses credential files (SSH keys, AWS credentials, etc.). This could indicate credential theft attempts.

Content

Scanner excerpt · UPLOAD_DISCLOSURE.md (reported line 32)May include surrounding context.

md
(re.compile(r"AIza[a-zA-Z0-9_\-]{35}"), "AIza***"),
    # AWS Access Key ID
    (re.compile(r"AKIA[A-Z0-9]{16}"), "AKIA***"),
    # GitHub Personal Access Token
    (re.compile(r"ghp_[a-zA-Z0-9]{36}"), "ghp_***"),
    # ClaWHub token
    (re.compile(r"clh_[a-zA-Z0-9]+"), "clh_***"),

Credential Access

High
Category
Privilege Escalation
Confidence
70% confidence
Finding

Code accesses credential files (SSH keys, AWS credentials, etc.). This could indicate credential theft attempts.

Content

Scanner excerpt · scripts/server.py (reported line 50)May include surrounding context.

python
(re.compile(r"AIza[a-zA-Z0-9_\-]{35}"), "AIza***"),
    # AWS Access Key ID
    (re.compile(r"AKIA[A-Z0-9]{16}"), "AKIA***"),
    # GitHub Personal Access Token
    (re.compile(r"ghp_[a-zA-Z0-9]{36}"), "ghp_***"),
    # ClaWHub token
    (re.compile(r"clh_[a-zA-Z0-9]+"), "clh_***"),

Credential Access

High
Category
Privilege Escalation
Confidence
60% confidence
Finding

Code accesses credential files (SSH keys, AWS credentials, etc.). This could indicate credential theft attempts.

Content

Scanner excerpt · UPLOAD_DISCLOSURE.md (reported line 129)May include surrounding context.

md
("本地路径 /root/",           "config at /root/.config/app.yaml",          "/root/***"),
        ("邮箱地址",                  "contact admin@example.com for help",        "***@***"),
        ("混合多条规则",
         "key=sk-abcdefghijklmnopqrstu path=/home/ci/.env email=foo@bar.com",
         None),  # None 表示只打印结果,不做单一断言
    ]

Credential Access

High
Category
Privilege Escalation
Confidence
60% confidence
Finding

Code accesses credential files (SSH keys, AWS credentials, etc.). This could indicate credential theft attempts.

Content

Scanner excerpt · scripts/server.py (reported line 511)May include surrounding context.

python
("本地路径 /root/",           "config at /root/.config/app.yaml",          "/root/***"),
        ("邮箱地址",                  "contact admin@example.com for help",        "***@***"),
        ("混合多条规则",
         "key=sk-abcdefghijklmnopqrstu path=/home/ci/.env email=foo@bar.com",
         None),  # None 表示只打印结果,不做单一断言
    ]

Direct Prompt Extraction

High
Category
System Prompt Leakage
Confidence
85% confidence
Finding

Skill contains instructions that could directly expose system prompts, internal rules, or hidden instructions to users or external parties.

Content

Scanner excerpt · scripts/verification.py (reported line 289)May include surrounding context.

python
target = ",".join(str(w) for w in required_words) if required_words else ""

    if not required_words:
        return RuleResult("keyword_match", target, True, score, score, "no keywords required")

    matched_words = [word for word in required_words if word in text]

Direct Prompt Extraction

High
Category
System Prompt Leakage
Confidence
85% confidence
Finding

Skill contains instructions that could directly expose system prompts, internal rules, or hidden instructions to users or external parties.

Content

Scanner excerpt · scripts/verification.py (reported line 299)May include surrounding context.

python
target = ",".join(str(w) for w in required_words) if required_words else ""

    if not required_words:
        return RuleResult("keyword_match", target, True, score, score, "no keywords required")

    matched_words = [word for word in required_words if word in text]

Direct Prompt Extraction

High
Category
System Prompt Leakage
Confidence
85% confidence
Finding

Skill contains instructions that could directly expose system prompts, internal rules, or hidden instructions to users or external parties.

Content

Scanner excerpt · scripts/verification.py (reported line 310)May include surrounding context.

python
target = ",".join(str(w) for w in required_words) if required_words else ""

    if not required_words:
        return RuleResult("keyword_match", target, True, score, score, "no keywords required")

    matched_words = [word for word in required_words if word in text]

Direct Prompt Extraction

High
Category
System Prompt Leakage
Confidence
85% confidence
Finding

Skill contains instructions that could directly expose system prompts, internal rules, or hidden instructions to users or external parties.

Content

Scanner excerpt · scripts/verification.py (reported line 315)May include surrounding context.

python
target = ",".join(str(w) for w in required_words) if required_words else ""

    if not required_words:
        return RuleResult("keyword_match", target, True, score, score, "no keywords required")

    matched_words = [word for word in required_words if word in text]

Direct Prompt Extraction

High
Category
System Prompt Leakage
Confidence
85% confidence
Finding

Skill contains instructions that could directly expose system prompts, internal rules, or hidden instructions to users or external parties.

Content

Scanner excerpt · scripts/verification.py (reported line 326)May include surrounding context.

python
target = ",".join(str(w) for w in required_words) if required_words else ""

    if not required_words:
        return RuleResult("keyword_match", target, True, score, score, "no keywords required")

    matched_words = [word for word in required_words if word in text]

Direct Prompt Extraction

High
Category
System Prompt Leakage
Confidence
85% confidence
Finding

Skill contains instructions that could directly expose system prompts, internal rules, or hidden instructions to users or external parties.

Content

Scanner excerpt · scripts/verification.py (reported line 332)May include surrounding context.

python
target = ",".join(str(w) for w in required_words) if required_words else ""

    if not required_words:
        return RuleResult("keyword_match", target, True, score, score, "no keywords required")

    matched_words = [word for word in required_words if word in text]

Direct Prompt Extraction

High
Category
System Prompt Leakage
Confidence
85% confidence
Finding

Skill contains instructions that could directly expose system prompts, internal rules, or hidden instructions to users or external parties.

Content

Scanner excerpt · scripts/verification.py (reported line 333)May include surrounding context.

python
target = ",".join(str(w) for w in required_words) if required_words else ""

    if not required_words:
        return RuleResult("keyword_match", target, True, score, score, "no keywords required")

    matched_words = [word for word in required_words if word in text]

Direct Prompt Extraction

High
Category
System Prompt Leakage
Confidence
85% confidence
Finding

Skill contains instructions that could directly expose system prompts, internal rules, or hidden instructions to users or external parties.

Content

Scanner excerpt · scripts/verification.py (reported line 341)May include surrounding context.

python
target = ",".join(str(w) for w in required_words) if required_words else ""

    if not required_words:
        return RuleResult("keyword_match", target, True, score, score, "no keywords required")

    matched_words = [word for word in required_words if word in text]

Direct Prompt Extraction

High
Category
System Prompt Leakage
Confidence
85% confidence
Finding

Skill contains instructions that could directly expose system prompts, internal rules, or hidden instructions to users or external parties.

Content

Scanner excerpt · scripts/verification.py (reported line 343)May include surrounding context.

python
target = ",".join(str(w) for w in required_words) if required_words else ""

    if not required_words:
        return RuleResult("keyword_match", target, True, score, score, "no keywords required")

    matched_words = [word for word in required_words if word in text]

Direct Prompt Extraction

High
Category
System Prompt Leakage
Confidence
85% confidence
Finding

Skill contains instructions that could directly expose system prompts, internal rules, or hidden instructions to users or external parties.

Content

Scanner excerpt · scripts/verification.py (reported line 351)May include surrounding context.

python
target = ",".join(str(w) for w in required_words) if required_words else ""

    if not required_words:
        return RuleResult("keyword_match", target, True, score, score, "no keywords required")

    matched_words = [word for word in required_words if word in text]

Undeclared Tool Scope

Medium
Category
MCP Least Privilege
Confidence
70% confidence
Finding

Without declared permissions the skill's intent is opaque and cannot be validated.

Content

No source excerpt is available for this finding.

Vague Triggers

Medium
Category
Not specified by scanner
Confidence
94% confidence
Finding

The Chinese trigger phrases include very broad everyday wording such as '跑分/跑个分', which may be interpreted in contexts unrelated to this skill. Because the skill is expensive, long-running, and network-active, ambiguous invocation materially raises the risk of unintended execution and unintended disclosure of benchmark artifacts.

Content

No source excerpt is available for this finding.

Vague Triggers

Medium
Category
Not specified by scanner
Confidence
95% confidence
Finding

Broad trigger phrases like 'run benchmark', 'get score', or general evaluation terms can cause unintended invocation of a long-running, networked, cost-incurring skill. In this skill's context, accidental activation is more dangerous because it may install packages, fetch remote question sets, consume millions of tokens, write files, and upload results by default unless the user explicitly opts out.

Content

No source excerpt is available for this finding.

Autonomous Decision Making

Medium
Category
Excessive Agency
Confidence
85% confidence
Finding

Skill enables autonomous high-impact decisions without human-in-the-loop verification. Critical operations (destructive commands, financial transactions, data deletion) should require explicit user confirmation.

Content

Scanner excerpt · scripts/agent_cli.py (reported line 50)May include surrounding context.

python
question_id = str(task.get("id") or "")
    answer = task.get("answer")
    if not isinstance(answer, dict):
        logger.debug(" [verify] no answer config, skip verification")
        return 0

    workspace_dir = os.path.join(get_openclaw_bot().openclaw_root, "workspace")

Missing User Warnings

Medium
Category
Not specified by scanner
Confidence
94% confidence
Finding

The code writes full task content to a temporary prompt.md file on disk without any user-facing disclosure or handling controls. Benchmark tasks may include proprietary prompts, secrets, or sensitive evaluation data, so persisting them locally increases the risk of unintended retention, exposure to other local processes, or later recovery from disk.

Content

No source excerpt is available for this finding.

Missing User Warnings

Medium
Category
Not specified by scanner
Confidence
92% confidence
Finding

The benchmark forwards task content to an external OpenClaw CLI subprocess without explicit disclosure or consent boundaries. This is dangerous because benchmark prompts may contain sensitive or customer-controlled data, and sending them to another tool may expand exposure, trigger remote model/API usage, or store additional copies in transcripts outside the user's expectations.

Content

No source excerpt is available for this finding.

Static analysis

No suspicious patterns detected.