Back to skill

Security audit

OpenClaw Smartness Eval

Security checks across malware telemetry and agentic risk

Overview

This skill is a disclosed local evaluation tool that reads OpenClaw runtime metrics, runs scoped test commands, and writes evaluation reports, with optional network judging only when explicitly enabled.

Install only in a trusted OpenClaw workspace. Review or redact local state files if they may contain sensitive prompts or operational data, keep generated reports private unless safe to share, and use --llm-judge only when you are comfortable sending score summaries, evidence metrics, and risk flags to the configured LLM provider.

SkillSpector

By NVIDIA
Vulnerability Patterns
  • Data ExfiltrationExternal Transmission, Env Variable Harvesting, File System Enumeration
  • Excessive AgencyUnrestricted Tool Access, Autonomous Decision Making, Scope Creep
  • Behavioral ASTexec() Call, eval() Call, Dynamic Import
  • Taint TrackingDirect Taint Flow, Variable-Mediated Taint Flow, Credential Exfiltration Chain
  • MCP Tool PoisoningHidden Instructions, Unicode Deception, Parameter Description Injection
Findings (10)

subprocess module call

Medium
Category
Dangerous Code Execution
Content
# ---------------------------------------------------------------------------

def run_cmd(command: list[str], timeout: int = 120) -> subprocess.CompletedProcess:
    return subprocess.run(command, cwd=str(WORKSPACE),
                          capture_output=True, text=True, timeout=timeout)
Confidence
86% confidence
Finding
return subprocess.run(command, cwd=str(WORKSPACE), capture_output=True, text=True, timeout=timeout)

Tainted flow: 'req' from os.environ.get (line 890, credential/environment) → urllib.request.urlopen (network output)

Critical
Category
Data Flow
Content
headers={'Content-Type': 'application/json', 'Authorization': f'Bearer {api_key}'},
    )
    try:
        with urllib.request.urlopen(req, timeout=15) as resp:
            body = json.loads(resp.read())
        text = body['choices'][0]['message']['content'].strip()
        if text.startswith('```'):
Confidence
96% confidence
Finding
with urllib.request.urlopen(req, timeout=15) as resp:

Description-Behavior Mismatch

Medium
Confidence
94% confidence
Finding
The skill description emphasizes smartness scoring/reporting, but the code also performs external API calls and consumes API keys. That mismatch increases supply-chain and privacy risk because operators may not expect network egress or secret usage from this skill. Hidden capability expansion is especially dangerous in agent ecosystems where skills are trusted based on metadata.

Context-Inappropriate Capability

Medium
Confidence
90% confidence
Finding
The evaluator reads broad workspace telemetry and operational artifacts, including logs, alerts, reasoning stores, benchmark history, and cron reports. Even if intended for scoring, this expands the data-access boundary far beyond narrow test execution and creates unnecessary exposure of potentially sensitive operational or user-derived information. In a shared workspace, this materially increases blast radius.

Missing User Warnings

Medium
Confidence
93% confidence
Finding
The guide instructs users to enable LLM Judge with third-party API keys but does not disclose that evaluation prompts, model outputs, or related data may be transmitted to external providers. This creates a real data-handling and privacy risk, especially if users run evaluations on sensitive prompts, reports, or internal benchmark material without understanding the external data flow.

Missing User Warnings

Medium
Confidence
94% confidence
Finding
The README prominently advertises evaluation on 'real runtime data' but does not clearly warn users that logs, telemetry, and agent traces may contain sensitive prompts, identifiers, operational metadata, or other private workspace data. In a skill intended to be run locally against agent state, this omission can lead users to execute the tool without informed consent about privacy exposure and persistence of analyzed data.

Missing User Warnings

Medium
Confidence
96% confidence
Finding
The Quick Start provides runnable commands without warning that execution may read existing local state and create persistent artifacts under state/smartness-eval, including reports and history files. Users may assume this is a harmless benchmark, when in fact it can process prior logs and leave additional copies of potentially sensitive evaluation results on disk.

Missing User Warnings

Medium
Confidence
97% confidence
Finding
The README enumerates data sources such as message logs, orchestrator logs, alerts, reflection reports, and a reasoning SQLite store, all of which can plausibly contain sensitive prompts, internal reasoning artifacts, credentials-adjacent metadata, or operational details. Listing these sources without an explicit sensitivity warning understates the privacy and security implications of running the skill in real environments.

Missing User Warnings

Medium
Confidence
95% confidence
Finding
The skill silently reads API keys and prepares outbound requests without any user-facing warning at the point of use. This can cause operators to unknowingly spend external API credits and transmit internal evaluation data to third parties. Lack of disclosure and consent makes the behavior risky even if technically optional.

Ssd 3

Medium
Confidence
91% confidence
Finding
The code samples real interaction logs and incorporates derived details such as intent distribution and high-risk interaction counts into reports and potentially into the external LLM judge prompt. Although it does not directly dump raw conversations here, it still republishes sensitive operational telemetry and may leak aspects of real user activity. In aggregate, this can expose confidential usage patterns.

VirusTotal

66/66 vendors flagged this skill as clean.

View on VirusTotal

Static analysis

Detected: suspicious.dynamic_code_execution

Dynamic code execution detected.

Critical
Code
suspicious.dynamic_code_execution
Location
scripts/eval.py:252