Security audit

PinchBench

Security checks across malware telemetry and agentic risk

Overview

PinchBench appears to be a real benchmark skill, but running it gives benchmark task files and OpenClaw agents enough authority to execute code, change local agent state, and upload detailed results.

Install or run this only in an isolated OpenClaw profile with disposable workspaces and test accounts. Review task markdown before running, avoid custom tasks unless you trust their automated checks, use --no-upload unless you intend to publish leaderboard data, and do not run it where OpenClaw has access to real email, calendars, credentials, or sensitive files.

SkillSpector

By NVIDIA

Vulnerability Patterns

Data ExfiltrationExternal Transmission, Env Variable Harvesting, File System Enumeration
Excessive AgencyUnrestricted Tool Access, Autonomous Decision Making, Scope Creep
Behavioral ASTexec() Call, eval() Call, Dynamic Import
Prompt InjectionInstruction Override, Hidden Instructions, Exfiltration Commands
Privilege EscalationExcessive Permissions, Sudo/Root Execution, Credential Access

Findings (15)

exec() call detected

High

Category: Dangerous Code Execution
Content: ) namespace: Dict[str, Any] = {} exec(grading_code, namespace) grade_func = namespace.get("grade") if not callable(grade_func): return GradeResult(
Confidence: 99% confidence
Finding: The code executes Python extracted from task metadata via exec() with no sandboxing, validation, or privilege restriction. If task definitions are attacker-controlled or even just untrusted benchmark content, this enables arbitrary code execution in the grader process, allowing file access, network activity, secret exfiltration, or system compromise.

Context-Inappropriate Capability

High

Confidence: 99% confidence
Finding: Automated grading accepts embedded Python from task.automated_checks and executes it directly, which is far broader than needed for benchmark scoring and creates a direct arbitrary-code-execution path. In a benchmarking skill, tasks may be externally supplied or updated over time, so compromising task content would compromise the grading environment.

Context-Inappropriate Capability

Medium

Confidence: 91% confidence
Finding: The upload payload includes extensive host metadata such as OS version, architecture, CPU model, memory characteristics, Python details, and a hostname-derived identifier. For a benchmark leaderboard upload, this exceeds minimally necessary data and increases device fingerprinting and privacy risk if the server is compromised, repurposed, or users are unaware of collection.

Context-Inappropriate Capability

Medium

Confidence: 88% confidence
Finding: The upload path runs local inspection commands to gather OpenClaw and platform state before sending results, which broadens the helper's behavior beyond merely uploading benchmark scores. In benchmark tooling, unexpected local probing is dangerous because it can leak additional environment details and violate user expectations about what the tool will access.

Missing User Warnings

Medium

Confidence: 90% confidence
Finding: The README states that benchmark results are auto-uploaded to a public leaderboard after registration, but it does not clearly warn users what data may be transmitted or that submissions become public. In a benchmarking skill that exercises real-world tasks involving email, files, research, and scheduling, results or metadata could unintentionally reveal sensitive prompts, outputs, environment details, or organizational information.

Missing User Warnings

Medium

Confidence: 90% confidence
Finding: The skill states that results are collected on a public leaderboard, but it does not prominently warn users what information may be published or transmitted alongside scores. In benchmarking contexts, result bundles can include prompts, outputs, model identifiers, timestamps, and host fingerprints, which can expose proprietary workflows or identifying metadata if uploaded by default.

Missing User Warnings

Medium

Confidence: 93% confidence
Finding: The script uploads benchmark results to a remote server by default unless `--no-upload` is specified, but it does not present a clear user-facing consent prompt or warning at the point of transmission. Because the results JSON includes task metadata, workspace paths, transcript length, and usage data, users may unintentionally transmit benchmark artifacts or potentially sensitive execution-derived information off-host.

Missing User Warnings

Medium

Confidence: 83% confidence
Finding: cleanup_agent_sessions deletes all matching session transcripts and the session index for an agent under ~/.openclaw without any confirmation, scope restriction beyond agent_id, backup, or retention policy. In a benchmarking context this can silently destroy logs needed for auditing, debugging, incident review, or evidence preservation, especially because it runs automatically before executions.

Missing User Warnings

Low

Confidence: 84% confidence
Finding: prepare_task_workspace writes files to paths derived from task.workspace_files into the resolved workspace without validating that the resulting destination stays inside that workspace. If task data is malicious or compromised, path traversal via file_spec['path'] or file_spec['dest'] could overwrite arbitrary files accessible to the process, made more concerning because the workspace may come from external OpenClaw configuration rather than a fresh temp directory.

Missing User Warnings

Medium

Confidence: 88% confidence
Finding: The LLM judge path summarizes transcript content and sends it to an external model endpoint via run_openclaw_prompt, potentially disclosing user prompts, tool outputs, and other sensitive task data. In a benchmark system that may process realistic email, calendar, research, or coding workflows, transcripts can easily contain confidential or proprietary information, making undisclosed third-party transmission a real privacy and data-governance risk.

Missing User Warnings

Medium

Confidence: 87% confidence
Finding: The main upload routine performs a network transmission of benchmark contents plus metadata without any visible user-facing disclosure or confirmation in this code path. In a benchmarking skill, silent exfiltration to a remote leaderboard is risky because task notes, frontmatter, timing, usage, and system metadata may contain sensitive operational details.

Missing User Warnings

Medium

Confidence: 90% confidence
Finding: The code stores the API token in a local JSON config file without setting restrictive file permissions or using an OS credential store. If the file is readable by other local users, backup systems, or malware, the token can be stolen and used to submit or manipulate leaderboard data under the victim's identity.

Missing User Warnings

Medium

Confidence: 94% confidence
Finding: The task explicitly instructs the agent to generate and potentially run code that makes an HTTP request to a workspace-provided endpoint, but it provides no guardrails about network safety, data handling, or approval requirements. In an agent benchmark context, this can normalize exfiltration-capable behavior and cause transmission of environment-derived URLs or future sensitive request metadata to external services.

Missing User Warnings

High

Confidence: 97% confidence
Finding: The task explicitly instructs the agent to write sensitive personal data and a secret code phrase into a persistent workspace file for later retrieval. Even in a benchmark context, this normalizes insecure handling of secrets, creates unnecessary retention of sensitive data, and can lead to later disclosure across sessions or to other tools, logs, or evaluators with workspace access.

Ssd 3

Medium

Confidence: 95% confidence
Finding: The benchmark requires the agent not only to store but also to later retrieve and disclose personal details and a secret phrase across sessions. This demonstrates cross-session persistence of sensitive data without consent, access controls, or data minimization, increasing the chance of unintended exposure if another session, user, or process can read the file.

VirusTotal

64/64 vendors flagged this skill as clean.

View on VirusTotal

Static analysis

Detected: suspicious.exposed_secret_literal

File appears to expose a hardcoded API secret or token.

Critical

Code: suspicious.exposed_secret_literal
Location: tasks/task_12_skill_search.md:35