ClawHub
Back to skill

Security audit

SkillBench

Security checks for vulnerabilities and agentic risk

Overview

The skill is a coherent benchmark helper, but it installs an unpinned global CLI and documents automatic ClawVault syncing without enough data-control detail.

Install only if you are comfortable with a globally installed npm CLI and potential ClawVault sync of benchmark metadata. Prefer a pinned package version, inspect the CLI source/package provenance, and confirm sync can be disabled or reviewed before recording sensitive task names or errors.

A.I.G

Vulnerability Patterns
  • Insecure DependenciesIntroduces malicious components through unsafe dependency sources
  • Insecure Skill Coding PracticesFinds exploitable flaws such as hardcoded secrets or command injection
  • Skill Instruction HijackingAlters the agent's session goals or safety constraints when the skill loads
  • Agent Memory PoisoningWrites attacker-controlled rules into memory that affect later sessions
  • Remote Payload Retrieval and ExecutionFetches external code whose behavior can change after review
Findings (2)

T08 · Insecure Dependencies

Warning
Location
SKILL.md:7
Finding

Unpinned Global npm Dependency Prevents Verification of Executable Behavior

Content
View full analysis

Vulnerability Details

File Location: SKILL.md, lines 7-12 and 24-28
Vulnerability Type: Unpinned third-party dependency installed globally
Risk Level: Medium

Complete Code Snippet

yaml
requires:
  bins: [skillbench]
install:
  - id: node
    kind: node
    package: "@versatly/skillbench"
    bins: [skillbench]
    label: Install SkillBench CLI (npm)
markdown
## Installation

```bash
npm install -g @versatly/skillbench
text

### Technical Analysis

The skill delegates all executable behavior to the external npm package `@versatly/skillbench`. The package is installed globally without an exact version, integrity hash, lockfile, or included source code.

Because npm resolves the current package release at installation time, the code executed by users can differ from the code available when this skill was reviewed. Global installation also exposes the package's installation lifecycle scripts and executable code to the permissions of the invoking user. The supplied project contains no implementation with which to verify the package's behavior.

This is a supply-chain weakness rather than evidence that the currently published package is malicious.

### Attack Path

1. An attacker compromises the npm publisher account, package repository, or release process for `@versatly/skillbench`.
2. The attacker publishes a malicious release under the same package name.
3. A user follows the documented `npm install -g @versatly/skillbench` instruction without an exact version or integrity constraint.
4. npm resolves and installs the attacker-controlled release.
5. Package lifecycle scripts or the installed CLI execute with the invoking user's privileges.
6. The malicious package can access resources available to that user, subject to operating-system controls.

### Impact Assessment

Successful exploitation could permit arbitrary code execution with the privileges of the user performing the installation or invoking the CLI. Potentially exposed 
...[truncated 288 chars]
Remediation
View remediation

Remediation Suggestions

  • Pin the dependency to a reviewed exact version, such as @versatly/skillbench@X.Y.Z.
  • Publish and verify the expected npm integrity digest and package provenance.
  • Include a lockfile or equivalent immutable dependency manifest where the installation model permits it.
  • Avoid global installation; use a project-local dependency or a version-pinned npx invocation with appropriate safeguards.
  • Audit package lifecycle scripts and the CLI source before recommending installation.
  • Link to the authoritative source repository and reproducible build information.
  • Use automated dependency monitoring and require review before updating the pinned version.
  • Document that installation must occur under a non-privileged account.

T09 · Insecure Skill Coding Practices

Warning
Location
SKILL.md:144
Finding

Automatic External Synchronization Is Documented Without Clear Data-Control Boundaries

Content
View full analysis

Vulnerability Details

File Location: SKILL.md, lines 83-88 and 144-149
Vulnerability Type: Insufficiently documented external data synchronization
Risk Level: Medium

Complete Code Snippet

markdown
### Sync
```bash
skillbench sync --clawhub               # Import installed skills
skillbench sync --vault                 # Sync to ClawVault
skillbench sync --all                   # Everything
text

```markdown
## ClawVault Integration

Benchmarks sync to [ClawVault](https://clawvault.dev) automatically.

Technical Analysis

The document states that benchmark information synchronizes automatically with the external ClawVault service. Elsewhere, the documented benchmark records include task names, success or failure status, duration, and error types. Such metadata may reveal project activity, workflow characteristics, operational failures, or internal task descriptions.

The documentation does not define:

  • Whether automatic synchronization is enabled by default.
  • The exact fields included in synchronization.
  • The destination API endpoints.
  • When user consent is obtained.
  • How authentication credentials are stored.
  • Whether synchronization can be disabled.
  • Data retention, deletion, and redaction behavior.

The supplied project contains only SKILL.md, so the implementation and actual network behavior cannot be verified. The finding is therefore limited to the unsafe or ambiguous behavior explicitly documented; it does not establish that credentials or file contents are exfiltrated.

Attack Path

  1. A user installs and uses the SkillBench CLI.
  2. The user records benchmarks containing task names, durations, outcomes, or error classifications.
  3. According to the documentation, benchmark information synchronizes automatically with ClawVault.
  4. If synchronization occurs without an explicit confirmation boundary or adequate redaction, task metadata leaves the local environment.
  5. Anyone with authorized or compromised ac ...[truncated 633 chars]
Remediation
View remediation

Remediation Suggestions

  • Make remote synchronization explicitly opt-in and disabled by default.
  • Require informed confirmation before the first transmission.
  • Display the destination endpoint and exact data fields before synchronization.
  • Provide a documented local-only mode.
  • Allow synchronization to be disabled permanently through configuration.
  • Redact or reject secrets and sensitive text in task names and error details.
  • Minimize transmitted fields and use pseudonymous identifiers where possible.
  • Encrypt all synchronization traffic using current TLS configurations.
  • Store authentication tokens using an operating-system credential store rather than plaintext configuration.
  • Document retention, deletion, access control, and incident-response policies.
  • Add verbose or dry-run modes that show the proposed payload without transmitting it.
  • Include implementation source and tests demonstrating that unrelated files, environment variables, and credentials are never included.

SkillSpector

Vulnerability Patterns
  • Data ExfiltrationExternal Transmission, Env Variable Harvesting, File System Enumeration
  • Rogue AgentSelf-Modification, Session Persistence
  • Prompt InjectionInstruction Override, Hidden Instructions, Exfiltration Commands
  • Privilege EscalationExcessive Permissions, Sudo/Root Execution, Credential Access
  • Supply ChainUnpinned Dependencies, External Script Fetching, Obfuscated Code
Findings (2)

Self-Modification

High
Category
Rogue Agent
Confidence
85% confidence
Finding

Skill modifies its own code, configuration, or behavior at runtime. Self-modification enables an agent to escalate privileges, disable safety constraints, or install persistent backdoors.

Content

Scanner excerpt · SKILL.md (reported line 37)May include surrounding context.

  1. Do the task → tt start "Create PR" && ... && tt stop
  2. Record result → skillbench record "Create PR" --success
  3. Check scores → skillbench score github
  4. Improve skill → Update skill, bump version
  5. Repeat → Compare v1.0.0 vs v1.1.0
text

Missing User Warnings

Medium
Category
Not specified by scanner
Confidence
93% confidence
Finding

The skill states that benchmarks sync to ClawVault automatically but does not disclose what data is transmitted, when transmission occurs, or how users can opt out. Benchmark records may include task names, timing, failures, and other operational metadata that could expose sensitive workflow details if sent off-host without explicit consent.

Content

No source excerpt is available for this finding.

Static analysis

No suspicious patterns detected.