Back to skill

Security audit

lobster-rank

Security checks for vulnerabilities and agentic risk

Overview

This ranking skill is mostly purpose-aligned, but it scans local skill metadata and can send it with an API key to a configurable endpoint, which creates a real review concern.

Install only if you are comfortable sharing installed skill names, descriptions, structural metadata, heuristic signals, and log-availability status with the leaderboard service. Prefer running --dry-run first, avoid --api-url and LOBSTER_API_URL unless you fully trust the endpoint, and use a revocable API key.

Vulnerability Patterns
  • Insecure Skill Coding PracticesFinds exploitable flaws such as hardcoded secrets or command injection
  • Skill Instruction HijackingAlters the agent's session goals or safety constraints when the skill loads
  • Agent Memory PoisoningWrites attacker-controlled rules into memory that affect later sessions
  • Remote Payload Retrieval and ExecutionFetches external code whose behavior can change after review
  • Embedded Malicious CodeShips malicious scripts inside the skill and executes them locally
Findings (1)

T09 · Insecure Skill Coding Practices

Warning
Location
scripts/lobster_submit.py:82
Finding

API Key and Local Metadata Can Be Sent to an Arbitrary Endpoint

Content
View full analysis

Vulnerability Details

File Location: scripts/lobster_submit.py:82-88, scripts/lobster_submit.py:251-260, scripts/lobster_submit.py:350, scripts/lobster_submit.py:446
Vulnerability Type: Unvalidated network destination for authenticated requests
Risk Level: Medium

Complete Vulnerable Code

python
def resolve_api_url(cli_url: Optional[str]) -> str:
    if cli_url:
        return cli_url.rstrip("/")
    env = os.environ.get("LOBSTER_API_URL")
    if env:
        return env.rstrip("/")
    return DEFAULT_API_URL
python
def post_json(url: str, payload: dict, api_key: str) -> dict:
    body = json.dumps(payload).encode("utf-8")
    req = urllib.request.Request(
        url,
        data=body,
        headers={
            "Content-Type": "application/json",
            "X-Api-Key": api_key,
        },
        method="POST",
    )
python
api_url = resolve_api_url(args.api_url)
python
result = post_json(f"{api_url}/score/evaluate", payload, api_key)

The same endpoint resolution is also used by the confirmation workflow:

python
result = post_confirm(f"{api_url}/score/confirm", args.confirm, api_key)

Technical Analysis

The --api-url argument and LOBSTER_API_URL environment variable can replace the intended API endpoint with any URL. The value is only normalized by removing trailing slashes; the code does not:

  • Require HTTPS.
  • Restrict the hostname to lobster-rank.wondercv.com.
  • Reject embedded credentials, unexpected ports, or malformed destinations.
  • Prevent authentication data from being sent to an untrusted endpoint.
  • Explicitly constrain or revalidate redirect destinations.

After resolving the endpoint, post_json unconditionally places the Lobster API key in the X-Api-Key header. During evaluation, the request also contains the locally collected Skill inventory, descriptions, file counts, str ...[truncated 2300 chars]

Remediation
View remediation

Remediation Suggestions

  1. Use a fixed production endpoint

    • Remove --api-url and LOBSTER_API_URL from production distributions.
    • Send authenticated requests only to the declared production API.
  2. Allowlist trusted destinations if overrides are necessary

    • Parse the URL with urllib.parse.urlsplit.
    • Require the https scheme.
    • Require the exact hostname lobster-rank.wondercv.com.
    • Allow only the expected port.
    • Reject user information, fragments, malformed hosts, and unexpected path prefixes.
  3. Protect authentication across redirects

    • Disable automatic redirects for authenticated requests, or revalidate every redirect destination.
    • Never forward X-Api-Key to a destination with a different origin.
    • Reject HTTPS-to-HTTP redirect downgrades.
  4. Separate development behavior

    • If custom endpoints are needed for testing, require an explicit development-only switch.
    • Do not load endpoint overrides implicitly from the environment in normal operation.
    • Use separate, non-production credentials for development endpoints.
  5. Improve destination transparency

    • Display the validated hostname before transmitting data.
    • Require explicit user confirmation whenever a non-production development endpoint is selected.
    • Include the effective destination in dry-run output without exposing the API key.
  6. Reduce credential exposure

    • Prefer a permission-restricted credential store or environment variable over the documented --api-key argument.
    • Warn that command-line secrets may be exposed through shell history and local process inspection.
    • Support revocation and rotation of potentially disclosed API keys.
Vulnerability Patterns
  • Prompt InjectionInstruction Override, Hidden Instructions, Exfiltration Commands
  • Data ExfiltrationExternal Transmission, Env Variable Harvesting, File System Enumeration
  • Excessive AgencyUnrestricted Tool Access, Autonomous Decision Making, Scope Creep
  • Trigger AbuseOverly Broad Trigger, Shadow Command Trigger, Keyword Baiting Trigger
  • MCP Least PrivilegeUnderdeclared Capability, Wildcard Permission, Missing Permission Declaration
Findings (14)

Tp4

High
Category
MCP Tool Poisoning
Confidence
95% confidence
Finding

The declared purpose frames the skill as ranking installed skills, but the documented behavior expands into reading API keys from config/env, inspecting all local SKILL.md files heuristically, and checking log directories. That mismatch undermines informed consent and can cause users to authorize broader local inspection and metadata disclosure than they reasonably expected. Because results are sent to a publisher-operated remote server, the mismatch increases privacy and trust risk substantially.

Content

No source excerpt is available for this finding.

Ae1

High
Category
analysis-evasion
Confidence
100% confidence
Finding

Referenced artifact was not completely inspected

Content

Scanner excerpt · SKILL.md (reported line 41)May include surrounding context.

md
python3 scripts/lobster_submit.py

Ae1

High
Category
analysis-evasion
Confidence
100% confidence
Finding

Referenced artifact was not completely inspected

Content

Scanner excerpt · SKILL.md (reported line 47)May include surrounding context.

md
python3 scripts/lobster_submit.py

Ae1

High
Category
analysis-evasion
Confidence
100% confidence
Finding

Referenced artifact was not completely inspected

Content

Scanner excerpt · SKILL.md (reported line 53)May include surrounding context.

md
python3 scripts/lobster_submit.py

Ae1

High
Category
analysis-evasion
Confidence
100% confidence
Finding

Referenced artifact was not completely inspected

Content

Scanner excerpt · SKILL.md (reported line 59)May include surrounding context.

md
python3 scripts/lobster_submit.py

Ae1

High
Category
analysis-evasion
Confidence
100% confidence
Finding

Referenced artifact was not completely inspected

Content

Scanner excerpt · SKILL.md (reported line 81)May include surrounding context.

md
python3 scripts/lobster_submit.py

Ae1

High
Category
analysis-evasion
Confidence
100% confidence
Finding

Referenced artifact was not completely inspected

Content

Scanner excerpt · SKILL.md (reported line 116)May include surrounding context.

md
python3 scripts/lobster_submit.py

Undeclared Tool Scope

Medium
Category
MCP Least Privilege
Confidence
91% confidence
Finding

The skill declares access to environment, local files, and network endpoints but does not define an explicit tool/permission scope. That makes the effective authority of the skill less transparent to users and reviewers, increasing the risk of unintended data access or exfiltration when the skill is invoked. In this context, the skill scans local installations and sends data to a third-party server, so missing explicit scoping is materially relevant.

Content

No source excerpt is available for this finding.

Vague Triggers

Medium
Category
Not specified by scanner
Confidence
97% confidence
Finding

The trigger phrases are broad enough to match generic requests like rating, evaluating, or certifying an agent, which could cause the skill to activate in situations where the user did not intend a local scan and remote submission workflow. Over-broad invocation increases the chance of surprise data collection from local skill directories and transmission of metadata to a third-party server. The context makes this more dangerous because the skill reaches into local config and performs network submission.

Content

No source excerpt is available for this finding.

Scope Creep

Medium
Category
Not specified by scanner
Confidence
95% confidence
Finding

The manifest claims local file access is limited to openclaw.json, but the documentation instructs scanning skill directories and log locations as well. This discrepancy defeats least-privilege review and can hide the true extent of local inspection from users and platform policy enforcement. In a skill that inventories installed capabilities and contacts an external service, understated file-read scope is a meaningful security issue.

Content

No source excerpt is available for this finding.

Natural-Language Policy Violations

Medium
Category
Not specified by scanner
Confidence
96% confidence
Finding

The script's printed UI strings and grade labels are predominantly in Chinese, while the CLI description and usage are otherwise English-facing. This creates a natural-language locale policy concern because the file does not offer user opt-in, language selection, or any documented reason the tool must operate only in Chinese.

Content

No source excerpt is available for this finding.

Missing User Warnings

Medium
Category
Not specified by scanner
Confidence
95% confidence
Finding

The script collects local skill inventory data, descriptions parsed from SKILL.md, multi-model/routing heuristics derived from local files, and a logs-availability signal, then transmits them to a remote server once the command is run, without clearly presenting a pre-submit disclosure of exactly what will leave the machine. In a ranking skill, this behavior is somewhat aligned with the feature’s purpose, but the lack of explicit informed consent and data preview increases privacy and metadata-exfiltration risk, especially because descriptions and environment characteristics may reveal sensitive internal project details.

Content

No source excerpt is available for this finding.

Intent-Code Divergence

Low
Category
Not specified by scanner
Confidence
76% confidence
Finding

The privacy note presents a narrowly scoped local-read behavior focused on extracting apiKey from openclaw.json, but elsewhere the documentation says the scanner reads installed skill directories and logs to derive metadata and heuristic signals. This creates an intent mismatch in the inline documentation about what local data is actually accessed.

Content

No source excerpt is available for this finding.

Natural-Language Policy Violations

Low
Category
Not specified by scanner
Confidence
98% confidence
Finding

The file instructs the agent to ask the user a confirmation question in Chinese only: "要将这个成绩上传到排行榜吗?" This imposes a language choice without offering the user a locale option or documenting a justified region-specific constraint.

Content

No source excerpt is available for this finding.