ClawHub
Back to skill

Security audit

TinkerClaw Memory Bench

Security checks for vulnerabilities and agentic risk

Overview

This skill is a disclosed memory benchmark that touches sensitive local memory data, but its risky actions are purpose-aligned, local by default, and gated before external sharing.

Install only if you are comfortable benchmarking a live OpenClaw memory database. Use --judge local and a copied database for the lowest-risk run; avoid --judge openai unless redacted memory excerpts may be sent to OpenAI, and review the full dry-run before any GitHub submission because a PR is public and includes aggregate report metadata. Prefer OPENAI_API_KEY over passing an API key on the command line.

A.I.G

Vulnerability Patterns
  • Insecure Skill Coding PracticesFinds exploitable flaws such as hardcoded secrets or command injection
  • Skill Instruction HijackingAlters the agent's session goals or safety constraints when the skill loads
  • Agent Memory PoisoningWrites attacker-controlled rules into memory that affect later sessions
  • Remote Payload Retrieval and ExecutionFetches external code whose behavior can change after review
  • Embedded Malicious CodeShips malicious scripts inside the skill and executes them locally
Findings (1)

T09 · Insecure Skill Coding Practices

Warning
Location
scripts/rate.py:519
Finding

OpenAI API Key Can Be Exposed Through Command-Line Arguments

Content
View full analysis
/cmdline`, or system monitoring software. - Agent execution transcripts and command logging. - CI/CD logs, terminal recording, and audit systems. - Wrapper scripts that retain the complete invocation. Although the key is not deliberately written into the generated benchmark report, accepting it as a process argument weakens the documentation’s assurance that the credential is never stored or logged. The safer environment-variable path already exists, but the insecure command-line path remains available and may be selected by users or automated instructions. ### Attack Path 1. A user or Agent invokes the benchmark with: ```bash python3 scripts/rate.py --judge openai --api-key sk-sensitive-value ``` 2. The complete command is retained in shell history, an Agent transcript, process metadata, or execution logs. 3. A local user, monitoring process, log reader, or compromised application with access to that data retrieves the API key. 4. The exposed credential is use ...[truncated 820 chars]
Remediation
View remediation

SkillSpector

Vulnerability Patterns
  • Data ExfiltrationExternal Transmission, Env Variable Harvesting, File System Enumeration
  • Privilege EscalationExcessive Permissions, Sudo/Root Execution, Credential Access
  • Excessive AgencyUnrestricted Tool Access, Autonomous Decision Making, Scope Creep
  • Tool MisuseTool Parameter Abuse, Chaining Abuse, Unsafe Defaults
  • Trigger AbuseOverly Broad Trigger, Shadow Command Trigger, Keyword Baiting Trigger
Findings (22)

Tp4

High
Category
MCP Tool Poisoning
Confidence
98% confidence
Finding

The description presents a much broader benchmarking/evaluation system with privacy/consent controls and submission workflow. This specific code chunk is only a collector for anonymized local statistics. While some privacy-related claims are directionally consistent (local operation, opt-in contributor name, opt-in token stats, no uploads in this script), the primary purpose and capabilities are materially narrower than described. It accesses local database/files and writes files despite no declared permissions. Therefore the declared description does not accurately represent what this code chunk actually does.

Content

No source excerpt is available for this finding.

Tp4

High
Category
MCP Tool Poisoning
Confidence
87% confidence
Finding

Most of the core description matches the code: it is local-first, redacts excerpts, computes IR metrics including nDCG/MAP/MRR with bootstrap CIs, supports ablations, and requires explicit terminal-gated consent before optional OpenAI transmission. However, there are material gaps. First, the code explicitly writes to the user's memory database by creating/migrating a retrieval_log table and storing one row per benchmark query; this is an undeclared capability relative to the stated no-leak/data-flow framing and empty permissions. Second, the description includes a separate confirmed submission/report-validation step, but that functionality is not present in this chunk. Third, despite declared permissions being empty, the code uses several resources: the local memory DB, localhost embedding service, optional external OpenAI API, and git metadata from the workspace. These differences are significant enough to mark as a mismatch, even though much of the benchmarking behavior is accurately represented.

Content

No source excerpt is available for this finding.

Tp4

High
Category
MCP Tool Poisoning
Confidence
96% confidence
Finding

The code is broadly related to the declared submission flow, and several claimed safeguards are indeed present: schema validation before upload, full preview of report and PR contents, dry-run support, typed confirmation or --yes with terminal requirement, and clear notice that publication is public. However, there is a material mismatch with the description's claim that submission 'identifies you only if you pass --contributor.' In this script, if no contributor argument is supplied, it extracts the contributor value from the report JSON and uses it for filenames, PR body, and index metadata; thus identity can still be published without explicitly passing --contributor. The code also performs a specific undeclared capability: it creates/updates a public aggregate INDEX.json summarizing all reports, including contributor and instance metadata. Overall, the script matches a public submission helper, but not all privacy/identification guarantees in the declared description are accurate.

Content

No source excerpt is available for this finding.

Tool Parameter Abuse

High
Category
Tool Misuse
Confidence
85% confidence
Finding

Tool parameters are crafted to achieve unintended or unsafe behavior. Parameter abuse can bypass intended safety checks (e.g. shell=True, --force, dangerous glob patterns).

Content

Scanner excerpt · SKILL.md (reported line 167)May include surrounding context.

python3 scripts/rate.py --db /tmp/bench.db --judge local

forget this installation ever ran

rm ~/.openclaw/workspace/db/.memory-bench-instance-id sqlite3 ~/.openclaw/workspace/db/memory.db 'DROP TABLE retrieval_log;'

text

Credential Access

High
Category
Privilege Escalation
Confidence
70% confidence
Finding

Code accesses credential files (SSH keys, AWS credentials, etc.). This could indicate credential theft attempts.

Content

Scanner excerpt · scripts/test_privacy.py (reported line 28)May include surrounding context.

python
SECRETS = (
    "Mail ops@example.com or ring +34 612 345 678. Key sk-abcdefghijklmnop1234 "
    "lives in /home/globalcaos/.config/secret.txt and the hash is "
    "deadbeefdeadbeefdeadbeefdeadbeefdeadbeef."
)

Env Variable Harvesting

High
Category
Data Exfiltration
Confidence
60% confidence
Finding

Code enumerates, copies, or searches environment variables for secrets. Bulk environment access can collect credentials unrelated to the skill's stated purpose.

Content

Scanner excerpt · scripts/test_privacy.py (reported line 250)May include surrounding context.

python
def _run_headless(cmd, cwd=None):
    """Run with no terminal on stdin and no route to the network."""
    env = dict(os.environ,
               OPENAI_API_KEY="sk-dummy-not-real",
               https_proxy="http://127.0.0.1:1",
               HTTPS_PROXY="http://127.0.0.1:1")

Env Variable Harvesting

High
Category
Data Exfiltration
Confidence
60% confidence
Finding

Code enumerates, copies, or searches environment variables for secrets. Bulk environment access can collect credentials unrelated to the skill's stated purpose.

Content

Scanner excerpt · scripts/test_privacy.py (reported line 299)May include surrounding context.

python
r = subprocess.run(["bash", str(SCRIPT_DIR / "submit.sh"), str(report), "--yes"],
                           capture_output=True, text=True, timeout=120,
                           stdin=subprocess.DEVNULL,
                           env=dict(os.environ, PATH=env_path))
        assert r.returncode != 0, f"unattended run was allowed to publish:\n{r.stdout}"
        assert "no terminal to confirm on" in r.stdout, r.stdout
        assert "SHIM-GH-REACHED" not in r.stdout, "execution reached the gh CLI"

Env Variable Harvesting

High
Category
Data Exfiltration
Confidence
60% confidence
Finding

Code enumerates, copies, or searches environment variables for secrets. Bulk environment access can collect credentials unrelated to the skill's stated purpose.

Content

Scanner excerpt · scripts/test_privacy.py (reported line 317)May include surrounding context.

python
r = subprocess.run(["bash", str(SCRIPT_DIR / "submit.sh"), str(report), "--dry-run"],
                           capture_output=True, text=True, timeout=120,
                           stdin=subprocess.DEVNULL,
                           env=dict(os.environ, PATH=f"{shim}:{os.environ['PATH']}"))
        assert r.returncode == 0, r.stdout + r.stderr
        assert "SHIM-GH-REACHED" not in r.stdout
        for path, _ in report_schema.walk(report_data):

Undeclared Tool Scope

Medium
Category
MCP Least Privilege
Confidence
97% confidence
Finding

The skill advertises significant capabilities including shell execution, file read/write, environment access, and optional network transmission, but it does not declare an explicit tool scope such as permissions or allowed-tools. That mismatch undermines least-privilege review and makes it harder for operators or automated policy systems to constrain what the skill is allowed to do before execution.

Content

No source excerpt is available for this finding.

subprocess module call

Medium
Category
Dangerous Code Execution
Confidence
70% confidence
Finding

subprocess module calls execute external commands. Without careful input validation, this enables command injection.

Content

Scanner excerpt · scripts/collect.py (reported line 43)May include surrounding context.

python
if not exe:
        return None
    try:
        out = subprocess.run([exe, "--version"], capture_output=True, text=True,
                             timeout=5, check=False, shell=False)
        return (out.stdout or "").strip() or None
    except Exception:

subprocess module call

Medium
Category
Dangerous Code Execution
Confidence
70% confidence
Finding

subprocess module calls execute external commands. Without careful input validation, this enables command injection.

Content

Scanner excerpt · scripts/rate.py (reported line 106)May include surrounding context.

python
try:
        import subprocess
        ws = Path.home() / ".openclaw" / "workspace"
        result = subprocess.run(
            ["git", "log", "-1", "--format=%h", "--", "skills/agent-memory-ultimate/"],
            capture_output=True, text=True, cwd=ws, timeout=5
        )

External Transmission

Medium
Category
Data Exfiltration
Confidence
91% confidence
Finding

The script can transmit benchmark queries and redacted memory excerpts to api.openai.com when --judge openai is selected. Although the code adds consent gates and best-effort redaction, it still exports potentially sensitive local memory data to a third party, and regex redaction cannot guarantee removal of all secrets or sensitive content.

Content

Scanner excerpt · scripts/rate.py (reported line 212)May include surrounding context.

python
content = str(content)
    return redact(content[:EXCERPT_CHARS])

OPENAI_ENDPOINT = "https://api.openai.com/v1/chat/completions"
OPENAI_MODEL = "gpt-4o-mini"

def build_openai_payload(query: str, result_content: str) -> dict:

Internal Network Request

Medium
Category
Server-Side Request Forgery
Confidence
70% confidence
Finding

Code issues a request to a loopback, link-local, or private-range host. This can reach internal services not meant to be exposed and is a common SSRF pivot.

Content

Scanner excerpt · scripts/rate.py (reported line 332)May include surrounding context.

python
try:
        import urllib.request
        data = json.dumps({"texts": [query, excerpt(result_content)]}).encode()
        req = urllib.request.Request(
            "http://127.0.0.1:8900/embed",
            data=data,
            headers={"Content-Type": "application/json"},

Unbounded Resource Access

Medium
Category
Excessive Agency
Confidence
75% confidence
Finding

Skill allows unbounded resource consumption (API calls, storage, compute). Without rate limits or quotas, a compromised or misbehaving agent can cause denial-of-service or cost overruns.

Content

Scanner excerpt · scripts/test_privacy.py (reported line 43)May include surrounding context.

python
def capture_request(payload_out, response):
    """Replace urlopen so nothing leaves the machine during a test."""
    def fake_urlopen(req, timeout=None):
        payload_out.append(req)
        return FakeResponse(json.dumps(response).encode())
    return fake_urlopen

External Transmission

Medium
Category
Data Exfiltration
Confidence
60% confidence
Finding

Data is being sent to an external URL. This could be legitimate telemetry or data exfiltration. Manual review is recommended.

Content

Scanner excerpt · scripts/test_privacy.py (reported line 127)May include surrounding context.

python
# Every line of the rendered body appears verbatim in the preview.
    for line in json.dumps(payload, indent=2, ensure_ascii=False).splitlines():
        assert line in preview, f"preview omits line: {line}"
    assert sent[0].full_url == rate.OPENAI_ENDPOINT == "https://api.openai.com/v1/chat/completions"

def test_preview_never_shows_the_api_key():
    preview = rate.render_openai_preview(rate.build_openai_payload("q", "c"), 30, True)

subprocess module call

Medium
Category
Dangerous Code Execution
Confidence
70% confidence
Finding

subprocess module calls execute external commands. Without careful input validation, this enables command injection.

Content

Scanner excerpt · scripts/test_privacy.py (reported line 161)May include surrounding context.

python
def _collect(tmp: Path) -> dict:
    """Run the real collector, so the schema is tested against real output."""
    out = tmp / "report.json"
    subprocess.run([sys.executable, str(SCRIPT_DIR / "collect.py"),
                    "--db", str(_fixture_db(tmp)), "--output", str(out)],
                   capture_output=True, check=True)
    return json.loads(out.read_text())

subprocess module call

Medium
Category
Dangerous Code Execution
Confidence
70% confidence
Finding

subprocess module calls execute external commands. Without careful input validation, this enables command injection.

Content

Scanner excerpt · scripts/test_privacy.py (reported line 254)May include surrounding context.

python
OPENAI_API_KEY="sk-dummy-not-real",
               https_proxy="http://127.0.0.1:1",
               HTTPS_PROXY="http://127.0.0.1:1")
    return subprocess.run(cmd, capture_output=True, text=True, timeout=120,
                          stdin=subprocess.DEVNULL, env=env, cwd=cwd)

def test_openai_judge_refuses_without_a_terminal_even_with_the_flag():

subprocess module call

Medium
Category
Dangerous Code Execution
Confidence
70% confidence
Finding

subprocess module calls execute external commands. Without careful input validation, this enables command injection.

Content

Scanner excerpt · scripts/test_privacy.py (reported line 296)May include surrounding context.

python
(shim / "gh").write_text('#!/bin/sh\necho SHIM-GH-REACHED\nexit 1\n')
        (shim / "gh").chmod(0o755)
        env_path = f"{shim}:{os.environ['PATH']}"
        r = subprocess.run(["bash", str(SCRIPT_DIR / "submit.sh"), str(report), "--yes"],
                           capture_output=True, text=True, timeout=120,
                           stdin=subprocess.DEVNULL,
                           env=dict(os.environ, PATH=env_path))

subprocess module call

Medium
Category
Dangerous Code Execution
Confidence
70% confidence
Finding

subprocess module calls execute external commands. Without careful input validation, this enables command injection.

Content

Scanner excerpt · scripts/test_privacy.py (reported line 314)May include surrounding context.

python
shim.mkdir()
        (shim / "gh").write_text('#!/bin/sh\necho SHIM-GH-REACHED\nexit 1\n')
        (shim / "gh").chmod(0o755)
        r = subprocess.run(["bash", str(SCRIPT_DIR / "submit.sh"), str(report), "--dry-run"],
                           capture_output=True, text=True, timeout=120,
                           stdin=subprocess.DEVNULL,
                           env=dict(os.environ, PATH=f"{shim}:{os.environ['PATH']}"))

subprocess module call

Medium
Category
Dangerous Code Execution
Confidence
70% confidence
Finding

subprocess module calls execute external commands. Without careful input validation, this enables command injection.

Content

Scanner excerpt · scripts/test_privacy.py (reported line 332)May include surrounding context.

python
shim.mkdir()
        (shim / "gh").write_text('#!/bin/sh\necho SHIM-GH-REACHED\nexit 1\n')
        (shim / "gh").chmod(0o755)
        r = subprocess.run(["bash", str(SCRIPT_DIR / "submit.sh"), str(report), "--dry-run"],
                           capture_output=True, text=True, timeout=120,
                           stdin=subprocess.DEVNULL,
                           env=dict(os.environ, PATH=f"{shim}:{os.environ['PATH']}"))

Vague Triggers

Medium
Category
Not specified by scanner
Confidence
89% confidence
Finding

This JSON file is a manifest-like test set, so SQP-1 applies. Entries such as "How should I handle sensitive data when sharing between agents?" are generic request phrasings rather than narrowly scoped invocation patterns, and the file provides no negative examples or activation constraints to distinguish these from ordinary conversation.

Content

No source excerpt is available for this finding.

Vague Triggers

Medium
Category
Not specified by scanner
Confidence
84% confidence
Finding

Phrases like "error handling best practices" are broad and resemble common everyday requests. Because the file does not specify that these are test fixtures only or provide exclusions, the trigger scope is ambiguous under the vague-trigger rule.

Content

No source excerpt is available for this finding.

Static analysis

Detected: suspicious.exposed_secret_literal

File appears to expose a hardcoded API secret or token.

Critical
Code
suspicious.exposed_secret_literal
Location
scripts/test_privacy.py:251