TinkerClaw Memory Bench

• Outcome: Review
• Latest audit: Jun 6, 2026 at 5:02 PM UTC
• Version:
  2.0.1

Overview

This skill is a real memory benchmark, but its privacy claims understate that live memory snippets can be sent to OpenAI and benchmark metadata can be published to GitHub.

Review the scripts before installing. Use the local judge unless you are comfortable sending retrieved memory excerpts and benchmark queries to OpenAI. Do not run this against memory stores containing secrets, personal data, client data, or internal notes unless you first isolate a test corpus. Before running `submit.sh`, inspect the generated report because it can publish your contributor name, persistent instance ID, system details, and aggregate memory/retrieval statistics to GitHub.

SkillSpector

By NVIDIA

Vulnerability Patterns

• Data Exfiltration: External Transmission, Env Variable Harvesting, File System Enumeration
• Trigger Abuse: Overly Broad Trigger, Shadow Command Trigger, Keyword Baiting Trigger
• MCP Tool Poisoning: Hidden Instructions, Unicode Deception, Parameter Description Injection
• Prompt Injection: Instruction Override, Hidden Instructions, Exfiltration Commands
• Privilege Escalation: Excessive Permissions, Sudo/Root Execution, Credential Access

Findings (8)

Description-Behavior Mismatch

Medium

• Confidence: 94% confidence
• Finding: The script claims anonymized reporting, yet includes `contributor` populated from a CLI argument or `GITHUB_USER`, which directly identifies the submitting user. This creates a privacy mismatch that can deanonymize submissions and link benchmark data to a specific person or account, especially when combined with persistent instance tracking.

Description-Behavior Mismatch

High

• Confidence: 99% confidence
• Finding: The script sends both the benchmark query and up to 300 characters of retrieved memory content to the OpenAI API for scoring. That directly contradicts the skill description that says user data stays private and only aggregate statistics leave, creating a real confidentiality and transparency risk if memories contain sensitive or personal data.

Description-Behavior Mismatch

Medium

• Confidence: 92% confidence
• Finding: The benchmark includes prompts about family members, emotional context, dietary preferences, meetings, contacts, and relationship dynamics, which go beyond narrowly evaluating memory retrieval quality and can induce extraction or processing of highly sensitive personal data. In a live-memory benchmarking skill, these broad personal-life prompts increase the chance of exposing private memories to the evaluator or downstream submission pipeline, even if results are described as anonymized.

Missing User Warnings

Medium

• Confidence: 93% confidence
• Finding: The instructions recommend an `openai` judge without a clear warning that evaluation may send retrieved memory content or excerpts to an external provider. In the context of benchmarking an agent memory system, retrieved data can contain sensitive user, operational, or proprietary information, so omission of that warning materially increases the risk of unintended data exfiltration.

Missing User Warnings

Low

• Confidence: 88% confidence
• Finding: The submission step instructs users to run `submit.sh` without clearly warning that it performs networked GitHub actions and publishes the collected report in a branch/PR. Even if the report is intended to be anonymized, public or shared publication of benchmark artifacts and associated metadata can create privacy, attribution, and operational disclosure risks when users are not clearly informed.

Missing User Warnings

Medium

• Confidence: 97% confidence
• Finding: The external API call transmits user query text and retrieved memory excerpts without strong upfront disclosure or consent gating in the workflow. Because this skill operates on a live memory database, the context makes that disclosure gap more dangerous: the retrieved snippets could include secrets, internal notes, or personal data.

Missing User Warnings

Medium

• Confidence: 95% confidence
• Finding: This script uploads benchmark data and contributor metadata to GitHub by creating a pull request, including fields pulled directly from the report such as instance_id, memory statistics, and retrieval statistics. Although the skill description mentions anonymized submission, the script does not present an explicit consent prompt, preview, or redaction step before transmitting data, so users may unintentionally publish sensitive operational metadata.

Vague Triggers

Medium

• Confidence: 88% confidence
• Finding: Many queries are vague and open-ended, such as asking about recent family discussions, creative ideas, failures, goals, or security incidents, without limiting which memory namespace or data class may be searched. In a memory-benchmark context, this can cause overbroad retrieval, accidental disclosure of unrelated sensitive records, and inconsistent evaluation behavior because common language queries may match far more content than intended.

VirusTotal

VirusTotal findings are pending for this skill version.