claw-rl-prm-judge

• Outcome: Review
• Latest audit: Sep 5, 2026 · 2:29 PM UTC
• Version: 1.0.5

Overview

This skill is a coherent agent-evaluation helper, but it under-discloses the risk of sending raw agent traces and tool records to an LLM judge.

Review before installing. Use this only on redacted traces, avoid sending raw tool results or credentials to remote judge models, and prefer a local or privacy-controlled judge for sensitive work. Narrow or manually control activation if normal corrections should not trigger evaluation mode.

A.I.G

Vulnerability Patterns

• Insecure Skill Coding Practices: Finds exploitable flaws such as hardcoded secrets or command injection
• Skill Instruction Hijacking: Alters the agent's session goals or safety constraints when the skill loads
• Agent Memory Poisoning: Writes attacker-controlled rules into memory that affect later sessions
• Remote Payload Retrieval and Execution: Fetches external code whose behavior can change after review
• Embedded Malicious Code: Ships malicious scripts inside the skill and executes them locally

Findings (2)

T09 · Insecure Skill Coding Practices

Error

• Location: examples/judge-prompt.md:44
• Finding: Potential Disclosure of Sensitive Agent Trace Data to External LLM Judges## Vulnerability Details **File Location**: `examples/judge-prompt.md`, lines 44–64; supporting workflow instruction in `SKILL.md`, lines 66–70 **Vulnerability Type**: Sensitive-data exposure across an external trust boundary **Risk Level**: High ### Vulnerable Code Snippet ```text ## Per-turn user prompt template ```text [用户输入] {user_input} [Agent 输出] {agent_output} [工具调用记录] {tool_calls_json_array} [完成状态] {completion_status} [使用模型] {model_used} 请对该 agent turn 进行六维度 PRM 评估。 ``` ## Field semantics | Field | Format | Required | |-------|--------|----------| | `user_input` | string (raw) | yes | | `agent_output` | string (raw) | yes | | `tool_calls` | JSON array of `{tool, args, result, ts}` | optional (empty if no tool calls) | | `completion_status` | "success" \| "partial" \| "failed" \| "user_aborted" | yes | | `model_used` | string (model identifier) | yes | ``` The associated workflow in `SKILL.md` directs users to pass each agent turn through the judge and persist its results. ### Technical Analysis The canonical judge prompt accepts raw user input, raw agent output, and tool-call records containing tool names, arguments, and results. These fields can contain personal data, API credentials, authentication tokens, private file contents, internal URLs, or other confidential information. The prompt is advertised for use with any LLM and is calibrated for DeepSeek. If an externally hosted judge is used, the raw trace may cross the local trust boundary and become subject to the provider's logging, retention, training, or access policies. Although `references/storage-schema.md` requires redaction before persistence, the Skill does not impose equivalent redaction or data-minimization requirements before submitting traces to the judge. The behavior exceeds minimum-data privileges because scoring generally requires semantic summaries and relevant outcomes, not complete raw tool results or ...[truncated 1072 chars]
• Remediation: ## Remediation Suggestions 1. Add a mandatory preprocessing stage that detects and redacts credentials, tokens, cookies, authorization headers, personal data, and private file contents before constructing the judge prompt. 2. Replace complete tool arguments and results with minimized records such as tool name, argument categories, success status, and a sanitized outcome summary. 3. Default to a local judge for traces classified as confidential or containing tool output. 4. Require explicit user or administrator approval before sending raw traces to a remote provider. 5. Document the judge provider as a separate trust boundary, including retention, logging, training-use, residency, and deletion policies. 6. Apply allowlists to trace fields and size limits to prevent accidental submission of complete files or large datasets. 7. Enforce the same privacy rules before judgment as before persistence; redaction after remote processing is too late. 8. Add automated tests using synthetic API keys, personal data, and credential-bearing tool results to verify that sensitive values never reach the judge request.

T09 · Insecure Skill Coding Practices

Warning

• Location: examples/judge-prompt.md:44
• Finding: Indirect Prompt Injection Through Untrusted Agent Trace Fields## Vulnerability Details **File Location**: `examples/judge-prompt.md`, lines 44–58 **Vulnerability Type**: Untrusted content inserted directly into an LLM judge prompt **Risk Level**: Medium ### Vulnerable Code Snippet ```text ## Per-turn user prompt template ```text [用户输入] {user_input} [Agent 输出] {agent_output} [工具调用记录] {tool_calls_json_array} [完成状态] {completion_status} [使用模型] {model_used} 请对该 agent turn 进行六维度 PRM 评估。 ``` ``` ### Technical Analysis The template directly interpolates potentially attacker-controlled user input, agent output, and tool results into the judge's instruction context. The system prompt requires a JSON evaluation but does not explicitly state that all embedded trace content is untrusted evidence that must never be followed as instructions. Textual section labels are not a security boundary for an LLM. A malicious trace can contain instructions such as assigning perfect scores, ignoring the rubric, changing the output format, concealing a safety violation, or reproducing confidential context. Tool results are especially relevant because they may contain attacker-controlled web content or repository files that the original agent retrieved. Successful manipulation can corrupt evaluation scores and downstream reward signals. If scores are persisted or used for reinforcement learning, A/B decisions, regression gates, or autonomous strategy updates, one injected trace can influence later operational decisions. ### Attack Path 1. An attacker places judge-directed instructions in a user message, web page, repository file, or other content likely to appear in an agent trace. 2. The agent processes that content, and it becomes part of `user_input`, `agent_output`, or `tool_calls_json_array`. 3. The evaluator inserts the content verbatim into the canonical judge prompt. 4. The judge interprets the embedded text as instructions rather than inert evidence. 5. The judge return ...[truncated 702 chars]
• Remediation: ## Remediation Suggestions 1. Add a high-priority system instruction stating that every trace field is untrusted evidence and that instructions appearing inside those fields must never be executed or obeyed. 2. Serialize trace fields in a strict structured-data envelope rather than relying only on natural-language labels. 3. Use robust delimiters and escape or encode delimiter-like content inside trace values. 4. Validate the response against a strict JSON Schema, including allowed properties, numeric score ranges, enum values, and rejection of additional fields. 5. Reject or retry outputs containing prose, tool requests, unexpected keys, or non-numeric score values. 6. Add an independent rule-based layer for critical safety findings so a single LLM judgment cannot suppress them. 7. Detect common prompt-injection phrases in trace content and flag the evaluation as adversarial rather than silently trusting its score. 8. Add adversarial tests covering instructions embedded in user input, agent output, tool arguments, web content, and tool results. 9. Do not use judge output as the sole input to autonomous training, deployment, or guardrail modification.

SkillSpector

Vulnerability Patterns

• Data Exfiltration: External Transmission, Env Variable Harvesting, File System Enumeration
• Trigger Abuse: Overly Broad Trigger, Shadow Command Trigger, Keyword Baiting Trigger
• Prompt Injection: Instruction Override, Hidden Instructions, Exfiltration Commands
• Privilege Escalation: Excessive Permissions, Sudo/Root Execution, Credential Access
• Supply Chain: Unpinned Dependencies, External Script Fetching, Obfuscated Code

Findings (3)

Missing User Warnings

Medium

• Confidence: 91% confidence
• Finding: The README encourages users to submit agent traces for evaluation but does not warn that traces can contain sensitive user prompts, system instructions, API keys, file paths, tool outputs, or other confidential data. In a judging/evaluation skill, this omission can lead to accidental disclosure or inappropriate retention/sharing of sensitive information when operators paste full traces into the tool.

Vague Triggers

Medium

• Confidence: 95% confidence
• Finding: The trigger list includes very broad conversational phrases such as "that's not right" and "actually no," which are common in normal dialogue and can cause the skill to activate unintentionally. In this context, unintended activation could misroute conversations into agent-evaluation mode, create confusing behavior, and potentially interfere with higher-priority skills or normal assistant handling.

Missing User Warnings

Medium

• Confidence: 95% confidence
• Finding: The template explicitly sends raw user input, agent output, and tool call records to a separate judge model without any minimization, redaction, or privacy notice. Those fields can contain secrets, personal data, internal system prompts, or tool outputs, so this creates a real data-exposure risk whenever evaluation traffic is routed to third-party models or logged.

Static analysis

No suspicious patterns detected.