Llm Judge

• Outcome: Review
• Latest audit: Sep 12, 2026 · 8:45 AM UTC
• Version: 1.0.4

Overview

This skill is a coherent repo-comparison judge, but it tells agents to run untrusted repository test commands on the host without clear sandboxing or user approval.

Install only if you will use it in a hardened environment. Before running it on third-party, vendor, or PR repositories, disable automatic test execution or run tests in disposable containers with no secrets, no host write access beyond a scratch area, resource limits, and restricted networking. Expect it to write a JSON report under .beagle in the current workspace.

A.I.G

Vulnerability Patterns

• Skill Instruction Hijacking: Alters the agent's session goals or safety constraints when the skill loads
• Insecure Skill Coding Practices: Finds exploitable flaws such as hardcoded secrets or command injection
• Agent Memory Poisoning: Writes attacker-controlled rules into memory that affect later sessions
• Remote Payload Retrieval and Execution: Fetches external code whose behavior can change after review
• Embedded Malicious Code: Ships malicious scripts inside the skill and executes them locally

Findings (3)

T09 · Insecure Skill Coding Practices

Error

• Location: references/repo-agent.md:44

Finding

Untrusted Repository Test Execution Without Isolation

Content

View full analysis

Vulnerability Details

File Location: references/repo-agent.md:44-59
Vulnerability Type: Arbitrary code execution through untrusted test suites
Risk Level: High

Vulnerable Code

bash

cd $REPO_PATH

# Python
if [ -f pytest.ini ] || [ -f pyproject.toml ] || [ -d tests ]; then
  pytest --tb=short 2>&1
fi

# JavaScript/TypeScript
if [ -f package.json ]; then
  npm test 2>&1 || yarn test 2>&1
fi

# Go
if [ -f go.mod ]; then
  go test ./... 2>&1
fi

Technical Analysis

The Skill directs the agent to execute test commands from repositories supplied for evaluation. Test execution is not a passive operation:

• Pytest may execute repository-controlled conftest.py, plugins, imported modules, fixtures, and test collection hooks.
• npm test and yarn test invoke repository-controlled package scripts and may trigger lifecycle behavior.
• go test ./... compiles and runs repository-controlled Go test code.

The workflow does not require a sandbox, container, virtual machine, non-privileged account, read-only mount, network restriction, credential removal, or explicit user confirmation. Consequently, a repository submitted for comparison can cause arbitrary code to run with the permissions and environment of the agent process.

Attack Path

1. An attacker creates or modifies a repository that will be evaluated by the Skill.
2. The attacker adds a malicious test initialization mechanism, such as:
   • Python code in conftest.py;
   • a malicious test script in package.json; or
   • malicious code in a Go _test.go file.
3. The repository includes the corresponding marker file, such as pyproject.toml, package.json, or go.mod.
4. The user invokes the Skill to compare that repository with another implementation.
5. The Phase 1 agent follows the documented workflow and runs the repository's test command.
6. The attacker's code executes with the agent process's filesystem, environment, and network access.

Impact Assessment

...[truncated 642 chars]

Remediation

View remediation

Remediation Suggestions

• Never execute tests from an untrusted repository directly on the host.
• Run each repository in a fresh disposable container or virtual machine.
• Use a non-root user with no host-level privileges.
• Remove credentials, API tokens, SSH agents, cloud metadata access, and sensitive environment variables.
• Disable outbound networking by default; allow only explicitly required destinations.
• Mount repository source read-only and provide a separate disposable writable workspace.
• Apply strict CPU, memory, process, file-size, and execution-time limits.
• Avoid mounting host sockets or sensitive directories.
• Display the exact command and require explicit user approval before executing repository code.
• If secure isolation is unavailable, perform static inspection only and report that tests were not executed.

T09 · Insecure Skill Coding Practices

Error

• Location: references/repo-agent.md:26

Finding

Shell Command Injection Through Unquoted Repository Paths

Content

View full analysis

Vulnerability Details

File Location: references/repo-agent.md:26-44
Vulnerability Type: Shell command injection through unsafe variable expansion
Risk Level: High

Vulnerable Code

bash

# Get branch name
git -C $REPO_PATH rev-parse --abbrev-ref HEAD

# Get diff stats
git -C $REPO_PATH diff --stat main...HEAD

# Count files changed
git -C $REPO_PATH diff --name-only main...HEAD | wc -l

The same unsafe expansion is subsequently used when changing directories:

bash

cd $REPO_PATH

Technical Analysis

$REPO_PATH is expanded without quotation in shell command examples. Shell expansion performs word splitting and pathname expansion on unquoted values. At minimum, repository paths containing spaces or wildcard characters can change argument boundaries or cause unintended path expansion.

If these instructions are implemented by constructing shell command text from an insufficiently constrained path, shell metacharacters may also enable command injection. The initial validation shown elsewhere in the Skill quotes "$repo", but this does not make later unquoted expansions safe.

The risk is amplified because the repository path is an input to the Skill and the commands run with the agent's permissions.

Attack Path

1. An attacker influences the repository path supplied to the Skill.
2. The path contains whitespace, glob characters, or shell syntax that is unsafe in a generated shell command.
3. Input validation confirms that the path references a directory containing .git, but does not canonicalize it or establish safe later command construction.
4. The Phase 1 workflow substitutes the path into an unquoted command.
5. The shell splits or expands the value; where command strings are constructed unsafely, injected syntax can be interpreted as additional shell operations.
6. The unintended command runs with the privileges of the agent process.

Impact Assessment

Exploitation may cause incorrect repository selection, a ...[truncated 516 chars]

Remediation

View remediation

Remediation Suggestions

Quote every path expansion and terminate options where supported:

bash

git -C "$REPO_PATH" rev-parse --abbrev-ref HEAD
git -C "$REPO_PATH" diff --stat main...HEAD
git -C "$REPO_PATH" diff --name-only main...HEAD | wc -l
cd -- "$REPO_PATH"

Additional hardening should include:

• Use a structured process API that passes an argument array directly instead of invoking a shell.
• Canonicalize the path before use and verify that it remains within an approved workspace.
• Reject control characters and invalid path representations.
• Do not construct commands by concatenating user-controlled strings.
• Apply the same quoting requirements to SPEC_PATH, branch names, labels, and all other externally influenced values.
• Add automated tests using paths containing spaces, wildcard characters, leading hyphens, and shell metacharacters.

T01 · Skill Instruction Hijacking

Warning

• Location: SKILL.md:104

Finding

Prompt Injection Through Untrusted Specification and Repository-Derived Content

Content

View full analysis

Vulnerability Details

File Location: SKILL.md:104-143
Vulnerability Type: Instruction injection into delegated agent prompts
Risk Level: Medium

Vulnerable Code

text

You are a Phase 1 Repo Agent for the LLM Judge evaluation.

**Your Repo:** $LABEL at $REPO_PATH

**Spec Document:**
$SPEC_CONTENT

**Instructions:**
1. Load the **llm-judge** skill's references/repo-agent.md for detailed instructions
2. Follow references/fact-schema.md for the output format
3. Load the **llm-artifacts-detection** skill ([../../../beagle-core/skills/llm-artifacts-detection/SKILL.md](../../../beagle-core/skills/llm-artifacts-detection/SKILL.md), if available) for dead-code/overengineering analysis

Explore the repository and gather facts. Return ONLY valid JSON following the fact schema.

Do NOT score or judge. Only gather facts.

Repository-derived facts are later embedded directly into scoring prompts:

text

You are the $DIMENSION Judge for the LLM Judge evaluation.

**Spec Document:**
$SPEC_CONTENT

**Facts from all repos:**
$ALL_FACTS_JSON

**Instructions:**
1. Load the **llm-judge** skill's references/judge-agents.md for detailed instructions
2. Follow references/scoring-rubrics.md for the $DIMENSION rubric

Score each repo on $DIMENSION. Return ONLY valid JSON with scores and justifications.

Technical Analysis

The Skill directly interpolates a user-provided specification and repository-derived facts into subagent prompts. These values are untrusted data but share the same textual instruction channel as the Skill's operational directions.

An attacker can place instruction-like content in the specification or in repository artifacts that are summarized into the facts object. A delegated model may interpret this content as instructions rather than evidence. The documented JSON check only establishes parseability; it does not prove that the object conforms to the expected schema, reject unexpected fields, constrain field sizes, or neutrali ...[truncated 1413 chars]

Remediation

View remediation

Remediation Suggestions

• Explicitly label specifications, repository text, and facts as untrusted data that must never be followed as instructions.
• Pass untrusted content through structured data fields rather than concatenating it into an instruction string.
• Validate Phase 1 output against a strict JSON Schema, including required keys, exact types, allowed enum values, maximum lengths, and rejection of unknown properties.
• Constrain evidence fields to repository facts and require file-and-line citations that can be independently verified.
• Delimit untrusted blocks clearly and instruct subagents to treat everything inside those blocks only as quoted evidence.
• Do not include unnecessary secrets, credentials, or unrelated context in delegated prompts.
• Independently verify cited facts against repository files before scoring.
• Add adversarial tests containing prompt-injection phrases in specification text, filenames, comments, test output, and JSON string values.
• Apply output length limits and fail closed when schema validation or evidence verification fails.

SkillSpector

Vulnerability Patterns

• Excessive Agency: Unrestricted Tool Access, Autonomous Decision Making, Scope Creep
• MCP Tool Poisoning: Hidden Instructions, Unicode Deception, Parameter Description Injection
• Prompt Injection: Instruction Override, Hidden Instructions, Exfiltration Commands
• Data Exfiltration: External Transmission, Env Variable Harvesting, File System Enumeration
• Privilege Escalation: Excessive Permissions, Sudo/Root Execution, Credential Access

Findings (8)

Description-Behavior Mismatch

High

• Category: Not specified by scanner
• Confidence: 97% confidence

Finding

The referenced file defines a single-repository fact-gathering agent, while the skill manifest says this skill should judge and rank two or more implementations against a spec. This scope drift is dangerous because it can cause the agent to perform broader repository inspection and collection than users expect, increasing the chance of unintended access, misuse in unsupported workflows, and downstream decision-making based on outputs produced by the wrong phase of the system.

Content

No source excerpt is available for this finding.

Context-Inappropriate Capability

Medium

• Category: Not specified by scanner
• Confidence: 98% confidence

Finding

The instructions explicitly tell the agent to execute repository tests using project-local tooling such as pytest, npm, yarn, and go test. Running tests from an untrusted repository is active code execution and can trigger malicious setup scripts, test hooks, dependency scripts, network calls, filesystem modification, or credential exfiltration, making this substantially more dangerous than passive comparative judging.

Content

No source excerpt is available for this finding.

Description-Behavior Mismatch

Medium

• Category: Not specified by scanner
• Confidence: 89% confidence

Finding

These sections add standalone repository review capabilities such as security analysis, test assessment, overengineering review, and dead-code inspection, which exceed the manifest's narrower comparative-judging role. While not inherently malicious, this capability expansion increases the skill's effective authority and can lead to collection or analysis actions that operators did not intend when enabling a judging skill.

Content

No source excerpt is available for this finding.

Skill Enumeration

Medium

• Category: Agent Snooping
• Confidence: 80% confidence

Finding

Skill enumerates or reads other installed skills. Access to other skills' SKILL.md files or the skills directory reveals prompt instructions, capabilities, and secrets that should be invisible to peer skills.

Content

Scanner excerpt · SKILL.md (reported line 114)May include surrounding context.

md

### 5. Analyze Overengineering

Use patterns from the **llm-artifacts-detection** skill ([../../../../beagle-core/skills/llm-artifacts-detection/SKILL.md](../../../../beagle-core/skills/llm-artifacts-detection/SKILL.md)):
- Unnecessary abstractions (interfaces with single impl)
- Factory patterns for simple objects
- Excessive defensive coding

Skill Enumeration

Medium

• Category: Agent Snooping
• Confidence: 80% confidence

Finding

Skill enumerates or reads other installed skills. Access to other skills' SKILL.md files or the skills directory reveals prompt instructions, capabilities, and secrets that should be invisible to peer skills.

Content

Scanner excerpt · SKILL.md (reported line 241)May include surrounding context.

md

### 5. Analyze Overengineering

Use patterns from the **llm-artifacts-detection** skill ([../../../../beagle-core/skills/llm-artifacts-detection/SKILL.md](../../../../beagle-core/skills/llm-artifacts-detection/SKILL.md)):
- Unnecessary abstractions (interfaces with single impl)
- Factory patterns for simple objects
- Excessive defensive coding

Skill Enumeration

Medium

• Category: Agent Snooping
• Confidence: 80% confidence

Finding

Skill enumerates or reads other installed skills. Access to other skills' SKILL.md files or the skills directory reveals prompt instructions, capabilities, and secrets that should be invisible to peer skills.

Content

Scanner excerpt · references/repo-agent.md (reported line 87)May include surrounding context.

md

### 5. Analyze Overengineering

Use patterns from the **llm-artifacts-detection** skill ([../../../../beagle-core/skills/llm-artifacts-detection/SKILL.md](../../../../beagle-core/skills/llm-artifacts-detection/SKILL.md)):
- Unnecessary abstractions (interfaces with single impl)
- Factory patterns for simple objects
- Excessive defensive coding

Skill Enumeration

Medium

• Category: Agent Snooping
• Confidence: 80% confidence

Finding

Skill enumerates or reads other installed skills. Access to other skills' SKILL.md files or the skills directory reveals prompt instructions, capabilities, and secrets that should be invisible to peer skills.

Content

Scanner excerpt · references/repo-agent.md (reported line 110)May include surrounding context.

md

### 5. Analyze Overengineering

Use patterns from the **llm-artifacts-detection** skill ([../../../../beagle-core/skills/llm-artifacts-detection/SKILL.md](../../../../beagle-core/skills/llm-artifacts-detection/SKILL.md)):
- Unnecessary abstractions (interfaces with single impl)
- Factory patterns for simple objects
- Excessive defensive coding

Missing User Warnings

Low

• Category: Not specified by scanner
• Confidence: 91% confidence

Finding

This markdown skill instructs the agent to create .beagle and write .beagle/llm-judge-report.json, which changes the user's filesystem. The skill description does not explicitly warn the user that running it will create or overwrite files in the workspace.

Content

No source excerpt is available for this finding.

Static analysis

No suspicious patterns detected.