Improvement Evaluator

• Outcome: Pass
• Latest audit: May 28, 2026 · 9:44 AM UTC
• Version: 1.1.1

Overview

This skill is a disclosed execution evaluator that runs task suites through Claude and optional pytest checks; its behavior matches its purpose but should be used with trusted inputs.

Install only if you intend to run execution-based evaluations. Use --mock for dry runs, review task suites and fixtures before running them, avoid putting secrets or private customer data in candidate skills or prompts, and run pytest-based judges in a sandbox when fixtures are not fully trusted.

SkillSpector

Vulnerability Patterns

• Data Exfiltration: External Transmission, Env Variable Harvesting, File System Enumeration
• Excessive Agency: Unrestricted Tool Access, Autonomous Decision Making, Scope Creep
• Output Handling: Unvalidated Output Injection, Cross-Context Output, Unbounded Output
• Trigger Abuse: Overly Broad Trigger, Shadow Command Trigger, Keyword Baiting Trigger
• Behavioral AST: exec() Call, eval() Call, Dynamic Import

Findings (9)

subprocess module call

Medium

• Category: Dangerous Code Execution

Content

output_file = Path(tmpdir) / "ai_output.txt"
            output_file.write_text(output, encoding="utf-8")

            result = subprocess.run(
                ["python3", "-m", "pytest", str(test_path), "-v",
                 "--tb=short", f"--rootdir={tmpdir}"],
                capture_output=True,

• Confidence: 88% confidence
• Finding: This code executes pytest in a subprocess against a test file selected from task data. Although it includes path-traversal checks, it still intentionally runs Python test code, and pytest test logic can execute arbitrary code during collection or execution. In the context of an evaluation skill, this makes task configuration and fixture contents a code-execution boundary, so compromise of test assets or untrusted task definitions can lead to arbitrary local execution.

subprocess module call

Medium

• Category: Dangerous Code Execution

Content

f"Respond with ONLY a JSON object: {{\"score\": <float>, \"reasoning\": \"<str>\"}}"
        )
        try:
            result = subprocess.run(
                ["claude", "-p", "--output-format", "json"],
                input=prompt,
                capture_output=True,

• Confidence: 91% confidence
• Finding: The skill invokes an external 'claude' CLI and sends it a prompt containing rubric text and up to 3000 characters of AI output. This creates a data egress boundary and delegates judgment to an external executable whose behavior, availability, and trust model are outside this file's control. If the output contains sensitive data or prompt-injection content, it may be exposed or influence scoring unexpectedly.

Context-Inappropriate Capability

Medium

• Confidence: 86% confidence
• Finding: The skill's stated purpose is evaluation, but this implementation achieves that by executing pytest-based test logic, which is effectively arbitrary Python code execution. In skill context, that is more dangerous because evaluators may be assumed to be passive/scoring-only components, while this one can run powerful code paths from test fixtures and plugin hooks.

Context-Inappropriate Capability

Medium

• Confidence: 84% confidence
• Finding: This judge expands the skill from local pass/fail evaluation into invocation of an external model CLI. That broadens the operational and trust boundary beyond the manifest description and can expose data or create non-deterministic behavior not obvious to users of the skill. The context increases concern because evaluators are often expected to be reproducible and self-contained.

Description-Behavior Mismatch

High

• Confidence: 97% confidence
• Finding: This task suite is materially misaligned with the declared purpose of the improvement-evaluator skill. Instead of measuring whether skill changes improve execution pass rate on predefined tasks, it evaluates a different skill's gate-ordering, decision logic, and review workflows, which can cause the evaluator to certify the wrong capabilities and produce misleading pass/fail evidence.

Context-Inappropriate Capability

Medium

• Confidence: 90% confidence
• Finding: The suite tests gate-validation and human-review CLI operations that are outside the stated scope of an execution-evaluation skill. This broadens the skill's apparent authority and can train or validate the agent to perform operational workflow actions unrelated to evaluation, increasing the chance of misuse, confused-deputy behavior, or over-privileged integrations being exercised under the wrong skill boundary.

Missing User Warnings

Medium

• Confidence: 90% confidence
• Finding: The code sends rubric content and AI output to an external CLI without any visible notice, consent, or redaction step. If evaluated outputs contain secrets, proprietary content, or personal data, this can result in unintended disclosure. In this skill context, outputs under evaluation may plausibly include exactly that kind of sensitive material.

Vague Triggers

Medium

• Confidence: 84% confidence
• Finding: The task suite repeatedly invokes `/tech-article` as a broad, unscoped command without documenting when it should or should not activate. In an agent environment, overly broad triggers can cause accidental routing to the wrong skill, mis-handle unrelated user input, or let prompt patterns unintentionally activate article-generation behavior in contexts where it is not appropriate.

Unvalidated Output Injection

High

• Category: Output Handling

Content

prompt_file = Path(tmpdir) / "prompt.txt"
            prompt_file.write_text(prompt, encoding="utf-8")

            result = subprocess.run(
                ["claude", "-p", "--output-format", "json"],
                input=prompt,
                capture_output=True,

• Confidence: 89% confidence
• Finding: The runner sends untrusted skill content and task prompt directly to an LLM subprocess, then treats the model output as authoritative input for downstream judging and storage. In this skill context, the whole purpose is to evaluate arbitrary skill documentation, so adversarial SKILL.md content can manipulate the model, exfiltrate sensitive context available to the CLI, or cause unsafe actions if the Claude CLI has tool, file, or network access.

Static analysis

No suspicious patterns detected.