Eval Driven Development

• Outcome: Review
• Latest audit: Sep 11, 2026 · 10:18 PM UTC
• Version: 0.1.11

Overview

This Python eval skill mostly matches its QA purpose, but it tells agents to automatically upgrade packages and reinstall the skill from remote sources before starting work.

Review this skill before installing. Its eval workflow is useful, but do not let it automatically upgrade pixie-qa or reinstall the skill unless you intentionally approve the exact source and version. Run it in a virtual environment or container, keep lockfiles under review, and treat pixie_qa traces and datasets as sensitive because they may contain prompts, user inputs, model outputs, and metadata.

A.I.G

Vulnerability Patterns

• Insecure Dependencies: Introduces malicious components through unsafe dependency sources
• Skill Instruction Hijacking: Alters the agent's session goals or safety constraints when the skill loads
• Agent Memory Poisoning: Writes attacker-controlled rules into memory that affect later sessions
• Remote Payload Retrieval and Execution: Fetches external code whose behavior can change after review
• Embedded Malicious Code: Ships malicious scripts inside the skill and executes them locally

Findings (1)

T08 · Insecure Dependencies

Warning

• Location: SKILL.md:14

Finding

Automatic Installation of Unpinned Dependencies and Skill Updates

Content

View full analysis

Vulnerability Details

File Location: SKILL.md:14-32
Vulnerability Type: Supply-chain risk from unpinned third-party installation and self-update instructions
Risk Level: Medium

Vulnerable Code Snippet:

markdown

## Startup checks (always first)

This skill uses the python `pixie-qa` package. Before doing anything else, perform these two steps:

### 1. Upgrade pixie-qa

Attempt to upgrade the `pixie-qa` package in the user's environment. Detect the package manager from the project (check for `uv.lock`, `poetry.lock`, `requirements.txt`, or a plain `pip` environment) and run the appropriate upgrade command:

- **uv**: `uv add pixie-qa --upgrade` (or `uv sync --upgrade-package pixie-qa`)
- **poetry**: `poetry add pixie-qa@latest`
- **pip**: `pip install --upgrade pixie-qa`

If the upgrade fails (e.g., no network, version conflict), log the error and continue — a failed upgrade must not block the rest of the skill.

### 2. Check skill version

Check whether a newer version of this skill itself is available by running the script `check_version.py`.

If there is a newer version, reinstall the skill with `npx skills add` before proceeding.

Technical Analysis

The skill requires an agent to upgrade pixie-qa to the latest available release before performing the requested work. The installation commands do not pin a reviewed version, require package hashes, validate publisher provenance, or otherwise constrain the installed artifact to content known at audit time.

The conditional self-update instruction has the same trust problem: it instructs the agent to reinstall the skill through npx skills add without pinning a repository commit or integrity digest. Consequently, the code ultimately installed and executed can differ from the code reviewed in this audit.

Python and Node package installation can execute package-controlled installation or runtime code with the permissions of the invoking u ...[truncated 1774 chars]

Remediation

View remediation

Remediation Suggestions

1. Remove automatic dependency upgrades from the mandatory startup workflow.
2. Require explicit user approval before changing the project's dependencies or reinstalling the skill.
3. Pin pixie-qa to a reviewed exact version rather than using --upgrade or @latest.
4. Preserve and honor the application's existing lockfile instead of rewriting dependency resolution automatically.
5. For pip, use a reviewed requirements file containing cryptographic hashes and install with --require-hashes.
6. Restrict installations to an approved package index and verify package publisher provenance where supported.
7. Pin skill updates to a reviewed repository commit or immutable release artifact and verify its checksum or signature before installation.
8. Run installation and evaluation in an isolated virtual environment or container with minimal filesystem and credential access.
9. Separate version checking from updating: report that an update is available, but do not install it automatically.
10. Record the exact installed package version and integrity information for reproducible audits.

SkillSpector

Vulnerability Patterns

• Prompt Injection: Instruction Override, Hidden Instructions, Exfiltration Commands
• Privilege Escalation: Excessive Permissions, Sudo/Root Execution, Credential Access
• Supply Chain: Unpinned Dependencies, External Script Fetching, Obfuscated Code
• Excessive Agency: Unrestricted Tool Access, Autonomous Decision Making, Scope Creep
• Trigger Abuse: Overly Broad Trigger, Shadow Command Trigger, Keyword Baiting Trigger

Findings (14)

Tp4

High

• Category: MCP Tool Poisoning
• Confidence: 99% confidence

Finding

The skill is presented as a QA/evaluation helper, but its instructions also include unrelated self-update, package upgrade, and remote-check behaviors. That mismatch can socially engineer an agent into performing risky environment and network actions that the user would not reasonably expect from the declared purpose.

Content

No source excerpt is available for this finding.

Credential Access

High

• Category: Privilege Escalation
• Confidence: 84% confidence

Finding

The skill instructs loading .env and checking for API keys as part of automated execution. Even without explicit exfiltration, encouraging agents to access credential-bearing files raises the risk of secret exposure in logs, traces, prompts, or follow-on tool calls, especially in an adversarial skill context.

Content

Scanner excerpt · SKILL.md (reported line 440)May include surrounding context.

md

"""
from pixie import enable_storage, flush

# Load .env if the project uses one for API keys
from dotenv import load_dotenv
load_dotenv()

Undeclared Tool Scope

Medium

• Category: MCP Least Privilege
• Confidence: 86% confidence

Finding

The skill instructs the agent to read project files and perform network/package operations, but it declares no explicit tool or permission scope. That creates an authorization gap where the agent may modify the environment or reach external resources without clear least-privilege boundaries, increasing the chance of unintended or excessive actions.

Content

No source excerpt is available for this finding.

Vague Triggers

Medium

• Category: Not specified by scanner
• Confidence: 95% confidence

Finding

The activation language is extremely broad, telling the agent to use this skill whenever a user is developing, testing, QA-ing, evaluating, or benchmarking a Python LLM project. Over-broad triggering increases the chance that risky instructions like package upgrades, credential checks, and environment modification run in contexts where the user did not specifically request them.

Content

No source excerpt is available for this finding.

Missing User Warnings

Medium

• Category: Not specified by scanner
• Confidence: 94% confidence

Finding

The skill directs the agent to upgrade/install packages and reinstall the skill itself before doing anything else, without first requiring a user warning or consent. Unprompted environment changes can break dependencies, alter project state, and expose the user to supply-chain compromise.

Content

No source excerpt is available for this finding.

Context-Inappropriate Capability

Medium

• Category: Not specified by scanner
• Confidence: 96% confidence

Finding

The skill explicitly tells the agent to check for a newer skill version and reinstall itself via npx skills add. Self-modifying or self-updating instructions are dangerous because they bypass stable review boundaries and can introduce unvetted instructions or code during execution.

Content

No source excerpt is available for this finding.

Rp1

Medium

• Category: MCP Rug Pull
• Confidence: 92% confidence

Finding

The self-update instruction uses npx skills add without pinning a version or integrity target. This enables supply-chain risk: the agent could fetch and install changed or malicious remote content at runtime, outside the user's reviewed dependency set.

Content

No source excerpt is available for this finding.

Behavior Manipulation

Medium

• Category: Prompt Injection
• Confidence: 75% confidence

Finding

Subtle instructions detected that may alter agent decision-making or introduce hidden biases.

Content

Scanner excerpt · SKILL.md (reported line 400)May include surrounding context.

md

- After instrumentation, call `flush()` at the end of runs to make sure all spans are written.
- For interactive apps (CLI loops, chat interfaces), instrument the **per-turn processing** function — the one that takes user input and produces a response. The eval `runnable` should call this same function.

**Important**: All pixie symbols are importable from the top-level `pixie` package. Never tell users to import from submodules (`pixie.instrumentation`, `pixie.evals`, `pixie.storage.evaluable`, etc.) — always use `from pixie import ...`.

---

Whitespace Padding

Medium

• Category: Prompt Injection
• Confidence: 70% confidence

Finding

Large whitespace padding was detected (a block of blank lines or a long run of spaces). This can push injected instructions below or to the right of the visible area so a human reviewer never sees them while the agent still reads them. Manual review of the hidden content is recommended.

Content

Scanner excerpt · references/pixie-api.md (reported line 23)May include surrounding context.

from pixie import enable_storage, observe, start_observation, flush, init, add_handler

text

| Function / Decorator | Signature                                                    | Notes                                                                                               |
| -------------------- | ------------------------------------------------------------ | --------------------------------------------------------------------------------------------------- |
| `enable_storage()`   | `() → StorageHandler`                                        | Idempotent. Creates DB, registers handler. Call at app startup.                                     |
| `init()`             | `(*, capture_content=True, queue_size=1000) → None`          | Called internally by `enable_storage`. Idempotent.                                                  |

Missing User Warnings

Medium

• Category: Not specified by scanner
• Confidence: 95% confidence

Finding

The documentation explicitly states that the observe decorator captures all keyword arguments as eval_input and the return value as eval_output, while other sections describe persistent storage to a local SQLite database and dataset files. In an LLM evaluation skill, those inputs and outputs commonly contain prompts, user data, secrets, API responses, or proprietary content, so documenting broad capture without a clear warning, minimization guidance, or redaction controls creates a realistic risk of sensitive data collection and retention.

Content

No source excerpt is available for this finding.

Description-Behavior Mismatch

Medium

• Category: Not specified by scanner
• Confidence: 97% confidence

Finding

The module docstring says it will 'check whether the qa-eval skill is outdated and update it if needed,' but the code only reads local metadata, fetches remote metadata, compares versions, and prints status. There is no code path that downloads, modifies, or updates the local skill, so the documented behavior materially overstates what the file actually does.

Content

No source excerpt is available for this finding.

Intent-Code Divergence

Medium

• Category: Not specified by scanner
• Confidence: 96% confidence

Finding

The docstring explicitly states the script will 'update it if needed,' yet every operation in the file is observational: reading SKILL.md, fetching remote text, comparing versions, and printing messages. This is an active contradiction between documentation and code behavior, not merely an omission.

Content

No source excerpt is available for this finding.

Skill Enumeration

Medium

• Category: Agent Snooping
• Confidence: 80% confidence

Finding

Skill enumerates or reads other installed skills. Access to other skills' SKILL.md files or the skills directory reveals prompt instructions, capabilities, and secrets that should be invisible to peer skills.

Content

Scanner excerpt · resources/check_version.py (reported line 12)May include surrounding context.

python

from urllib.request import urlopen

SKILL_REPO = "/yiouli/pixie-qa/"
SKILL_URL = f"https://raw.githubusercontent.com{SKILL_REPO}main/skills/qa-eval/SKILL.md"

_RE_FRONTMATTER = re.compile(r"^---\s*\n(.*?)\n---", re.DOTALL)
_RE_NAME = re.compile(r"^name:\s*(.+)$", re.MULTILINE)

Scope Creep

Low

• Category: Excessive Agency
• Confidence: 70% confidence

Finding

Skill's behavior or capabilities extend beyond its stated purpose. Scope creep allows an agent to perform actions unrelated to its documented functionality, increasing the attack surface.

Content

Scanner excerpt · SKILL.md (reported line 164)May include surrounding context.

md

1. **How the software runs**: What is the entry point? How do you start it? Is it a CLI, a server, a library function? What are the required arguments, config files, or environment variables?

2. **All inputs to the LLM**: This is not limited to the user's message. Trace every piece of data that gets incorporated into any LLM prompt:
   - User input (queries, messages, uploaded files)
   - System prompts (hardcoded or templated)
   - Retrieved context (RAG chunks, search results, database records)

Static analysis

No suspicious patterns detected.