Skill-Eval

• Outcome: Review
• Latest audit: Sep 11, 2026 · 3:05 PM UTC
• Version: 0.4.0

Overview

The skill is a disclosed self-evolving evaluation engine, but it asks agents to execute untrusted skills, write persistent methodology files, and run local scripts without clear containment or review gates.

Install only in a controlled evaluation workspace with disposable outputs, no unrelated credentials, and clear human review before any knowledge-base, SKILL-EVAL.md, or SKILL.md changes are accepted. Treat evaluated skills as untrusted and avoid running the local script steps unless the repository contents are trusted.

A.I.G

Vulnerability Patterns

• Skill Instruction Hijacking: Alters the agent's session goals or safety constraints when the skill loads
• Agent Memory Poisoning: Writes attacker-controlled rules into memory that affect later sessions
• Insecure Skill Coding Practices: Finds exploitable flaws such as hardcoded secrets or command injection
• Remote Payload Retrieval and Execution: Fetches external code whose behavior can change after review
• Embedded Malicious Code: Ships malicious scripts inside the skill and executes them locally

Findings (3)

T01 · Skill Instruction Hijacking

Error

• Location: SKILL.md:216
• Finding: Untrusted Skill Instructions Are Executed Without an Isolation Boundary<![CDATA[ ## Vulnerability Details **File Location**: `SKILL.md`, lines 216-223 **Vulnerability Type**: Untrusted instruction execution **Risk Level**: High ### Vulnerable Code ```text **Single-model mode** (default): Spawn two subagents simultaneously on the same execution model: **With-skill subagent:** ``` [Model: <execution_model>] Read the skill at <skill-path>/SKILL.md and follow its instructions. Task: <prompt> Save all outputs to: <workspace>/iteration-<N>/<test-name>/with_skill/outputs/ ``` ``` ### Technical Analysis The evaluated `SKILL.md` is untrusted input, but the evaluation subagent is explicitly instructed to read and follow it. The workflow does not establish an instruction hierarchy that prevents the evaluated Skill from overriding evaluation goals, requesting unrelated actions, or influencing safety constraints. No sandbox, network restriction, filesystem allowlist, tool allowlist, credential isolation, or output-path enforcement is specified. Consequently, a malicious Skill can embed instructions directing the subagent to access files, invoke available tools, transmit information, modify evaluation artifacts, or falsify results. The effective impact depends on the tools and permissions granted to the spawned subagent. The document does not itself prove that privileged tools are always available, but it fails to constrain any capabilities that are available. ### Attack Path 1. An attacker publishes or supplies a malicious Skill for evaluation. 2. The evaluation engine loads the attacker's `SKILL.md`. 3. The with-Skill subagent receives the instruction to follow that file. 4. Embedded instructions redirect the subagent from the intended evaluation task. 5. The subagent uses any available filesystem, command, or network capabilities to perform attacker-selected actions. 6. The malicious Skill may additionally direct the subagent to manipulate output files so the behavior appears legitimate or receives a favorable grade. ### Impact As ...[truncated 626 chars]
• Remediation: <![CDATA[ ## Remediation Suggestions 1. Treat every evaluated Skill as hostile data rather than trusted governing instructions. 2. Run each evaluation in a disposable sandbox or isolated container. 3. Disable network access by default and permit only explicitly approved destinations when network access is essential to the test. 4. Expose only a strict allowlist of tools required for the individual test case. 5. Mount the target Skill and test fixtures read-only. 6. Restrict writes to a newly created per-run output directory and verify the resolved path before every write. 7. Remove credentials, SSH agents, cloud metadata access, API tokens, and unrelated environment variables from the subagent environment. 8. Add immutable higher-priority instructions stating that the target Skill cannot change the evaluator's security constraints, tool policy, output destination, or test objective. 9. Reject or manually review Skills that request sensitive-file access, unrelated command execution, credential use, persistence, or external communication. 10. Independently validate generated artifacts rather than trusting claims made by the evaluated Skill. ]]>

T02 · Agent Memory Poisoning

Error

• Location: SKILL.md:309
• Finding: Evaluation-Derived Content Can Poison Persistent Methodology and Future Runs<![CDATA[ ## Vulnerability Details **File Location**: `SKILL.md`, lines 309-336 **Vulnerability Type**: Persistent knowledge poisoning **Risk Level**: High ### Vulnerable Code ```text ### Phase 8: Learning (Self-Evolution) After each evaluation batch, update the knowledge base: 1. **lessons.md** — What worked? What didn't? New patterns discovered? 2. **eval-patterns.md** — New assertion templates for this skill category? 3. **failures.md** — New failure modes found? 4. **skill-profiles/<slug>.md** — Skill-specific notes for future re-evaluation Key questions for the learning step: - Which assertions discriminated well? (different pass rates with/without skill) - Which assertions were useless? (always pass or always fail regardless) - Were test prompts realistic enough? - Did the grading catch the right things? - What would we do differently next time? ### Phase 9: Absorb Knowledge into SKILL-EVAL.md **This is the critical closing step.** Without it, the engine documents lessons but doesn't actually evolve. After updating the knowledge files, review them and fold actionable improvements back into this document: 1. **failures.md -> Phases 1-5** — Each new failure mode should produce a concrete change to the relevant phase (pre-flight gates, assertion templates, scoring adjustments, benchmark annotations). 2. **lessons.md -> Phase 2** — Proven discriminating assertion patterns should be added to the assertion design guidance, not just noted in lessons. 3. **eval-patterns.md -> Phase 2** — New category-specific patterns should be reflected in the assertion guidance for that category. 4. **Verify absorption** — After updating, re-read the knowledge files and confirm every actionable item has a corresponding change in SKILL-EVAL.md. If something was noted but not absorbed, either absorb it or document why it was deferred. When enough knowledge accumulates, bump VERSION. The version bump signals that the methodology itself has changed, not just the knowledg ...[truncated 2701 chars]
• Remediation: <![CDATA[ ## Remediation Suggestions 1. Never automatically convert evaluation-derived text into governing instructions. 2. Require explicit human approval for changes to `SKILL-EVAL.md`, model configuration, registries, or reusable improvement rules. 3. Record provenance for every learned item, including the source Skill, run identifier, model, prompt, evidence, and reviewer. 4. Store untrusted observations as quoted, non-executable data in a separate repository or schema. 5. Reject imperative instructions, tool commands, external links, and policy-changing text extracted from evaluated content. 6. Require corroboration across multiple independent Skills or controlled test cases before promoting an observation into reusable guidance. 7. Apply code review and signed commits to methodology and persistent knowledge changes. 8. Protect governing files with integrity checks and restore them from a trusted baseline after each untrusted evaluation. 9. Separate per-Skill profiles from global knowledge; do not promote profile content automatically. 10. Add rollback support and maintain an auditable history of every accepted methodology change. 11. Use a trusted deterministic transformation process for proposed changes, followed by security review, instead of allowing a model to rewrite its own governing document directly. ]]>

T09 · Insecure Skill Coding Practices

Warning

• Location: SKILL.md:78
• Finding: External Model Processing Lacks Sensitive-Data Minimization and Redaction Controls<![CDATA[ ## Vulnerability Details **File Location**: `SKILL.md`, lines 78-89 **Vulnerability Type**: Uncontrolled external processing of evaluation data **Risk Level**: Medium ### Vulnerable Code ```json { "execution_models": [ "anthropic/claude-opus-4-6", "openai/gpt-4.1", "google/gemini-2.5-pro" ], "judge_model": "anthropic/claude-opus-4-6", "improvement_model": "anthropic/claude-opus-4-6", "default_execution_model": "anthropic/claude-opus-4-6" } ``` Related instructions at lines 146-148 state: ```text 3. **Dependency check** — required CLI tools, API keys, env vars. Log any that are missing. - **Dependency-gated skills:** If a skill requires paid APIs or credentials that aren't available, mark it as `dependency-gated` in `evals/<slug>.json` and the benchmark. Do not run the eval -- it will produce environment failures, not skill-quality signals. Re-evaluate after credential provisioning. - **Dependency matrix:** For data-fetch or finance skills, document the full dependency matrix (API key, freshness source, fallback behavior) before proceeding. ``` The execution and grading workflow at lines 216-256 sends target instructions and generated outputs through execution and judge models. ### Technical Analysis The Skill is designed to process target Skill content, user prompts, generated outputs, and grading evidence through configurable model providers. The documented providers include externally hosted model services. The workflow does not require: - Secret detection before model invocation. - Redaction of API keys, tokens, personal information, or proprietary code. - Data-classification checks. - User consent for third-party processing. - Provider-specific retention or training-policy validation. - Restrictions on what the judge and improvement models may receive. - A local-only execution option for sensitive evaluations. The text instructs the evaluator to log missing dependencies, not credential values. Therefore, the ...[truncated 1669 chars]
• Remediation: <![CDATA[ ## Remediation Suggestions 1. Add mandatory secret and personal-data scanning before every model invocation. 2. Redact API keys, access tokens, passwords, private keys, session cookies, connection strings, and sensitive environment values. 3. Prohibit logging credential values; record only the dependency name and an availability boolean. 4. Require explicit authorization before sending project content to an external provider. 5. Provide a local-model or offline mode for confidential evaluations. 6. Minimize model inputs so each model receives only the content required for its role. 7. Do not automatically send full execution outputs to judge or improvement models; extract and sanitize the necessary evidence first. 8. Maintain provider allowlists and verify retention, training-use, regional-processing, and deletion policies. 9. Use separate short-lived credentials scoped to model invocation only. 10. Disable network access for evaluated Skills unless a test explicitly requires it. 11. Add output scanning between phases so a malicious Skill cannot place sensitive data into artifacts that are later transmitted. 12. Document data flows and retention periods for execution, judge, and improvement providers. ]]>

SkillSpector

Vulnerability Patterns

• Data Exfiltration: External Transmission, Env Variable Harvesting, File System Enumeration
• Rogue Agent: Self-Modification, Session Persistence
• Prompt Injection: Instruction Override, Hidden Instructions, Exfiltration Commands
• Privilege Escalation: Excessive Permissions, Sudo/Root Execution, Credential Access
• Supply Chain: Unpinned Dependencies, External Script Fetching, Obfuscated Code

Findings (4)

Self-Modification

High

• Category: Rogue Agent

Content

- If no matching pattern exists, design a new strategy and document rationale
   - If a similar improvement previously failed (per `knowledge/improve/failures.md`), try a different approach or document why this case is different

5. **Rewrite SKILL.md:**
   - Apply the selected strategy
   - Default formula: Remove > Add (delete 60-80% first, then add behavioral mandates)
   - Add specific, enforceable conventions (banned words, required sections, output schemas)

• Confidence: 94% confidence
• Finding: The skill contains a built-in self-modification loop that instructs the agent to rewrite skill files and absorb learned changes back into the core methodology document. Self-rewriting behavior is dangerous because it enables persistent prompt/instruction drift, can propagate adversarial or low-quality changes across future runs, and can entrench unsafe behaviors without strong human review.

Ae1

High

• Category: analysis-evasion

Content

1. Execute with `SKILL-improved.md` instead of original `SKILL.md`

• Confidence: 100% confidence
• Finding: Referenced artifact was not completely inspected

Missing User Warnings

Medium

• Confidence: 91% confidence
• Finding: The skill instructs the agent to create and overwrite numerous files in the workspace (`evals/*.json`, `grading.json`, `benchmark.json`, skill cards, leaderboard output, knowledge files) without requiring explicit user consent or a dry-run/preview step. In an agent setting, undisclosed write behavior can cause unintended repository changes, data clobbering, or persistence of model-generated content that the user did not authorize.

Missing User Warnings

Medium

• Confidence: 89% confidence
• Finding: The skill directs the agent to run local commands and scripts such as `python scripts/generate_skill_card.py` and `python scripts/generate_leaderboard.py` without an explicit warning, confirmation step, or trust boundary for those local files. If the repository or workspace is untrusted, this can lead to execution of attacker-controlled code, with consequences ranging from data exfiltration to arbitrary file modification.

Static analysis

No suspicious patterns detected.