Security audit

Skill-Eval

Security checks for vulnerabilities and agentic risk

Overview

The skill is a disclosed self-evolving evaluation engine, but it asks agents to execute untrusted skills, write persistent methodology files, and run local scripts without clear containment or review gates.

Install only in a controlled evaluation workspace with disposable outputs, no unrelated credentials, and clear human review before any knowledge-base, SKILL-EVAL.md, or SKILL.md changes are accepted. Treat evaluated skills as untrusted and avoid running the local script steps unless the repository contents are trusted.

A.I.G

Vulnerability Patterns

Skill Instruction HijackingAlters the agent's session goals or safety constraints when the skill loads
Agent Memory PoisoningWrites attacker-controlled rules into memory that affect later sessions
Insecure Skill Coding PracticesFinds exploitable flaws such as hardcoded secrets or command injection
Remote Payload Retrieval and ExecutionFetches external code whose behavior can change after review
Embedded Malicious CodeShips malicious scripts inside the skill and executes them locally

Findings (3)

T01 · Skill Instruction Hijacking

Error

Location: SKILL.md:216
Finding: Untrusted Skill Instructions Are Executed Without an Isolation Boundary
Content: View full analysis
] Read the skill at /SKILL.md and follow its instructions. Task: Save all outputs to: /iteration-//with_skill/outputs/ ``` ``` ### Technical Analysis The evaluated `SKILL.md` is untrusted input, but the evaluation subagent is explicitly instructed to read and follow it. The workflow does not establish an instruction hierarchy that prevents the evaluated Skill from overriding evaluation goals, requesting unrelated actions, or influencing safety constraints. No sandbox, network restriction, filesystem allowlist, tool allowlist, credential isolation, or output-path enforcement is specified. Consequently, a malicious Skill can embed instructions directing the subagent to access files, invoke available tools, transmit information, modify evaluation artifacts, or falsify results. The effective impact depends on the tools and permissions granted to the spawned subagent. The document does not itself prove that privileged tools are always available, but it fails to constrain any capabilities that are available. ### Attack Path 1. An attacker publishes or supplies a malicious Skill for evaluation. 2. The evaluation engine loads the attacker's `SKILL.md`. 3. The with-Skill subagent receives the instruction to follow that file. 4. Embedded instructions redirect the subagent from the intended evaluation task. 5. The subagent uses any available filesystem, command, or network capabilities to perform attacker-selected actions. 6. The malicious Skill may additionally direct the subagent to manipulate output files so the behavior appears legitimate or receives a favorable grade. ### Impact As ...[truncated 626 chars]
Remediation: View remediation

T02 · Agent Memory Poisoning

Error

Location: SKILL.md:309
Finding: Evaluation-Derived Content Can Poison Persistent Methodology and Future Runs
Content: View full analysis
.md** — Skill-specific notes for future re-evaluation Key questions for the learning step: - Which assertions discriminated well? (different pass rates with/without skill) - Which assertions were useless? (always pass or always fail regardless) - Were test prompts realistic enough? - Did the grading catch the right things? - What would we do differently next time? ### Phase 9: Absorb Knowledge into SKILL-EVAL.md **This is the critical closing step.** Without it, the engine documents lessons but doesn't actually evolve. After updating the knowledge files, review them and fold actionable improvements back into this document: 1. **failures.md -> Phases 1-5** — Each new failure mode should produce a concrete change to the relevant phase (pre-flight gates, assertion templates, scoring adjustments, benchmark annotations). 2. **lessons.md -> Phase 2** — Proven discriminating assertion patterns should be added to the assertion design guidance, not just noted in lessons. 3. **eval-patterns.md -> Phase 2** — New category-specific patterns should be reflected in the assertion guidance for that category. 4. **Verify absorption** — After updating, re-read the knowledge files and confirm every actionable item has a corresponding change in SKILL-EVAL.md. If something was noted but not absorbed, either absorb it or document why it was deferred. When enough knowledge accumulates, bump VERSION. The version bump signals that the methodology itself has changed, not just the knowledg ...[truncated 2701 chars]
Remediation: View remediation

T09 · Insecure Skill Coding Practices

Warning

Location: SKILL.md:78
Finding: External Model Processing Lacks Sensitive-Data Minimization and Redaction Controls
Content: View full analysis
.json` and the benchmark. Do not run the eval -- it will produce environment failures, not skill-quality signals. Re-evaluate after credential provisioning. - **Dependency matrix:** For data-fetch or finance skills, document the full dependency matrix (API key, freshness source, fallback behavior) before proceeding. ``` The execution and grading workflow at lines 216-256 sends target instructions and generated outputs through execution and judge models. ### Technical Analysis The Skill is designed to process target Skill content, user prompts, generated outputs, and grading evidence through configurable model providers. The documented providers include externally hosted model services. The workflow does not require: - Secret detection before model invocation. - Redaction of API keys, tokens, personal information, or proprietary code. - Data-classification checks. - User consent for third-party processing. - Provider-specific retention or training-policy validation. - Restrictions on what the judge and improvement models may receive. - A local-only execution option for sensitive evaluations. The text instructs the evaluator to log missing dependencies, not credential values. Therefore, the ...[truncated 1669 chars]
Remediation: View remediation

SkillSpector

Vulnerability Patterns

Data ExfiltrationExternal Transmission, Env Variable Harvesting, File System Enumeration
Rogue AgentSelf-Modification, Session Persistence
Prompt InjectionInstruction Override, Hidden Instructions, Exfiltration Commands
Privilege EscalationExcessive Permissions, Sudo/Root Execution, Credential Access
Supply ChainUnpinned Dependencies, External Script Fetching, Obfuscated Code

Findings (4)

Self-Modification

High

Category: Rogue Agent
Confidence: 94% confidence
Finding: The skill contains a built-in self-modification loop that instructs the agent to rewrite skill files and absorb learned changes back into the core methodology document. Self-rewriting behavior is dangerous because it enables persistent prompt/instruction drift, can propagate adversarial or low-quality changes across future runs, and can entrench unsafe behaviors without strong human review.
Content: Scanner excerpt · SKILL.md (reported line 383)May include surrounding context.
md
- If no matching pattern exists, design a new strategy and document rationale - If a similar improvement previously failed (per `knowledge/improve/failures.md`), try a different approach or document why this case is different 5. **Rewrite SKILL.md:** - Apply the selected strategy - Default formula: Remove > Add (delete 60-80% first, then add behavioral mandates) - Add specific, enforceable conventions (banned words, required sections, output schemas)

Ae1

High

Category: analysis-evasion
Confidence: 100% confidence
Finding: Referenced artifact was not completely inspected
Content: Scanner excerpt · SKILL.md (reported line 415)May include surrounding context.
md
1. Execute with `SKILL-improved.md` instead of original `SKILL.md`

Missing User Warnings

Medium

Category: Not specified by scanner
Confidence: 91% confidence
Finding: The skill instructs the agent to create and overwrite numerous files in the workspace (evals/*.json, grading.json, benchmark.json, skill cards, leaderboard output, knowledge files) without requiring explicit user consent or a dry-run/preview step. In an agent setting, undisclosed write behavior can cause unintended repository changes, data clobbering, or persistence of model-generated content that the user did not authorize.
Content: No source excerpt is available for this finding.

Missing User Warnings

Medium

Category: Not specified by scanner
Confidence: 89% confidence
Finding: The skill directs the agent to run local commands and scripts such as python scripts/generate_skill_card.py and python scripts/generate_leaderboard.py without an explicit warning, confirmation step, or trust boundary for those local files. If the repository or workspace is untrusted, this can lead to execution of attacker-controlled code, with consequences ranging from data exfiltration to arbitrary file modification.
Content: No source excerpt is available for this finding.

Static analysis

No suspicious patterns detected.