agent-eval-harness

• Outcome: Review
• Latest audit: Aug 14, 2026 · 1:46 AM UTC
• Version: 1.0.0

Overview

The evaluation harness is mostly local and non-destructive, but it also includes a generic persistent learning module that stores user preferences and usage history beyond the skill's stated regression-testing purpose.

Install only if you are comfortable with a local skill storing regression baselines plus usage history, errors, notes, and preferences in plaintext. Treat the learning module as optional, avoid recording sensitive prompts or private operational details, and review or delete learned_patterns.json if you do not want retained history.

SkillSpector

Vulnerability Patterns

• Data Exfiltration: External Transmission, Env Variable Harvesting, File System Enumeration
• Excessive Agency: Unrestricted Tool Access, Autonomous Decision Making, Scope Creep
• MCP Least Privilege: Underdeclared Capability, Wildcard Permission, Missing Permission Declaration
• MCP Tool Poisoning: Hidden Instructions, Unicode Deception, Parameter Description Injection
• Prompt Injection: Instruction Override, Hidden Instructions, Exfiltration Commands

Findings (10)

Lp3

Medium

• Category: MCP Least Privilege
• Confidence: 90% confidence
• Finding: The skill documentation instructs the use of scripts that persist data to files such as regression baselines and learned_patterns.json, but the manifest does not declare corresponding file-write capabilities. This creates a transparency and policy-enforcement gap: operators may approve the skill as low-risk while it can still store state locally, including potentially sensitive usage or preference data.

Tp4

High

• Category: MCP Tool Poisoning
• Confidence: 97% confidence
• Finding: The stated purpose is a regression-evaluation harness, but the documentation also embeds a generic cross-skill learning and memory subsystem that records failures, preferences, and operational history. This mismatch is dangerous because it hides materially different behavior—persistent tracking and profiling—behind a narrower evaluation label, reducing informed consent and increasing the chance of unauthorized data collection.

Description-Behavior Mismatch

Medium

• Confidence: 91% confidence
• Finding: A regression test harness has a limited, well-scoped purpose, but this section adds a generic self-learning memory system unrelated to core evaluation. Expanding scope in documentation like this increases attack surface and reviewer confusion, and can be used to justify persistent collection or modification behaviors that would not normally be approved for a test harness.

Context-Inappropriate Capability

Medium

• Confidence: 95% confidence
• Finding: The documentation authorizes persistent storage of user preferences, usage frequency, error history, and generated improvement suggestions, none of which is justified by the stated function of a regression harness. In context, this makes the skill more dangerous because the extra tracking is orthogonal to testing and could enable unnecessary profiling across sessions or across unrelated skills.

Description-Behavior Mismatch

High

• Confidence: 95% confidence
• Finding: The module is explicitly designed as a generic cross-skill self-improvement component that any skill can call, which exceeds the stated purpose of an evaluation-harness skill. In this context, scope creep is security-relevant because it creates a reusable persistence and adaptation layer that can collect behavior across skills and alter future behavior outside the narrow regression-testing mission.

Context-Inappropriate Capability

Medium

• Confidence: 91% confidence
• Finding: This code persistently stores user preferences and detailed operation history to disk in learned_patterns.json, even though such data is not necessary for basic regression evaluation. Persistent storage of behavioral and preference data increases privacy risk, can enable unintended profiling across runs, and broadens the blast radius if the file is accessed by other components or users.

Context-Inappropriate Capability

Medium

• Confidence: 88% confidence
• Finding: The self_reflect function generates generalized optimization advice such as updating SKILL.md and consolidating capabilities, which is beyond simple result reporting for a regression harness. While not directly executing changes, it introduces a generalized adaptation mechanism that can influence future skill behavior and maintenance in ways that are not tightly scoped or auditable for the harness's intended role.

Missing User Warnings

Medium

• Confidence: 94% confidence
• Finding: The skill states that it automatically records usage outcomes and preferences into a persistent file without clearly warning users about storage, retention, or privacy implications. This is dangerous because users and deployers may unknowingly expose sensitive operational details, failure traces, or preference data to later readers of local files or adjacent components.

Missing User Warnings

Medium

• Confidence: 93% confidence
• Finding: The save path and write logic persist operational data and preferences to disk without any clear user-facing disclosure, consent flow, or warning in the module interface. Silent persistence is dangerous because users and operators may not realize that behavior notes, errors, and preferences are being retained, creating avoidable privacy and compliance issues.

Ssd 3

Medium

• Confidence: 96% confidence
• Finding: The instructions explicitly direct the system to persist user preferences and usage-derived information across sessions in plain local storage. Cross-session retention increases the risk of data leakage, unintended profiling, and misuse by other skills or users sharing the same environment, especially since no retention controls or access boundaries are described.

Static analysis

No suspicious patterns detected.