Security audit

distillation-adversarial-verify

Security checks for vulnerabilities and agentic risk

Overview

This skill mostly performs local adversarial rule testing, but it also silently executes target-skill Python code and includes a broad persistent learning/memory subsystem that is not well scoped to that purpose.

Review before installing. Use --selftest only in normal environments, and run verification against untrusted skills only inside a sandbox because scripts/adv_rules.py will be executed as Python. Remove or disable the learner/memory files unless you explicitly want local history and preference storage, and inspect/delete learned_patterns.json regularly.

SkillSpector

Vulnerability Patterns

Data ExfiltrationExternal Transmission, Env Variable Harvesting, File System Enumeration
Excessive AgencyUnrestricted Tool Access, Autonomous Decision Making, Scope Creep
MCP Least PrivilegeUnderdeclared Capability, Wildcard Permission, Missing Permission Declaration
MCP Tool PoisoningHidden Instructions, Unicode Deception, Parameter Description Injection
Prompt InjectionInstruction Override, Hidden Instructions, Exfiltration Commands

Findings (17)

Lp3

Medium

Category: MCP Least Privilege
Confidence: 90% confidence
Finding: The skill declares itself as a local adversarial verification tool, but the documentation also instructs writing persistent state to a local file. Undeclared file-write capability creates hidden side effects and weakens least-privilege expectations, which can lead to unintended data retention or filesystem modification when operators assume the skill is read-only or analysis-only.

Tp4

High

Category: MCP Tool Poisoning
Confidence: 97% confidence
Finding: There is a material mismatch between the stated purpose of the skill and the additional behavior described later: persistent experience logging, preference storage, and self-modifying workflow. This is dangerous because users may invoke a supposedly narrow verification tool while unintentionally enabling collection and retention of operational history and preferences unrelated to the stated task.

Description-Behavior Mismatch

Medium

Confidence: 93% confidence
Finding: The self-learning subsystem introduces persistent state and behavior changes over time, which contradicts the expectation of a simple local zero-dependency verifier. Stateful evolution increases audit difficulty and can produce non-deterministic outputs, while memory files may accumulate sensitive operational data across sessions.

Context-Inappropriate Capability

Medium

Confidence: 95% confidence
Finding: The learner component stores user preferences and usage history even though those data are not necessary for adversarial rule verification. Collecting unrelated data expands the privacy and security footprint of the skill and creates avoidable exposure if the local environment is shared, backed up, or later accessed by other tools.

Context-Inappropriate Capability

Medium

Confidence: 98% confidence
Finding: The code imports scripts/adv_rules.py from an arbitrary user-supplied skill directory using importlib and then executes it via spec.loader.exec_module(mod). In Python, importing a module runs its top-level code, so merely analyzing an untrusted skill can trigger arbitrary code execution on the host running this verifier.

Description-Behavior Mismatch

High

Confidence: 98% confidence
Finding: The file's implemented behavior materially diverges from the advertised skill purpose: instead of adversarial verification of distilled rules, it provides a generic cross-skill telemetry and persistence mechanism. This kind of capability mismatch is dangerous because it expands data collection and stateful behavior beyond user expectations, making it easier to hide surveillance, unintended retention, or unauthorized capability growth inside an unrelated skill.

Context-Inappropriate Capability

Medium

Confidence: 94% confidence
Finding: The module records arbitrary preferences, error notes, and recent operational history for any skill directory, which is broader than necessary for adversarial verification. In this context, the danger is uncontrolled local data retention: sensitive user inputs or workflow details can be persisted to disk without clear minimization, scoping, or disclosure, increasing privacy and cross-skill data exposure risks.

Description-Behavior Mismatch

High

Confidence: 97% confidence
Finding: The section documents a persistent self-evolving learning system that stores memories, preferences, and even instructs writing changes back into SKILL.md, which is materially unrelated to the declared purpose of a local adversarial verification tool. This creates an unexpected stateful behavior and expands the skill’s authority from deterministic analysis into persistent modification of local files, increasing the risk of privacy leakage, configuration drift, and unauthorized prompt/skill tampering.

Context-Inappropriate Capability

Medium

Confidence: 94% confidence
Finding: The documentation introduces persistent user-preference and usage-memory storage that is not needed for adversarial rule verification, creating unnecessary data collection and retention. Because stored notes and preferences may include sensitive or identifying content, this broadens the privacy and security exposure without serving the skill’s stated function.

Intent-Code Divergence

Medium

Confidence: 90% confidence
Finding: The intent framing presents the skill as a general auto-improving learning system rather than a bounded verification utility, which can mislead operators about what the skill will do. This mismatch is dangerous because users may grant trust or permissions appropriate for a deterministic analyzer while the documentation normalizes ongoing state changes and adaptive behavior.

Missing User Warnings

Medium

Confidence: 94% confidence
Finding: The skill describes persistence of user preferences and usage history without any clear warning about retention, privacy impact, or operator consent. Silent or poorly disclosed retention is dangerous because even local logs can capture sensitive work patterns, errors, and preferences that users did not expect to be stored.

Missing User Warnings

Medium

Confidence: 96% confidence
Finding: The verifier dynamically executes student-provided code without any visible warning to the operator that analysis has become code execution. This creates a trust-boundary violation: a user may believe they are only validating rules, while an attacker can place arbitrary payloads in adv_rules.py that run during verification.

Missing User Warnings

Medium

Confidence: 92% confidence
Finding: The skill describes storing usage history, error patterns, user preferences, and suggestions in a local memory file without warning the user that data will persist across runs. Hidden persistence is risky because users may provide operational details or sensitive inputs assuming the tool is ephemeral, and those details can remain on disk for future unintended access or reuse.

Missing User Warnings

Medium

Confidence: 95% confidence
Finding: The instructions explicitly direct writing learned experience back into SKILL.md without warning that the skill will modify its own files. Self-modifying documentation or prompt files can silently alter future behavior, undermine review guarantees, and create a persistence mechanism for unsafe instructions or prompt injection residues.

Natural-Language Policy Violations

Medium

Confidence: 88% confidence
Finding: The example instructs storing '输出语言=中文' as a reusable preference without presenting it as a user-chosen opt-in example, which normalizes silent persistence of a behavioral preference. While lower impact than arbitrary note storage, it still demonstrates preference capture beyond the stated purpose and may bias later interactions without transparent consent.

Ssd 3

Medium

Confidence: 95% confidence
Finding: The instructions explicitly promote cross-session persistence of user preferences and usage history in a local memory file. Cross-session state can reveal historical behavior, preferences, and failure patterns over time, making the skill more dangerous in shared or multi-user environments and broadening the consequences of accidental disclosure.

Ssd 3

Medium

Confidence: 93% confidence
Finding: The section instructs persistent storage of preferences, failure reasons, and free-form notes for later automatic reuse, which can easily capture sensitive user-provided information or operational context. Because the skill’s purpose does not require such retention, the context makes this more dangerous: the data collection is both unnecessary and likely unexpected, increasing privacy and data-handling risk.

Static analysis

Detected: suspicious.dynamic_code_execution

Dynamic code execution detected.

Critical

Code: suspicious.dynamic_code_execution
Location: scripts/adversarial_verify.py:82