Back to skill

Security audit

distillation-adversarial-verify

Security checks for vulnerabilities and agentic risk

Overview

This skill should be reviewed carefully because it can run code from the skill it is checking and it stores cross-session notes and preferences without tight controls.

Use this only in a sandbox or with student skill directories you already trust, because verification may execute their Python code. Avoid storing secrets or sensitive notes through learner.py, and inspect or delete learned_patterns.json if you do not want retained preferences or history.

Vulnerability Patterns
  • Agent Memory PoisoningWrites attacker-controlled rules into memory that affect later sessions
  • Insecure Skill Coding PracticesFinds exploitable flaws such as hardcoded secrets or command injection
  • Skill Instruction HijackingAlters the agent's session goals or safety constraints when the skill loads
  • Remote Payload Retrieval and ExecutionFetches external code whose behavior can change after review
  • Embedded Malicious CodeShips malicious scripts inside the skill and executes them locally
Findings (2)

T09 · Insecure Skill Coding Practices

Error
Location
scripts/adversarial_verify.py:71
Finding

Untrusted Target Module Executes During Rule Verification

Content
View full analysis

Vulnerability Details

File Location: scripts/adversarial_verify.py, lines 71-79
Vulnerability Type: Unsafe in-process execution of untrusted Python code
Risk Level: High

Vulnerable Code

python
def _load_rules_from_skill(skill_dir):
    """Attempt to load RULES from scripts/adv_rules.py in the student skill."""
    cand = os.path.join(skill_dir, "scripts", "adv_rules.py")
    if not os.path.isfile(cand):
        return None
    spec = importlib.util.spec_from_file_location("adv_rules", cand)
    mod = importlib.util.module_from_spec(spec)
    spec.loader.exec_module(mod)
    return getattr(mod, "RULES", None)

Technical Analysis

The verifier loads a Python module from a user-supplied skill directory and invokes exec_module() in the verifier process. Importing a Python module executes all of its top-level statements before the RULES attribute is retrieved.

The target skill is precisely the artifact being assessed, so its scripts/adv_rules.py file cannot safely be assumed to be trusted. No validation, sandbox, subprocess boundary, privilege restriction, resource limit, network restriction, or user confirmation is applied before execution.

Although the path is constructed with os.path.join() rather than passed to a shell, this does not prevent code execution because the selected file is deliberately interpreted as Python.

Attack Path

  1. An attacker creates or modifies a student skill directory.
  2. The attacker places malicious top-level Python statements in scripts/adv_rules.py.
  3. A user runs the documented verification command against that directory.
  4. _load_rules_from_skill() creates an import specification for the attacker-controlled file.
  5. spec.loader.exec_module(mod) executes the malicious statements before validating or retrieving RULES.
  6. The statements run with the same operating-system identity, environment, filesystem access, and network acce ...[truncated 815 chars]
Remediation
View remediation

Remediation Suggestions

  1. Replace executable Python rule modules with a declarative, strictly validated format such as JSON containing supported operators, operands, weights, and test cases.
  2. If callable Python rules are unavoidable, never import them into the verifier process.
  3. Execute target rules in a disposable sandbox or container using an unprivileged identity.
  4. Remove secrets from the sandbox environment and mount host data read-only, exposing only the minimum required input files.
  5. Disable outbound network access and prevent access to sensitive host paths.
  6. Apply CPU, memory, process-count, file-size, and execution-time limits.
  7. Use a narrowly defined serialization protocol for inputs and outputs rather than sharing Python objects.
  8. Validate the returned rule schema, data types, weights, and case limits before processing results.
  9. Clearly warn users that executable rules are untrusted and require explicit confirmation before evaluation.
  10. Do not treat AST inspection or import-name filtering as a sufficient sandbox; Python code must be isolated at the process and operating-system level.

T02 · Agent Memory Poisoning

Note
Location
scripts/learner.py:66
Finding

Unrestricted Persistent Storage of User Notes and Preferences

Content
View full analysis

Vulnerability Details

File Location: scripts/learner.py, lines 66-96
Vulnerability Type: Persistent memory poisoning and sensitive-data retention
Risk Level: Low

Vulnerable Code

python
def record_op(skill_dir, capability, success=True, error=None, note=None):
    data = load_learning(skill_dir)
    data["totalOps"] += 1
    cap = data["capabilityStats"].setdefault(capability, {"count": 0, "success": 0, "fail": 0})
    cap["count"] += 1
    if success:
        cap["success"] += 1
    else:
        cap["fail"] += 1
        data["totalErrors"] += 1
    if error:
        ep = data["errorPatterns"].setdefault(error, {"count": 0, "lastNote": "", "lastTime": ""})
        ep["count"] += 1
        ep["lastNote"] = note or ""
        ep["lastTime"] = data["lastUpdated"] or datetime.datetime.now().strftime("%Y-%m-%dT%H:%M")
    rec = {"t": data["lastUpdated"], "cap": capability, "ok": success, "err": error, "note": note}
    data["recentOps"].append(rec)
    data["recentOps"] = data["recentOps"][-20:]
    save_learning(skill_dir, data)
    return data

def record_preference(skill_dir, key, value):
    data = load_learning(skill_dir)
    data["preferences"][key] = value
    save_learning(skill_dir, data)
    return data

Technical Analysis

The learning component persistently stores unrestricted, free-form capability, error, note, preference-key, and preference-value data in learned_patterns.json. The accompanying skill instructions state that important preferences should be used during later invocations.

The implementation does not provide:

  • Sensitivity or secret detection.
  • Trust or provenance metadata.
  • Input length or structural restrictions.
  • Expiration or retention controls.
  • Confirmation before a stored preference influences later behavior.
  • Separation between trusted configuration and attacker-influenced observations.

Consequently, sen ...[truncated 1453 chars]

Remediation
View remediation

Remediation Suggestions

  1. Make persistent learning explicitly opt-in and disclose exactly which fields will be stored.
  2. Store only a documented allowlist of structured preference types instead of arbitrary free-form keys and values.
  3. Reject or redact likely credentials, tokens, private keys, authorization headers, and other sensitive values.
  4. Apply strict length limits and normalize all stored fields.
  5. Record provenance, creation time, trust level, and expiration time for every preference.
  6. Treat persisted values as untrusted data and never interpret them as instructions, code, command arguments, or policy overrides.
  7. Require user confirmation before applying a persisted preference to a later session.
  8. Separate operational statistics from behavioral preferences so observations cannot silently become instructions.
  9. Provide commands to inspect, selectively delete, expire, and fully reset stored state.
  10. Use restrictive filesystem permissions for learned_patterns.json and avoid placing secrets in the file.
Vulnerability Patterns
  • Data ExfiltrationExternal Transmission, Env Variable Harvesting, File System Enumeration
  • Excessive AgencyUnrestricted Tool Access, Autonomous Decision Making, Scope Creep
  • MCP Least PrivilegeUnderdeclared Capability, Wildcard Permission, Missing Permission Declaration
  • MCP Tool PoisoningHidden Instructions, Unicode Deception, Parameter Description Injection
  • Prompt InjectionInstruction Override, Hidden Instructions, Exfiltration Commands
Findings (20)

Tp4

High
Category
MCP Tool Poisoning
Confidence
99% confidence
Finding

The stated purpose is adversarial verification of distilled rules, but the documented behavior instead adds logging, preference persistence, and self-reflection features while omitting the promised verification logic. This is dangerous because users and orchestrators may trust the skill for security-relevant validation it does not actually perform, leading to false assurance, while also performing undeclared data collection side effects.

Content

No source excerpt is available for this finding.

Context-Inappropriate Capability

High
Category
Not specified by scanner
Confidence
99% confidence
Finding

The verifier imports and executes scripts/adv_rules.py from the supplied skill directory via spec.loader.exec_module(mod). That means merely analyzing an untrusted student skill triggers arbitrary Python execution on the analyst's machine, enabling code execution, file access, environment exfiltration, or persistence if a malicious skill is provided.

Content

No source excerpt is available for this finding.

Description-Behavior Mismatch

High
Category
Not specified by scanner
Confidence
98% confidence
Finding

The file’s functionality materially diverges from the manifest: instead of adversarial distillation verification, it implements a generic cross-skill learning and telemetry module. In an agent-skill ecosystem, this kind of scope mismatch is dangerous because it hides persistent data collection and behavior-tracking capabilities under an unrelated description, undermining review assumptions and enabling unauthorized monitoring across arbitrary skills.

Content

No source excerpt is available for this finding.

Undeclared Tool Scope

Medium
Category
MCP Least Privilege
Confidence
86% confidence
Finding

The skill documentation instructs use of scripts that write persistent state files, but the manifest declares no tool scope or permissions. This creates a transparency and governance gap: operators may believe the skill is read-only/local-analysis oriented when it can modify files on disk, including cross-session memory files. In an agent setting, undeclared write capability increases the chance of unauthorized persistence or policy bypass.

Content

No source excerpt is available for this finding.

Description-Behavior Mismatch

Medium
Category
Not specified by scanner
Confidence
97% confidence
Finding

The skill presents itself as a pure local adversarial verification utility, yet its documentation adds a persistent self-learning subsystem that records usage history and user preferences. That mismatch broadens the skill from one-shot analysis into cross-session state accumulation, which can create hidden retention of potentially sensitive workflow data and violates least surprise.

Content

No source excerpt is available for this finding.

Context-Inappropriate Capability

Medium
Category
Not specified by scanner
Confidence
94% confidence
Finding

Collecting user preferences and cross-session memory is not necessary for adversarial verification of rules, so the data collection lacks a clear purpose limitation. Unnecessary persistence increases privacy risk and can be abused to accumulate behavioral metadata unrelated to the stated function of the skill.

Content

No source excerpt is available for this finding.

Missing User Warnings

Medium
Category
Not specified by scanner
Confidence
94% confidence
Finding

The skill describes storing usage history and user preferences without an explicit privacy warning or consent framing. In a security-sensitive agent ecosystem, silent retention of user-linked data can expose sensitive operational patterns and create compliance and trust issues, especially because the skill's primary purpose does not require such storage.

Content

No source excerpt is available for this finding.

Ssd 3

Medium
Category
Not specified by scanner
Confidence
92% confidence
Finding

The documentation explicitly instructs persistent storage of user preferences and usage history in a local memory file across sessions. Cross-session persistence expands the attack surface by retaining behavioral data that could later be read, misused, or correlated, and it is especially unjustified given the skill's advertised role as a local rule-verification tool.

Content

No source excerpt is available for this finding.

Missing User Warnings

Medium
Category
Not specified by scanner
Confidence
95% confidence
Finding

The code dynamically executes student-provided Python during verification without any user-facing disclosure beyond normal control flow. This is dangerous because users may reasonably expect a verifier to inspect artifacts, not execute attacker-controlled code from the target directory, increasing the chance of accidental compromise.

Content

No source excerpt is available for this finding.

Natural-Language Policy Violations

Medium
Category
Not specified by scanner
Confidence
97% confidence
Finding

This Python file contains its primary docstring, CLI description, comments, and printed status messages entirely in Chinese, which imposes a specific language on users. The file does not provide any opt-in, locale selection, or justification that this is a region-specific skill, so it matches the language/locale policy violation criteria.

Content

No source excerpt is available for this finding.

Context-Inappropriate Capability

Medium
Category
Not specified by scanner
Confidence
94% confidence
Finding

The code persistently stores user preferences, notes, capability usage, and error histories for arbitrary skills in learned_patterns.json without any evident minimization, retention control, or consent boundary. Even if not overtly exfiltrating data, this creates an unnecessary telemetry surface that can expose sensitive workflow details or user-provided notes, especially because the declared skill purpose does not justify broad cross-skill profiling.

Content

No source excerpt is available for this finding.

Description-Behavior Mismatch

Medium
Category
Not specified by scanner
Confidence
95% confidence
Finding

The section introduces a generic self-evolving learning and persistence mechanism that goes beyond the skill’s stated purpose of local adversarial verification. This expands the skill’s behavioral scope to collect and retain operational data, increasing privacy and integrity risk and creating an unexpected capability surface for a narrowly scoped tool.

Content

No source excerpt is available for this finding.

Missing User Warnings

Medium
Category
Not specified by scanner
Confidence
97% confidence
Finding

The skill documents persistent storage of usage data, errors, and user preferences without any warning about retention, privacy implications, or consent. This is dangerous because users may provide sensitive inputs expecting ephemeral processing, while the skill quietly preserves data locally for future reuse.

Content

No source excerpt is available for this finding.

Context-Inappropriate Capability

Medium
Category
Not specified by scanner
Confidence
94% confidence
Finding

Persistently accumulating user preferences, error history, and operational history is not necessary for adversarial rule verification and creates avoidable data retention risk. Because the behavior is framed as automatic and reusable across sessions, it can silently build a profile unrelated to the core function of the skill.

Content

No source excerpt is available for this finding.

Ssd 3

Medium
Category
Not specified by scanner
Confidence
95% confidence
Finding

The natural-language instructions explicitly encourage storing and reusing user preferences and usage details, which creates a data leakage and over-retention risk even without complex code. Free-form notes such as errors and user-specific preferences can easily capture sensitive or identifying information that later gets surfaced in unrelated contexts.

Content

No source excerpt is available for this finding.

Missing User Warnings

Medium
Category
Not specified by scanner
Confidence
96% confidence
Finding

The instructions tell the agent to write learned behavior back into SKILL.md without warning that it modifies skill files. Silent self-modification is risky because it can alter trusted instructions, introduce prompt-injection persistence, and make future executions behave differently without review.

Content

No source excerpt is available for this finding.

Intent-Code Divergence

Low
Category
Not specified by scanner
Confidence
96% confidence
Finding

L61 states that after each use the system automatically performs review and accumulates experience, implying built-in automatic behavior. But L66-L78 instruct the operator to manually call python scripts/learner.py ... commands to record success, failure, preferences, and reflection, which contradicts the claim of automatic post-use learning.

Content

No source excerpt is available for this finding.

Natural-Language Policy Violations

Low
Category
Not specified by scanner
Confidence
76% confidence
Finding

The example command records 输出语言 with value 中文, and nearby text says future runs will directly apply stored preferences. This can create a language-choice policy issue because the skill normalizes a specific output language without explicitly stating that language selection is user-driven and optional.

Content

No source excerpt is available for this finding.

Natural-Language Policy Violations

Low
Category
Not specified by scanner
Confidence
89% confidence
Finding

This JSON file contains natural-language values such as "对抗验证" for capability names, and there is no accompanying indication that users may choose their preferred language or locale. Because SQP-3 applies to all file types, forcing a specific language in user-facing metadata without opt-in is a policy concern.

Content

No source excerpt is available for this finding.

Natural-Language Policy Violations

Low
Category
Not specified by scanner
Confidence
87% confidence
Finding

The module docstring and user-facing messages are written exclusively in Chinese, including usage and operational output, which imposes a language choice on users. The file does not offer an alternate language option or explain that the tool is intentionally limited to a Chinese-speaking context.

Content

No source excerpt is available for this finding.