Back to skill

Security audit

adversarial-robustness

Security checks for vulnerabilities and agentic risk

Overview

The skill has a real local robustness tester, but its self-learning module persistently stores free-form history and preferences that can influence future behavior without enough user control.

Install only if you are comfortable with local persistent learning state. Do not put secrets, confidential prompts, customer data, or sensitive incident details in learner notes, errors, or preferences, and review or delete learned_patterns.json before sharing, publishing, or reusing the skill directory.

Vulnerability Patterns
  • Agent Memory PoisoningWrites attacker-controlled rules into memory that affect later sessions
  • Insecure Skill Coding PracticesFinds exploitable flaws such as hardcoded secrets or command injection
  • Skill Instruction HijackingAlters the agent's session goals or safety constraints when the skill loads
  • Remote Payload Retrieval and ExecutionFetches external code whose behavior can change after review
  • Embedded Malicious CodeShips malicious scripts inside the skill and executes them locally
Findings (2)

T02 · Agent Memory Poisoning

Warning
Location
scripts/learner.py:88
Finding

Unvalidated Persistent Preferences Can Influence Future Agent Behavior

Content
View full analysis

Vulnerability Details

File Location: scripts/learner.py:88-92; related behavioral instructions in SKILL.md:60,67-70
Vulnerability Type: Persistent memory poisoning through attacker-controlled preference data
Risk Level: Medium

Vulnerable Code

python
def record_preference(skill_dir, key, value):
    data = load_learning(skill_dir)
    data["preferences"][key] = value
    save_learning(skill_dir, data)
    return data

The command-line interface accepts the preference key and value without validation:

python
p_pre = sub.add_parser("prefer"); p_pre.add_argument("skill_dir")
p_pre.add_argument("--key", required=True); p_pre.add_argument("--val", required=True)

The associated Skill instructions state that important preferences are persisted in learned_patterns.json and should be applied during subsequent use. They also encourage accumulated experience to be written back into SKILL.md.

Technical Analysis

record_preference persists arbitrary key and value strings without a schema, allowlist, provenance marker, integrity protection, or review workflow. The stored preferences are not interpreted or executed by learner.py itself. However, the Skill documentation instructs future agents to consult and apply this persistent state.

This creates a cross-session influence channel: data supplied by a user or another process can be stored as if it were a trusted operational preference and later affect an agent that loads the Skill. An instruction-like value could attempt to alter evaluation behavior, suppress checks, or cause unsafe guidance to be incorporated into future Skill instructions.

The issue is constrained by two factors:

  • The attacker must be able to invoke the learner or modify the Skill's state file.
  • The current Python implementation does not automatically execute preference values or rewrite SKILL.md; exploitation depends on a future agent treating the store ...[truncated 1295 chars]
Remediation
View remediation

Remediation Suggestions

  1. Define a strict schema for preferences and permit only documented keys and typed values.
  2. Reject multiline values, instruction-like content, control characters, and values exceeding a conservative length limit.
  3. Store provenance with every preference, including who supplied it, when it was supplied, and whether it was explicitly approved.
  4. Treat persisted preferences as untrusted data rather than agent instructions.
  5. Require explicit user confirmation before a stored preference affects behavior in a later session.
  6. Do not automatically or procedurally copy learned state into SKILL.md.
  7. Require human review and a visible diff before modifying Skill instructions.
  8. Add integrity controls or access restrictions if the state is shared among multiple users.
  9. Provide commands to inspect, revoke, and reset persisted preferences.

T09 · Insecure Skill Coding Practices

Note
Location
scripts/learner.py:60
Finding

Free-Form Operational Data Is Persisted in Plaintext Without Data Protection Controls

Content
View full analysis

Vulnerability Details

File Location: scripts/learner.py:60-65,77-92
Vulnerability Type: Plaintext storage of potentially sensitive user-supplied data
Risk Level: Low

Vulnerable Code

python
def save_learning(skill_dir, data):
    data["lastUpdated"] = datetime.datetime.now().strftime("%Y-%m-%dT%H:%M")
    p = learn_path(skill_dir)
    with open(p, "w", encoding="utf-8") as f:
        f.write(json.dumps(data, indent=2, ensure_ascii=False))

User-controlled error details and notes are stored verbatim:

python
if error:
    ep = data["errorPatterns"].setdefault(error, {"count": 0, "lastNote": "", "lastTime": ""})
    ep["count"] += 1
    ep["lastNote"] = note or ""
    ep["lastTime"] = data["lastUpdated"] or datetime.datetime.now().strftime("%Y-%m-%dT%H:%M")
rec = {"t": data["lastUpdated"], "cap": capability, "ok": success, "err": error, "note": note}
data["recentOps"].append(rec)
data["recentOps"] = data["recentOps"][-20:]
save_learning(skill_dir, data)

Preferences are also stored without filtering:

python
def record_preference(skill_dir, key, value):
    data = load_learning(skill_dir)
    data["preferences"][key] = value
    save_learning(skill_dir, data)
    return data

Technical Analysis

The learner accepts free-form capability names, error descriptions, notes, preference keys, and preference values. These fields are serialized directly into a predictable plaintext file named learned_patterns.json.

The implementation does not:

  • Warn users not to provide secrets or personal data.
  • Detect or redact likely credentials.
  • Limit field sizes.
  • Set restrictive file permissions explicitly.
  • Define retention limits for error patterns or preferences.
  • Provide selective deletion or redaction.
  • Use an atomic write operation to reduce state corruption risks.

Although recentOps is limited to 20 entries, errorPatterns and `pref ...[truncated 1243 chars]

Remediation
View remediation

Remediation Suggestions

  1. Clearly prohibit secrets, credentials, personal data, and confidential input in notes, errors, and preferences.
  2. Apply conservative maximum lengths to all user-controlled fields.
  3. Detect and reject or redact common secret formats, including API keys, bearer tokens, private keys, and passwords.
  4. Create the state file with restrictive permissions such as 0600, and verify that its parent directory is not broadly writable.
  5. Write updates to a securely created temporary file in the same directory, apply the required permissions, and atomically replace the destination.
  6. Establish retention limits for preferences and error history rather than retaining values indefinitely.
  7. Add commands for selective deletion, complete reset, and redaction of stored records.
  8. Add learned_patterns.json to source-control ignore rules when it contains runtime-generated user data.
  9. Consider storing only normalized counters and non-sensitive identifiers instead of raw free-form text.
Vulnerability Patterns
  • Excessive AgencyUnrestricted Tool Access, Autonomous Decision Making, Scope Creep
  • Trigger AbuseOverly Broad Trigger, Shadow Command Trigger, Keyword Baiting Trigger
  • MCP Least PrivilegeUnderdeclared Capability, Wildcard Permission, Missing Permission Declaration
  • MCP Tool PoisoningHidden Instructions, Unicode Deception, Parameter Description Injection
  • Prompt InjectionInstruction Override, Hidden Instructions, Exfiltration Commands
Findings (14)

Tp4

High
Category
MCP Tool Poisoning
Confidence
99% confidence
Finding

The documented purpose is adversarial robustness testing, but the detected behavior centers on logging, preference storage, and self-improvement rather than the claimed security evaluation functions. This mismatch is dangerous because users may grant trust or run the skill expecting one security function while it performs materially different stateful actions, creating room for hidden data collection and unsafe automation.

Content

No source excerpt is available for this finding.

Description-Behavior Mismatch

High
Category
Not specified by scanner
Confidence
98% confidence
Finding

This file implements a generic cross-skill telemetry and learning component that can be invoked against arbitrary skill directories, which is unrelated to the stated adversarial-robustness purpose. In a plugin/skill ecosystem, a reusable persistence module broadens data collection scope and creates an unexpected surveillance surface across skills, increasing privacy and misuse risk.

Content

No source excerpt is available for this finding.

Undeclared Tool Scope

Medium
Category
MCP Least Privilege
Confidence
84% confidence
Finding

The skill advertises itself as a local robustness tool, but it declares no explicit tool scope while documentation indicates file-writing behavior via learner state persistence. Missing permission boundaries makes it easier for an agent to invoke the skill without users understanding that it can modify local files, which weakens trust and containment.

Content

No source excerpt is available for this finding.

Natural-Language Policy Violations

Medium
Category
Not specified by scanner
Confidence
90% confidence
Finding

The skill description and user-facing documentation are presented entirely in Chinese, with no indication that the user can choose another language or that the locale restriction is required for a region-specific purpose. This can violate language/locale policy when a skill implicitly mandates a single language without opt-in.

Content

No source excerpt is available for this finding.

Vague Triggers

Medium
Category
Not specified by scanner
Confidence
86% confidence
Finding

The activation guidance is broad and lacks explicit exclusions, so an agent may over-apply the skill to general text workflows rather than limiting it to deliberate robustness assessments. In context, that increases the chance that sensitive user text is unnecessarily transformed, evaluated, or recorded by associated learner components.

Content

No source excerpt is available for this finding.

Description-Behavior Mismatch

Medium
Category
Not specified by scanner
Confidence
93% confidence
Finding

The manifest frames the skill as a pure local robustness module, but the documentation adds persistent self-learning that records user preferences, attack history, and failure notes to disk. Persistent storage of evaluation history can expose sensitive prompts, decision outcomes, and operator habits, especially in a red-team or moderation setting where inputs may contain confidential content.

Content

No source excerpt is available for this finding.

Missing User Warnings

Medium
Category
Not specified by scanner
Confidence
93% confidence
Finding

The documentation describes persistent learning and memory but does not provide a clear warning that data will be retained across runs. In a security-testing skill, retained failure notes, attack variants, and preferences may contain sensitive operational information, making silent persistence a real privacy and data-handling risk.

Content

No source excerpt is available for this finding.

Intent-Code Divergence

Medium
Category
Not specified by scanner
Confidence
82% confidence
Finding

The docstring says the module 'only' records usage, preferences, insights, and reflection. However, the reflection output explicitly recommends writing experience back into the skill's SKILL.md, which extends beyond the narrowly described responsibilities and contradicts the 'only responsible for' framing.

Content

No source excerpt is available for this finding.

Context-Inappropriate Capability

Medium
Category
Not specified by scanner
Confidence
95% confidence
Finding

The module persistently stores arbitrary capability usage, failures, error types, notes, and user preferences to disk for any provided skill directory. Because the skill's declared purpose is adversarial robustness evaluation, this broad operational telemetry is unjustified and may capture sensitive user data or workflow details without necessity.

Content

No source excerpt is available for this finding.

Missing User Warnings

Medium
Category
Not specified by scanner
Confidence
96% confidence
Finding

The record operation stores recent activity, error labels, and free-form notes to disk, and those notes may contain sensitive prompts, internal data, or user context. Because retention is automatic and undisclosed, users may unknowingly leave a local audit trail of sensitive troubleshooting content.

Content

No source excerpt is available for this finding.

Missing User Warnings

Medium
Category
Not specified by scanner
Confidence
95% confidence
Finding

User preferences are written to disk without any warning, consent flow, or indication that potentially sensitive values will persist in learned_patterns.json. In a local skill environment, this can expose personal settings or secrets entered as values to other local users, backups, or unrelated tooling.

Content

No source excerpt is available for this finding.

Intent-Code Divergence

Low
Category
Not specified by scanner
Confidence
89% confidence
Finding

At L48 the text states that after each evaluation the skill 'automatically' performs review and accumulates experience. However, L53-L65 show that recording, preference saving, insight, and reflection are all separate explicit CLI commands, so the documented automation contradicts the described operational flow.

Content

No source excerpt is available for this finding.

Natural-Language Policy Violations

Low
Category
Not specified by scanner
Confidence
88% confidence
Finding

The JSON contains the capability name "对抗鲁棒性" in user-visible metadata fields, and there is no surrounding text indicating that language selection is optional or that the skill is intentionally region-specific. Under the stated policy, forcing a specific language without opt-in can be a natural-language policy violation.

Content

No source excerpt is available for this finding.

Natural-Language Policy Violations

Low
Category
Not specified by scanner
Confidence
88% confidence
Finding

The file's natural-language documentation and command output strings are written in Chinese only, with no indication that users can select another language. This can violate language/locale policy when a skill is expected to avoid forcing a specific language absent user opt-in or documented regional scope.

Content

No source excerpt is available for this finding.