Back to skill

Security audit

Extract Error Patterns

Security checks for vulnerabilities and agentic risk

Overview

This log-analysis skill is not malicious, but it needs review because its helper script can copy sensitive log contents into reports without redaction and its documentation is under-scoped.

Review before installing or using on real production logs. Prefer redacted or synthetic logs, avoid untrusted Markdown rendering of generated reports, do not provide an API key unless the publisher explains why it is needed, and write reports only to controlled locations with appropriate access restrictions.

Vulnerability Patterns
  • Insecure Skill Coding PracticesFinds exploitable flaws such as hardcoded secrets or command injection
  • Skill Instruction HijackingAlters the agent's session goals or safety constraints when the skill loads
  • Agent Memory PoisoningWrites attacker-controlled rules into memory that affect later sessions
  • Remote Payload Retrieval and ExecutionFetches external code whose behavior can change after review
  • Embedded Malicious CodeShips malicious scripts inside the skill and executes them locally
Findings (2)

T09 · Insecure Skill Coding Practices

Warning
Location
scripts/parse.py:42
Finding
Attacker-Controlled Log Content Can Break Out of Markdown Code Fences## Vulnerability Details **File Location**: `scripts/parse.py:42-49` **Vulnerability Type**: Improper neutralization of untrusted content in generated Markdown **Risk Level**: Medium ### Vulnerable Code ```python def to_markdown_alerts(rules): lines = ["# Alert Rules\n"] for r in rules: lines.append(f"## {r['name']} (`{r['type']}`)\n") lines.append(f"- **Severity:** {r['severity']}") lines.append(f"- **Occurrences:** {r['count']}") lines.append(f"- **Regex:** ````{r['pattern']}````") lines.append(f"- **Samples:**") for s in r["top_samples"]: lines.append(f" ```\n {s}\n ```") ``` ### Technical Analysis Log samples are attacker-controlled data and are inserted verbatim into Markdown code fences. The implementation neither escapes backtick sequences nor selects a fence longer than those appearing in the sample. The parser separates records only on the line-feed character: ```python lines = [l.strip() for l in log_text.split("\n") if l.strip()] ``` Consequently, a record containing carriage-return-only separators can remain a single Python string while being interpreted as multiple lines by Markdown processors that normalize carriage returns. Such content can include a valid closing fence followed by attacker-controlled Markdown. Even when the generated report is not rendered, passing it to another AI Agent without preserving the trust boundary can turn injected report text into indirect prompt-injection content. The parser itself does not execute the injected content. ### Attack Path 1. An attacker causes a server log to contain a record with a recognized term such as `error`. 2. The record uses carriage-return-only separators and includes a closing Markdown fence followed by attacker-controlled Markdown or instructions. 3. `extract_patterns()` retains the record as a sample because `split("\n")` does not divide it at carriage ret ...[truncated 787 chars]
Remediation
## Remediation Suggestions - Normalize all supported line endings before parsing, for example with `splitlines()`. - Escape or replace backtick runs in untrusted samples before Markdown serialization. - Alternatively, calculate the longest backtick run in each sample and use a strictly longer fence. - Consider encoding samples as indented code blocks or structured JSON rather than interpolating them into Markdown. - Clearly label all extracted samples as untrusted input. - Do not pass generated reports to an AI Agent as trusted instructions; isolate samples in a data-only channel where available. - Add tests covering triple backticks, carriage-return-only logs, mixed line endings, Markdown links, and Agent-instruction-like content.

T09 · Insecure Skill Coding Practices

Warning
Location
scripts/parse.py:17
Finding
Log Samples May Disclose PII, Credentials, and Other Sensitive Data## Vulnerability Details **File Location**: `scripts/parse.py:17-27`, `scripts/parse.py:60-72`; declared safeguard at `SKILL.md:45` **Vulnerability Type**: Plaintext sensitive-data exposure and missing PII control **Risk Level**: Medium ### Vulnerable Code ```python def extract_patterns(log_text): lines = [l.strip() for l in log_text.split("\n") if l.strip()] counters = {label: Counter() for _, label, _ in KNOWN_ERRORS} for line in lines: for pattern, label, _ in KNOWN_ERRORS: if re.search(pattern, line): counters[label][line[:120]] += 1 rules = [] for pattern, label, default_sev in KNOWN_ERRORS: count = sum(counters[label].values()) ``` The retained samples are subsequently printed or written to a file: ```python with open(args.file) as f: text = f.read() rules = extract_patterns(text) out = to_markdown_alerts(rules) if args.format == "markdown" else json.dumps(rules, indent=2) if args.output: open(args.output, "w").write(out) print(f"✅ Written to {args.output}") else: print(out) ``` This contradicts the declared control in `SKILL.md:45`: ```markdown - Do not extract personal data (PII) without explicit user confirmation and data handling rules ``` ### Technical Analysis Every matching log line is copied into a counter after only a 120-character truncation. Truncation is not sanitization: emails, usernames, IP addresses, authorization headers, session identifiers, access tokens, request parameters, and other sensitive values frequently occur near the beginning of log records. The implementation performs no sensitive-data detection, redaction, user confirmation, output warning, or restrictive output-file permission handling. Samples are included in both Markdown and JSON output. Because server logs often aggregate data from multiple users and services, generating a report creates an additional plain ...[truncated 1222 chars]
Remediation
## Remediation Suggestions - Detect and redact common sensitive fields before retaining or serializing samples, including authorization headers, API keys, access tokens, cookies, email addresses, and session identifiers. - Replace sensitive values with stable placeholders or keyed hashes when correlation is required. - Require explicit confirmation before producing unredacted samples, as required by `SKILL.md`. - Add a safe default such as `--no-samples`, with a separate explicit option for unredacted context. - Display a clear warning when report output may contain sensitive log content. - Create output files with restrictive permissions and avoid following unsafe symbolic links where the execution environment is not trusted. - Document appropriate retention, access-control, and deletion requirements for generated reports. - Add tests proving that representative PII and credential formats are redacted from both Markdown and JSON output.
Vulnerability Patterns
  • Prompt InjectionInstruction Override, Hidden Instructions, Exfiltration Commands
  • Excessive AgencyUnrestricted Tool Access, Autonomous Decision Making, Scope Creep
  • System Prompt LeakageDirect Leakage, Indirect Extraction, Tool-Based Exfiltration
  • Trigger AbuseOverly Broad Trigger, Shadow Command Trigger, Keyword Baiting Trigger
  • MCP Least PrivilegeUnderdeclared Capability, Wildcard Permission, Missing Permission Declaration
Findings (20)

Tp4

High
Category
MCP Tool Poisoning
Confidence
99% confidence
Finding
The code does not parse server logs, extract error patterns, or generate alert rules. Its primary purpose is quality/compliance testing of a skill repository's documentation and file structure. This is a materially different behavior from the declared description. There are no indicators of log analysis or alert-rule generation; instead, it performs metadata checks, markdown section validation, existence checks for ancillary files, and secret scanning. Therefore the declared purpose does not accurately represent the actual code behavior.

Anti-Refusal Statement

High
Category
Anti-Refusal
Content
|---------|-----|------|
| Low confidence | Marks as high confidence, flags no review | Reports "Extraction [X]: confidence 0.4 — flagged for manual review" |
| Ambiguous input | Forces extraction, guesses | Reports "Cannot classify [X] — insufficient context, skipped" |
| PII detected | Extracts without warning | Stops and asks: "PII detected in [location] — confirm before extracting" |
| Pattern miss | Silent failure, returns empty | Reports "No patterns matched input — check format or add custom pattern" |
| Large input | Loads all, crashes | Streams, reports "Processed 10K lines, found 142 matches (truncated at limit)" |
Confidence
80% confidence
Finding
Skill instructs the agent to omit warnings, disclaimers, or ethical commentary. Stripping safety caveats hides risk from the user and is a common jailbreak preamble.

Direct Prompt Extraction

High
Category
System Prompt Leakage
Content
"pattern": pattern,
                "top_samples": [s for s, _ in top],
            })
    return rules

def to_markdown_alerts(rules):
    lines = ["# Alert Rules\n"]
Confidence
85% confidence
Finding
Skill contains instructions that could directly expose system prompts, internal rules, or hidden instructions to users or external parties.

Natural-Language Policy Violations

Medium
Confidence
94% confidence
Finding
The line explicitly instructs contributors to keep SKILL.md body content in English, which is a language policy constraint. The file does not provide an opt-in choice or a documented justification for this restriction, so it matches the natural-language language policy violation criteria.

Natural-Language Policy Violations

Medium
Confidence
95% confidence
Finding
This requirement mandates that SKILL.md body content and reference documentation be in English. Because the policy is enforced as a blanket language constraint and no user choice or clear regional/compliance justification is provided, it falls under the language/locale policy violation category.

Vague Triggers

Medium
Confidence
96% confidence
Finding
The README says "When triggered: [trigger condition]." without providing any actual trigger phrase, scope, or exclusion criteria. For a markdown file, this leaves activation behavior undefined and could lead to overly broad or unintended invocation.

Intent-Code Divergence

Medium
Confidence
84% confidence
Finding
The README advertises a write mode and JSON input workflow that does not align with the stated purpose of log-pattern extraction, which is typically read/analysis oriented. Ambiguous write capability can normalize unexpected data modification behavior and may lead users to run the skill with broader permissions or on sensitive files without understanding side effects.

Missing User Warnings

Medium
Confidence
87% confidence
Finding
Documenting a write mode without any warning about modification of input or output data is dangerous because users may assume the skill is purely analytical and grant access to important files or directories. In the context of server log processing, undocumented writes could overwrite evidence, alter datasets, or create unsafe output in sensitive locations.

Context-Inappropriate Capability

Medium
Confidence
88% confidence
Finding
Requiring an external API key for a log-analysis skill without explaining the need for network access raises a real security concern because server logs often contain sensitive operational data. Users may unknowingly transmit internal logs, credentials, tokens, or personal data to a third-party service, expanding the attack surface and creating data exfiltration risk.

Vague Triggers

Medium
Confidence
97% confidence
Finding
This markdown file describes activation using the unresolved placeholder '触发条件: [trigger condition]' and generic phrasing like '将 extract error patterns 输入转换为结构化输出' without clearly defining when the skill should or should not activate. The lack of explicit trigger phrases, constraints, or negative examples makes invocation behavior ambiguous and could cause unintended use.

Intent-Code Divergence

Medium
Confidence
83% confidence
Finding
The README states the skill extracts error patterns from server logs and generates alert rules, but the usage section presents generic '--mode read' and '--mode write' operations with JSON input instead of log analysis behavior. This is not just incomplete documentation; it actively frames the skill as a generic reader/writer, which conflicts with the stated intent of a log-analysis skill.

Lp3

Medium
Category
MCP Least Privilege
Confidence
70% confidence
Finding
Without declared permissions the skill's intent is opaque and cannot be validated.

Vague Triggers

Medium
Confidence
95% confidence
Finding
The activation condition is vague: 'Use when (1) Extract error patterns from server logs and generate actionable alert rules.' Ambiguous triggering can cause the agent to invoke the skill in broader contexts than intended, increasing the chance it processes sensitive logs, unrelated text, or unreviewed data flows without appropriate safeguards.

Description-Behavior Mismatch

Medium
Confidence
97% confidence
Finding
The manifest describes a skill for analyzing server logs and producing actionable alert rules, but this file implements a broad repository compliance test suite for frontmatter, README files, licenses, section headings, and file presence. These operations are not an obvious implementation detail of error-pattern extraction and instead reflect unrelated packaging/documentation validation behavior.

Natural-Language Policy Violations

Medium
Confidence
96% confidence
Finding
The test `test_readme_zh_exists` requires `README_zh.md` to exist, which encodes a specific language/locale expectation in the skill's quality gate. This is a natural-language policy concern because it imposes a locale-specific documentation requirement without any visible user opt-in or documented regional justification in this file.

Intent-Code Divergence

Low
Confidence
78% confidence
Finding
The feature list asserts specific behavioral guarantees such as preserving data integrity and avoiding silent drops or fabrication, yet the surrounding README content is largely templated and does not document concrete logic that would support those claims. This creates intent-documentation divergence because the documentation presents assured behavior without evidence of actual implemented mechanisms in the provided skill file.

Natural-Language Policy Violations

Low
Confidence
72% confidence
Finding
The primary instructional content in this file is written in Chinese, and the file itself does not offer an explicit language choice within the instructions beyond a link to an English file. This can be a locale/language policy concern if users are expected to consume the skill documentation without prior language opt-in.

Context-Inappropriate Capability

Low
Confidence
72% confidence
Finding
The manifest-level purpose is limited to extracting patterns from server logs and generating alert rules, which does not obviously require external service authentication. The README's required 'API_KEY' suggests a network-backed or external-service capability that is not justified by the declared purpose in the provided documentation.

Intent-Code Divergence

Low
Confidence
88% confidence
Finding
The docstring implies the file is an auto-generated test suite for this specific skill, which suggests behavior-focused validation of error-pattern extraction. In practice, the tests are mostly generic checks for markdown structure, licensing, file existence, and other repository conventions, creating a misleading statement about what is actually being tested.

Context-Inappropriate Capability

Low
Confidence
90% confidence
Finding
The secret-scanning routine walks files under the test directory and inspects contents for credential patterns. Secret detection may be useful in a repository hygiene tool, but it is not directly justified by a skill whose declared purpose is extracting error patterns from server logs and generating alert rules.

Static analysis

No suspicious patterns detected.