Back to skill

Security audit

holmes skill

Security checks for vulnerabilities and agentic risk

Overview

This skill is a Chinese Holmes-style reasoning helper, but it also tells agents to persist cases, rewrite its own instructions, and commit repository changes without adequate controls.

Install only if you explicitly want a Chinese-language reasoning skill with local case memory and self-evolution. Before using it, require confirmation before any case logging, web search, SKILL.md edit, lesson update, or Git action; replace git add -A with an allowlist; and do not let unreviewed case records become future instructions.

Vulnerability Patterns
  • Agent Memory PoisoningWrites attacker-controlled rules into memory that affect later sessions
  • Insecure Skill Coding PracticesFinds exploitable flaws such as hardcoded secrets or command injection
  • Skill Instruction HijackingAlters the agent's session goals or safety constraints when the skill loads
  • Remote Payload Retrieval and ExecutionFetches external code whose behavior can change after review
  • Embedded Malicious CodeShips malicious scripts inside the skill and executes them locally
Findings (2)

T02 · Agent Memory Poisoning

Error
Location
scripts/holmes_case_logger.py:120
Finding

Persistent Agent Memory Poisoning Through Untrusted Case Records

Content
View full analysis
**如果** `references/holmes-lessons.md` **存在,必须进行阅读** > 这是执行纪律,不是可选项。 ``` ### Technical Analysis The `--problem`, `--description`, and optional `--switch` arguments accept arbitrary strings. The only validation applied to the description is a required prefix based on the selected result. The remaining content is written verbatim into `.autoresearch/cases/ ...[truncated 2563 chars]
Remediation
View remediation

T09 · Insecure Skill Coding Practices

Warning
Location
references/autoresearch.md:107
Finding

Overbroad Git Staging Can Persist Unrelated or Sensitive Repository Files

Content
View full analysis
Remediation
View remediation
Vulnerability Patterns
  • Prompt InjectionInstruction Override, Hidden Instructions, Exfiltration Commands
  • Data ExfiltrationExternal Transmission, Env Variable Harvesting, File System Enumeration
  • Excessive AgencyUnrestricted Tool Access, Autonomous Decision Making, Scope Creep
  • MCP Least PrivilegeUnderdeclared Capability, Wildcard Permission, Missing Permission Declaration
  • MCP Tool PoisoningHidden Instructions, Unicode Deception, Parameter Description Injection
Findings (31)

Tp4

High
Category
MCP Tool Poisoning
Confidence
95% confidence
Finding

The description is too vague to accurately represent the code's concrete behavior. The code is not a generic "holmes skill"; it specifically acts as an interactive/CLI case recorder and local knowledge-management utility, persisting structured records and updating an index. It also includes threshold-based workflow prompting after every third case. With no declared permissions, triggers, or operational description, the declared purpose omits significant actual capabilities and resource access.

Content

No source excerpt is available for this finding.

Natural-Language Policy Violations

Medium
Category
Not specified by scanner
Confidence
92% confidence
Finding

This README presents all user-facing instructions, examples, and operational guidance exclusively in Chinese. Under the policy, forcing a specific language without giving the user a language or locale choice is a natural-language policy violation.

Content

No source excerpt is available for this finding.

Description-Behavior Mismatch

Medium
Category
Not specified by scanner
Confidence
93% confidence
Finding

The README explicitly instructs the agent to persist case data, create files under .autoresearch/, and update SKILL.md based on prior runs. That creates autonomous statefulness and self-modification in a reasoning skill, which can let untrusted prompts, poisoned case records, or bad prior outputs alter future agent behavior without review.

Content

No source excerpt is available for this finding.

Context-Inappropriate Capability

Medium
Category
Not specified by scanner
Confidence
91% confidence
Finding

The documented 'evolution' mechanism gives the skill long-term memory and iterative behavior changes beyond simple problem analysis. In context, this broadens the attack surface: prompt-injected or low-quality cases can accumulate into persistent instruction drift, making later runs less predictable and potentially unsafe.

Content

No source excerpt is available for this finding.

Undeclared Tool Scope

Medium
Category
MCP Least Privilege
Confidence
70% confidence
Finding

Without declared permissions the skill's intent is opaque and cannot be validated.

Content

No source excerpt is available for this finding.

Description-Behavior Mismatch

Medium
Category
Not specified by scanner
Confidence
94% confidence
Finding

The skill metadata says only 'holmes skill', while the document defines an autonomous iterative workflow that reads case files, modifies SKILL.md, writes lessons, and performs Git commits. This capability mismatch is dangerous because it hides materially sensitive behaviors from users and reviewers, reducing informed consent and making unexpected repository modification more likely.

Content

No source excerpt is available for this finding.

Context-Inappropriate Capability

Medium
Category
Not specified by scanner
Confidence
95% confidence
Finding

The skill explicitly directs the agent to update SKILL.md based on its own prior case analysis, creating a self-modification loop. Self-modifying instructions are dangerous because they can silently change future agent behavior, amplify earlier mistakes, and bypass normal review if the skill's mutable behavior is not clearly declared and controlled.

Content

No source excerpt is available for this finding.

Missing User Warnings

Medium
Category
Not specified by scanner
Confidence
97% confidence
Finding

These instructions tell the agent to update repository files and create a Git commit, but provide no warning, consent step, or safety boundary around modifying project contents and history. In practice, this increases the chance of unauthorized state changes, accidental propagation of flawed instructions, and difficult-to-revert commits made under the guise of routine operation.

Content

No source excerpt is available for this finding.

Context-Inappropriate Capability

Medium
Category
Not specified by scanner
Confidence
96% confidence
Finding

The document instructs the agent to run git add -A && git commit with a generated multi-line message, which changes repository history and can capture unrelated workspace files. In the context of a vaguely described skill, this creates a significant integrity risk: the agent may persist unintended changes, overwrite audit expectations, or commit sensitive artifacts without a clear user request.

Content

No source excerpt is available for this finding.

Natural-Language Policy Violations

Medium
Category
Not specified by scanner
Confidence
95% confidence
Finding

The file title and instruction explicitly require the skill to output a case-closing report in Chinese, as shown by the Chinese heading and the directive '解决案件后 → 输出结案报告'. This imposes a specific language on all users without any opt-in or alternative, which is a natural-language policy violation under the language/locale rule.

Content

No source excerpt is available for this finding.

Natural-Language Policy Violations

Medium
Category
Not specified by scanner
Confidence
88% confidence
Finding

The checklist is written as a mandatory instruction set in Chinese and states that after every reasoning pass, the model must follow the checklist before outputting a conclusion. This imposes a language/locale constraint without offering the user any language choice or documenting a justified region-specific limitation.

Content

No source excerpt is available for this finding.

Natural-Language Policy Violations

Medium
Category
Not specified by scanner
Confidence
95% confidence
Finding

The skill content is written entirely in Chinese and provides operational instructions without any apparent language negotiation or fallback. In a multilingual agent environment, this can cause the agent to ignore or override the user's language preference, leading to misunderstanding, inaccessible behavior, or unsafe task execution if users cannot accurately interpret or challenge the agent's reasoning.

Content

No source excerpt is available for this finding.

Natural-Language Policy Violations

Medium
Category
Not specified by scanner
Confidence
97% confidence
Finding

The file's visible instructions and headings are entirely in Chinese, which imposes a specific language on users without any opt-in or stated regional justification. This matches the policy concern for language or locale constraints expressed in natural language.

Content

No source excerpt is available for this finding.

Natural-Language Policy Violations

Medium
Category
Not specified by scanner
Confidence
97% confidence
Finding

The entire document is written in Chinese and presents the guidance as the default interaction/instruction format, with no indication that users may choose another language or that the skill is intentionally limited to a Chinese-speaking context. Under the stated policy, forcing a specific language without user opt-in is a natural-language policy violation.

Content

No source excerpt is available for this finding.

Missing User Warnings

Medium
Category
Not specified by scanner
Confidence
92% confidence
Finding

The document explicitly packages interrogation and psychological steering tactics for direct AI reuse, including techniques to make a counterpart self-contradict, self-expose, and follow the model’s framing. While not malware, this materially increases misuse risk for coercive, deceptive, or manipulative interactions because it lacks warnings, ethical boundaries, or constraints limiting use to fiction, training, or consensual roleplay.

Content

No source excerpt is available for this finding.

Missing User Warnings

Medium
Category
Not specified by scanner
Confidence
95% confidence
Finding

The document explicitly recommends demeaning and insulting phrasing patterns such as calling others '呆头鹅' and similar ridicule, and later frames them as reusable AI dialogue patterns. In an agent skill, this can cause the model to insult or belittle users or third parties without consent, increasing harassment, brand, and trust risks even if the file is presented as stylistic research.

Content

No source excerpt is available for this finding.

Natural-Language Policy Violations

Medium
Category
Not specified by scanner
Confidence
88% confidence
Finding

The guidance in the '禁区清单' and surrounding AI usage sections prescriptively forbids alternative phrasing styles and directs the assistant toward a narrow Chinese Holmes persona, without any user opt-in or adaptation. That can override user preferences, reduce accessibility, and create prompt-level behavioral lock-in that conflicts with higher-priority instructions or multilingual use cases.

Content

No source excerpt is available for this finding.

Natural-Language Policy Violations

Medium
Category
Not specified by scanner
Confidence
89% confidence
Finding

The title and entire instruction set are written exclusively in Chinese, presenting the framework as a general-purpose AI action guide rather than a region-specific or audience-limited tool. Because there is no opt-in, alternative language option, or documented justification for a Chinese-only constraint, this creates a natural-language locale policy concern.

Content

No source excerpt is available for this finding.

Natural-Language Policy Violations

Medium
Category
Not specified by scanner
Confidence
91% confidence
Finding

SQP-3 applies to all file types and covers language or locale policy violations. This file's instructional content is fully Chinese-language and does not indicate that the user can opt into another language or that the locale restriction is intentional for a region-specific use case.

Content

No source excerpt is available for this finding.

Natural-Language Policy Violations

Medium
Category
Not specified by scanner
Confidence
95% confidence
Finding

This Python file contains user-facing natural-language strings entirely in Chinese, beginning with the module description and continuing through CLI help and runtime messages. Under the stated policy, forcing a specific language without user opt-in is a locale-policy violation unless the constraint is explicitly justified, which is not documented here.

Content

No source excerpt is available for this finding.

Natural-Language Policy Violations

Medium
Category
Not specified by scanner
Confidence
83% confidence
Finding

The entire skill content is written as mandatory Chinese-language operational guidance and does not indicate that the user may choose another language or locale. This can violate language/locale policy because it imposes a specific language without explicit opt-in or documented regional justification.

Content

No source excerpt is available for this finding.

Missing User Warnings

Medium
Category
Not specified by scanner
Confidence
89% confidence
Finding

The skill instructs the agent to read and record case data in local reference files and to maintain iterative lesson/case logs, but it does not mention persistence, retention limits, or user consent. That creates a risk of storing sensitive prompts, derived conclusions, or proprietary information to disk without the user's knowledge, potentially causing privacy leakage and unintended cross-session data retention.

Content

No source excerpt is available for this finding.

Missing User Warnings

Medium
Category
Not specified by scanner
Confidence
96% confidence
Finding

The skill repeatedly mandates web_search for unknowns, technical details, dates, customs, and any ambiguity, but never warns that queries may send user-provided or sensitive context to external services. This can cause unnecessary external disclosure of private data and expands the attack surface by making outbound retrieval a required behavior rather than an explicit, user-consented choice.

Content

No source excerpt is available for this finding.

Natural-Language Policy Violations

Low
Category
Not specified by scanner
Confidence
97% confidence
Finding

This markdown file presents the skill title and all user-facing instructions in Chinese, and nowhere indicates that users may choose another language or that the skill is intentionally limited to a Chinese-speaking context. Under the language/locale policy rule, forcing a specific language without user opt-in is a natural-language policy concern.

Content

No source excerpt is available for this finding.

Description-Behavior Mismatch

Low
Category
Not specified by scanner
Confidence
85% confidence
Finding

The manifest describes the skill only as "holmes skill," which does not meaningfully convey that the skill is intended for multi-step causal inference and enforced web search when knowledge is uncertain. This creates a semantic mismatch between the claimed scope and the documented operational behavior in the skill materials.

Content

No source excerpt is available for this finding.