Back to skill

Security audit

Karpathy Wiki

Security checks for vulnerabilities and agentic risk

Overview

This skill is a disclosed research-wiki helper that writes local knowledge-base files and can optionally download arXiv PDFs, with some quality and consent-scope caveats but no evidence of deception or malicious behavior.

Install only if you want an agent to maintain a persistent local research wiki and are comfortable with it creating or updating files under raw/ and wiki/. Run the bundled arXiv downloader only when you explicitly want recent PDFs saved locally, and review diffs or use version control because query and lint workflows can also persist new notes and log entries.

Vulnerability Patterns
  • Skill Instruction HijackingAlters the agent's session goals or safety constraints when the skill loads
  • Agent Memory PoisoningWrites attacker-controlled rules into memory that affect later sessions
  • Remote Payload Retrieval and ExecutionFetches external code whose behavior can change after review
  • Embedded Malicious CodeShips malicious scripts inside the skill and executes them locally
  • Unauthorized Access and Privilege EscalationObtains permissions beyond the task's legitimate needs
Vulnerability Patterns
  • Excessive AgencyUnrestricted Tool Access, Autonomous Decision Making, Scope Creep
  • Trigger AbuseOverly Broad Trigger, Shadow Command Trigger, Keyword Baiting Trigger
  • MCP Least PrivilegeUnderdeclared Capability, Wildcard Permission, Missing Permission Declaration
  • MCP Tool PoisoningHidden Instructions, Unicode Deception, Parameter Description Injection
  • Prompt InjectionInstruction Override, Hidden Instructions, Exfiltration Commands
Findings (12)

Tp4

High
Category
MCP Tool Poisoning
Confidence
97% confidence
Finding

A second aspect of the same mismatch is that the skill claims to support wiki ingestion/maintenance, yet the concrete automation described is focused on fetching recent agent papers from arXiv. When behavior exceeds the declared scope, downstream orchestrators may invoke it in contexts where users did not intend external access or bulk file writes.

Content

No source excerpt is available for this finding.

Tp4

High
Category
MCP Tool Poisoning
Confidence
95% confidence
Finding

A second aspect of the same mismatch is that the skill claims to support wiki ingestion/maintenance, yet the concrete automation described is focused on fetching recent agent papers from arXiv. When behavior exceeds the declared scope, downstream orchestrators may invoke it in contexts where users did not intend external access or bulk file writes.

Content

No source excerpt is available for this finding.

Undeclared Tool Scope

Medium
Category
MCP Least Privilege
Confidence
86% confidence
Finding

The skill advertises and bundles network-capable behavior via a script that queries the arXiv API and downloads PDFs, but it does not declare any explicit tool scope or permissions boundary. In an agent environment, undeclared network access expands the effective trust surface and can lead to unexpected outbound requests or file downloads without clear user awareness or policy gating.

Content

No source excerpt is available for this finding.

Vague Triggers

Medium
Category
Not specified by scanner
Confidence
94% confidence
Finding

The trigger conditions are broad enough that the skill could be activated for ordinary discussions about note-taking, papers, or wiki maintenance. In this skill's context, accidental invocation matters because activation can lead to file creation/modification and potentially network-backed acquisition workflows, increasing the chance of unintended side effects.

Content

No source excerpt is available for this finding.

Missing User Warnings

Medium
Category
Not specified by scanner
Confidence
93% confidence
Finding

The skill describes creating and updating many local files across raw/, wiki/, and logs, but it does not clearly surface these side effects as a user-consent boundary. Silent or insufficiently disclosed filesystem changes are risky because they can overwrite structure, bloat repositories, or persist unintended content beyond the current conversation.

Content

No source excerpt is available for this finding.

Description-Behavior Mismatch

Medium
Category
Not specified by scanner
Confidence
89% confidence
Finding

The skill manifest describes building and maintaining a persistent research knowledge base, which naturally includes organizing notes and papers, but this script goes further by querying the arXiv API and downloading remote PDF files. External paper discovery and bulk retrieval are materially broader behaviors than local wiki maintenance and are not stated in the manifest description.

Content

No source excerpt is available for this finding.

Context-Inappropriate Capability

Medium
Category
Not specified by scanner
Confidence
84% confidence
Finding

The script performs outbound requests to the arXiv API and downloads remote PDF content to disk without any explicit trust controls, domain allowlist enforcement beyond hardcoded URLs, size/content-type validation, or user confirmation. In a skill environment, hidden or insufficiently disclosed network/file-write behavior can expose users to unwanted data transfer, storage abuse, or ingestion of untrusted files that may later be processed by other tooling.

Content

No source excerpt is available for this finding.

Intent-Code Divergence

Medium
Category
Not specified by scanner
Confidence
94% confidence
Finding

The header comments describe usage of a Bash script, and line L012 unconditionally replaces the shell process with a separate Python file. The Python code embedded from L013 onward will never execute in this file, so the documented behavior of this file as the implementation is contradicted by what the code actually does.

Content

No source excerpt is available for this finding.

Natural-Language Policy Violations

Medium
Category
Not specified by scanner
Confidence
90% confidence
Finding

The file's natural-language description and usage comments are written entirely in Chinese, which imposes a specific language context without offering user opt-in or documenting a justified locale restriction. Under the policy, language-specific constraints should either provide choice or be clearly justified as region-specific.

Content

No source excerpt is available for this finding.

Intent-Code Divergence

Low
Category
Not specified by scanner
Confidence
95% confidence
Finding

L030 明确写明知识库层“LLM 全权负责,你只读不写”,表达的是用户不应手动修改 wiki 内容。但从审计角度看,这句话与该 skill 后文的实际操作意图容易形成主体歧义,因为 L061-L067、L075-L080、L093-L096 都要求代理主动写入和修订 wiki/ 文件。文档并非缺失细节,而是用语上将“谁在写”说反了。

Content

No source excerpt is available for this finding.

Natural-Language Policy Violations

Low
Category
Not specified by scanner
Confidence
88% confidence
Finding

L190 明确写明“本 skill 将其适配为中文科研场景”,而全文也以中文固定描述行为和输出格式,没有说明是否可按用户偏好改用其他语言。这构成了语言/locale 的默认强制约束,缺少用户选择或显式 opt-in。

Content

No source excerpt is available for this finding.

Natural-Language Policy Violations

Low
Category
Not specified by scanner
Confidence
89% confidence
Finding

The top-level docstring presents the script description and usage exclusively in Chinese. Under the language/locale policy, forcing a specific language without offering a user choice or documenting a justified locale constraint is a natural-language policy violation.

Content

No source excerpt is available for this finding.

Static analysis

No suspicious patterns detected.