Back to skill

Security audit

Agent Survey Corpus

Security checks for vulnerabilities and agentic risk

Overview

The advertised arXiv PDF helper is partly real, but the package also bundles broad research-pipeline tooling and has path handling weaknesses that can exceed the stated `ref/` boundary.

Review this package before installing and use it only in an isolated workspace. The simple downloader should be limited to trusted arXiv IDs under `ref/agent-surveys/`, but the package also ships broader pipeline machinery that can create or modify many project files. Do not pass untrusted `--inputs` values or untrusted identifier files unless path containment and arXiv ID validation are fixed.

Vulnerability Patterns
  • Insecure Skill Coding PracticesFinds exploitable flaws such as hardcoded secrets or command injection
  • Skill Instruction HijackingAlters the agent's session goals or safety constraints when the skill loads
  • Agent Memory PoisoningWrites attacker-controlled rules into memory that affect later sessions
  • Remote Payload Retrieval and ExecutionFetches external code whose behavior can change after review
  • Embedded Malicious CodeShips malicious scripts inside the skill and executes them locally
Findings (2)

T09 · Insecure Skill Coding Practices

Warning
Location
scripts/run.py:269
Finding

Workspace Escape Enables Arbitrary Local File Reads and Identifier-Derived Network Disclosure

Content
View full analysis
dict[str, Any]: url = f"https://export.arxiv.org/api/query?id_list={arxiv_id}" req = urllib.request.Request(url, headers={"User-Agent": "codex-agent-survey-corpus/1.0"}) with urllib.request.urlopen(req, timeout=30) as resp: data = resp.read() ``` ```python for arxiv_id in arxiv_ids: rec: dict[str, Any] = { "arxiv_id": arxiv_id, "pdf_url": f"https://arxiv.org/pdf/{arxiv_id}.pdf", "pdf_path": str((pdf_dir / f"{arxiv_id}.pdf").relative_to(workspace)), "text_path": str((text_dir / f"{arxiv_id}.txt").relative_to(workspace)), "max_pages": int(args.max_pages), "status": "", "error": "", "pages_extracted": 0, } try: rec.update(_fetch_arxiv_meta(arxiv_id)) ``` ### Technical Analysis `--inputs` is treated as a workspace-relative path without enforcing that constraint. In `pathlib`, when the right-hand operand of `/` is absolute, the left-hand workspace path is discarded. Relative traversal components such as `../` are also not rejected. Consequently, a caller that controls `--inputs`, or a pipeline unit whose `inputs` field is attacker-controlled, can direct `_read_ids()` to any readable text file accessible to the process. `_read_ids()` interprets each nonempty, non-comment line as an arXiv identifier. Those lines are then in ...[truncated 1991 chars]
Remediation
View remediation

T09 · Insecure Skill Coding Practices

Warning
Location
scripts/run.py:291
Finding

Unvalidated arXiv Identifiers Permit Output Path Traversal Outside the Workspace

Content
View full analysis
list[str]: ids: list[str] = [] if not path.exists(): raise SystemExit(f"Missing id list: {path}") for raw in path.read_text(encoding="utf-8", errors="ignore").splitlines(): line = raw.strip() if not line or line.startswith("#"): continue line = line.replace("https://arxiv.org/abs/", "").replace("http://arxiv.org/abs/", "") line = line.replace("https://arxiv.org/pdf/", "").replace("http://arxiv.org/pdf/", "") line = line.removesuffix(".pdf") ids.append(line) ``` The unchecked value is incorporated directly into output paths: ```python for arxiv_id in arxiv_ids: rec: dict[str, Any] = { "arxiv_id": arxiv_id, "pdf_url": f"https://arxiv.org/pdf/{arxiv_id}.pdf", "pdf_path": str((pdf_dir / f"{arxiv_id}.pdf").relative_to(workspace)), "text_path": str((text_dir / f"{arxiv_id}.txt").relative_to(workspace)), "max_pages": int(args.max_pages), "status": "", "error": "", "pages_extracted": 0, } ``` The paths are later reconstructed and written: ```python pdf_path = workspace / rec["pdf_path"] text_path = workspace / rec["text_path"] try: _download(rec["pdf_url"], pdf_path, overwrite=bool(args.overwrite), sleep_s=float(args.sleep)) pages, text = _extract_text(pdf_path, max_pages=int(args.max_pages)) rec["pages_extracted"] = int(pages) if bool(args.overwrite) or not text_path.exists(): text_path.write_text(text, encoding="utf-8") ``` The downloader creates parent directories and writes the response: ```python def _download(url: str, dst: Path, *, overwrite: bool, sleep_s ...[truncated 2984 chars]
Remediation
View remediation
Vulnerability Patterns
  • Excessive AgencyUnrestricted Tool Access, Autonomous Decision Making, Scope Creep
  • Trigger AbuseOverly Broad Trigger, Shadow Command Trigger, Keyword Baiting Trigger
  • MCP Least PrivilegeUnderdeclared Capability, Wildcard Permission, Missing Permission Declaration
  • MCP Tool PoisoningHidden Instructions, Unicode Deception, Parameter Description Injection
  • Prompt InjectionInstruction Override, Hidden Instructions, Exfiltration Commands
Findings (41)

Tp4

High
Category
MCP Tool Poisoning
Confidence
99% confidence
Finding

The declared description is narrowly about fetching arXiv survey/review PDFs for an LLM-agent reference corpus and extracting their text. The supplied code chunk instead is a generic utility module for local repository/workspace management: atomic writes, backups, YAML/JSONL/TSV/CSV read-write helpers, copying trees, title/label normalization, keyword extraction from titles, status and decisions log mutation, human approval checklist handling, repo-root and pipeline-spec discovery, and seeding query files from a topic. These behaviors are materially different from the declared primary purpose. There is no evidence of network calls, HTTP clients, PDF download logic, arXiv URL validation, storage under ref/, or PDF parsing/text extraction. This is therefore a clear description-behavior mismatch.

Content

No source excerpt is available for this finding.

Tp4

High
Category
MCP Tool Poisoning
Confidence
99% confidence
Finding

The declared description is about acquiring arXiv PDFs and extracting their text for a reference corpus. The provided code does none of that: it contains no PDF downloading, no network access, no arXiv filtering, no storage under ref/, and no text extraction. Instead, its primary purpose is orchestrating execution of workspace pipeline units, including dependency resolution, approvals, subprocess invocation, output validation, and error/status logging. This is a materially different function and introduces substantial undeclared capabilities unrelated to the stated corpus-download skill.

Content

No source excerpt is available for this finding.

Tp4

High
Category
MCP Tool Poisoning
Confidence
99% confidence
Finding

The declared description says this skill should fetch arXiv survey/review PDFs and extract their text for reference/style learning, with network access required and storage under ref/. The supplied code chunk contains no networking, no URL handling, no arXiv filtering, no PDF parsing, and no file placement under ref/. Instead, it is a substantial ideation/reporting module that processes existing notes and workspace configs to produce research-signal tables, direction pools, screening tables, shortlist memos, JSON payloads, and appendices. This is a materially different primary purpose and capability set, so the description does not accurately represent the code.

Content

No source excerpt is available for this finding.

Tp4

High
Category
MCP Tool Poisoning
Confidence
99% confidence
Finding

The declared description is about downloading arXiv PDFs and extracting their text for stylistic study. The supplied code does none of that. It only reads local files, parses YAML front matter, validates schema-like fields, resolves variant_of references, and merges configuration structures for pipeline specs. This is a materially different primary purpose and resource profile: local configuration parsing rather than network-based PDF collection and text extraction. Therefore the description does not accurately represent the code.

Content

No source excerpt is available for this finding.

Tp4

High
Category
MCP Tool Poisoning
Confidence
98% confidence
Finding

The declared description centers on downloading arXiv survey PDFs and extracting their text for corpus/style learning. The supplied code instead provides generic local helpers for processing outlines, JSONL, BibTeX, markdown headings, and citation formatting. These behaviors are materially different from the declared primary purpose. There is no implementation for network requests, PDF handling, filesystem placement under ref/, or arXiv-only restrictions. This is therefore a clear description-behavior mismatch.

Content

No source excerpt is available for this finding.

Tp4

High
Category
MCP Tool Poisoning
Confidence
99% confidence
Finding

The declared description is about network-based acquisition of arXiv survey PDFs and text extraction into a local reference corpus. The supplied code instead is a generic quality-control module for a paper/survey generation pipeline. It parses configuration from queries.md, validates outputs for many named skills, checks citation/mapping/outline/taxonomy/draft artifacts, and writes a QUALITY_GATE.md status report. The only PDF-related logic is a validator for a preexisting fulltext index in _check_pdf_text_extractor; it does not itself download or extract PDFs. Resource usage also differs: declared behavior requires network and storing PDFs under ref/, while this code primarily reads and writes local workspace metadata and reports. This is a clear description-behavior mismatch with materially different primary purpose and capabilities.

Content

No source excerpt is available for this finding.

Tp4

High
Category
MCP Tool Poisoning
Confidence
99% confidence
Finding

The declared description says the skill downloads arXiv PDFs and extracts their text for style learning. The supplied code does none of that. Instead, it reads and validates many local files under paths like outline/, sections/, output/, citations/, and papers/, enforcing schema, completeness, citation, and prose-quality constraints for a writing pipeline. This is a materially different primary purpose and resource profile: local artifact validation versus network-based PDF acquisition and extraction. Therefore the description does not accurately represent the code.

Content

No source excerpt is available for this finding.

Tp4

High
Category
MCP Tool Poisoning
Confidence
99% confidence
Finding

The declared description is about acquiring an arXiv survey corpus and extracting PDF text. The provided code chunk does something fundamentally different: it validates locally produced markdown/LaTeX research-writing artifacts using heuristics around citations, structure, repetition, and build success. The code reads workspace files such as drafts, reports, TeX, logs, and PDFs, but contains no HTTP/network logic, no arXiv-only restriction enforcement, no PDF download handling, and no extraction pipeline for reference corpus creation. This is a material description-behavior mismatch, not just an implementation detail difference.

Content

No source excerpt is available for this finding.

Description-Behavior Mismatch

High
Category
Not specified by scanner
Confidence
99% confidence
Finding

The manifest claims this skill downloads open-access arXiv survey PDFs for style learning, but the actual file defines a Chinese graduate-thesis reconstruction pipeline with a different purpose, workflow, outputs, and likely permissions/use context. This mismatch is dangerous because agents or users may invoke the skill under false assumptions, causing unauthorized access to local thesis materials, unexpected file modifications, or execution in contexts where the user only consented to a narrow corpus-download task.

Content

No source excerpt is available for this finding.

Scope Creep

High
Category
Not specified by scanner
Confidence
95% confidence
Finding

The code writes to broad workspace locations such as output/RUN_ERRORS.md, STATUS.md, UNITS.csv, DECISIONS.md, and other workflow files instead of remaining within the manifest's stated ref/ storage boundary. For a skill advertised as only downloading arXiv PDFs and extracting text, this violates least privilege and can alter project control/state files in ways the user would not expect.

Content

No source excerpt is available for this finding.

Description-Behavior Mismatch

High
Category
Not specified by scanner
Confidence
98% confidence
Finding

The file implements a general workflow executor that mutates workspace state, runs arbitrary pipeline units, and handles approvals/checkpoints, which is materially broader than the declared skill purpose of downloading arXiv survey PDFs into ref/. That mismatch is dangerous because users may authorize a narrowly scoped reference-corpus skill while actually introducing a generic orchestration primitive capable of driving unrelated actions across the workspace.

Content

No source excerpt is available for this finding.

Context-Inappropriate Capability

High
Category
Not specified by scanner
Confidence
95% confidence
Finding

The code constructs and executes scripts/run.py with user/workspace-derived parameters, giving the skill a generic subprocess execution path unrelated to the advertised corpus-download task. In the context of a supposedly narrow data-download skill, this creates an unjustified execution capability that could be repurposed to run arbitrary workflow actions if the surrounding repository scripts are modified or malicious.

Content

No source excerpt is available for this finding.

Description-Behavior Mismatch

High
Category
Not specified by scanner
Confidence
99% confidence
Finding

This file materially diverges from the declared skill purpose. Instead of downloading only arXiv PDFs into a constrained ref/ corpus, it implements a large ideation and memo-generation pipeline that can transform workspace inputs into new analytical outputs, increasing capability scope beyond user expectations and declared guardrails. In an agent skill, this kind of hidden scope expansion is dangerous because it can cause unauthorized data processing and artifact generation under the guise of a narrow reference-corpus tool.

Content

No source excerpt is available for this finding.

Context-Inappropriate Capability

High
Category
Not specified by scanner
Confidence
97% confidence
Finding

The direction-scoring, screening, and shortlist/report construction logic gives the skill broad research-analysis capabilities unrelated to 'style learning' from survey PDFs. That mismatch increases the risk of confused-deputy behavior: operators may grant network/file access expecting a narrow downloader, while the code performs higher-level inference, ranking, and output synthesis on workspace data. The skill context makes this more dangerous because the declared guardrail is narrow, so undeclared analytical behavior is especially out of policy.

Content

No source excerpt is available for this finding.

Description-Behavior Mismatch

High
Category
Not specified by scanner
Confidence
99% confidence
Finding

This file implements a large multi-stage survey-writing and pipeline-orchestration quality gate, far beyond the declared skill purpose of downloading arXiv survey PDFs into ref/ for style learning. That mismatch is dangerous because activating this skill can read, validate, and write many unrelated workspace artifacts, violating least privilege and potentially causing unintended modifications or data exposure in projects that expected a narrowly scoped corpus downloader.

Content

No source excerpt is available for this finding.

Scope Creep

High
Category
Not specified by scanner
Confidence
99% confidence
Finding

The code reads from and writes to many workspace locations such as output/, outline/, papers/, citations/, latex/, and sections/, which conflicts with the guardrail that this skill should store downloads under ref/. In this context, the mismatch is especially risky because users would reasonably trust a corpus-download skill not to modify broad portions of their workspace.

Content

No source excerpt is available for this finding.

Undeclared Tool Scope

Medium
Category
MCP Least Privilege
Confidence
93% confidence
Finding

The skill declares executable prerequisites and provides shell-based workflow instructions, but it does not declare any explicit tool/permission scope. In an agent ecosystem, missing scope boundaries can let the runtime infer or grant broader file, shell, and network capabilities than reviewers expect, which weakens least-privilege controls.

Content

No source excerpt is available for this finding.

Vague Triggers

Medium
Category
Not specified by scanner
Confidence
89% confidence
Finding

Broad triggers such as 'download surveys' and 'ref corpus' can cause the skill to activate for ordinary research requests, unintentionally granting a networked, file-writing workflow in contexts where the user did not explicitly ask for downloads. In a multi-skill agent, overbroad invocation criteria increase the chance of surprise side effects and misuse.

Content

No source excerpt is available for this finding.

Vague Triggers

Medium
Category
Not specified by scanner
Confidence
95% confidence
Finding

The pipeline enables routing_default: true with broad multilingual routing hints such as survey, review, and literature review, which are common terms in many benign research requests. That increases the chance this skill is auto-invoked outside the user's intent, leading to unintended network access, downloads, and large workspace modifications. The skill context makes this more dangerous because the pipeline is high-impact: it performs multi-stage retrieval, PDF extraction, and extensive artifact generation rather than a narrow read-only action.

Content

No source excerpt is available for this finding.

Missing User Warnings

Medium
Category
Not specified by scanner
Confidence
92% confidence
Finding

The skill enumerates a very large set of generated artifacts and describes overwrite/freeze behavior, but it does not present a clear upfront warning that execution will create and modify many files across the workspace. Users may trigger it expecting lightweight corpus download behavior and instead get broad filesystem changes, making accidental data clutter, overwrite confusion, and unsafe operation in sensitive repositories more likely. The risk is elevated here because the pipeline is explicitly stateful, iterative, and designed to write dozens of files over multiple stages.

Content

No source excerpt is available for this finding.

Natural-Language Policy Violations

Medium
Category
Not specified by scanner
Confidence
95% confidence
Finding

This markdown repeatedly frames the pipeline as specifically for Chinese thesis writing, starting with the title and core goal. Because the file does not present this as an optional language setting or user opt-in, it reads as a forced language/locale constraint, which matches the policy-violation category for natural-language locale restrictions.

Content

No source excerpt is available for this finding.

Vague Triggers

Medium
Category
Not specified by scanner
Confidence
91% confidence
Finding

The routing hints are broad and include common ideation terms such as 'idea', 'brainstorm', and general Chinese equivalents, which can cause this pipeline to activate for loosely related requests. Because the pipeline performs multi-stage literature retrieval and artifact generation, accidental invocation can waste resources, create unintended files, and route users into a workflow that exceeds their intent or consent.

Content

No source excerpt is available for this finding.

Vague Triggers

Medium
Category
Not specified by scanner
Confidence
94% confidence
Finding

The routing_hints include very generic terms such as "snapshot" and "one page", which can match many unrelated user requests and cause this pipeline to be invoked unintentionally. In this skill, unintended invocation is more concerning because the pipeline leads into network-enabled literature retrieval and file creation workflows, increasing the chance of unnecessary downloads, workspace changes, or execution of the wrong process.

Content

No source excerpt is available for this finding.

Vague Triggers

Medium
Category
Not specified by scanner
Confidence
91% confidence
Finding

The routing hint list includes very broad English trigger phrases such as "peer review" and "review report," which can match many ordinary user requests beyond this specific pipeline. That can cause unintended activation of the peer-review workflow, leading the agent to operate on the wrong task context or expose/manipulate artifacts not relevant to the user's actual intent.

Content

No source excerpt is available for this finding.

Autonomous Decision Making

Medium
Category
Excessive Agency
Confidence
85% confidence
Finding

Skill enables autonomous high-impact decisions without human-in-the-loop verification. Critical operations (destructive commands, financial transactions, data deletion) should require explicit user confirmation.

Content

Scanner excerpt · tooling/executor.py (reported line 105)May include surrounding context.

python
workspace: Path,
    repo_root: Path,
    strict: bool = False,
    auto_approve: set[str] | None = None,
) -> RunResult:
    units_path = workspace / "UNITS.csv"
    status_path = workspace / "STATUS.md"

Static analysis

No suspicious patterns detected.