Back to skill

Security audit

Github Experiment Accuracy

Security checks for vulnerabilities and agentic risk

Overview

This skill has a coherent accuracy-testing purpose, but it tells agents to load models from arbitrary GitHub repositories using unsafe PyTorch deserialization, which can run untrusted code.

Only use this skill in a disposable, sandboxed environment with no secrets, credentials, or sensitive files available. Do not run it against arbitrary repositories or model files unless the repository, exact commit, and model artifact are trusted and verified; prefer safe model formats or weights-only loading.

Vulnerability Patterns
  • Remote Payload Retrieval and ExecutionFetches external code whose behavior can change after review
  • Skill Instruction HijackingAlters the agent's session goals or safety constraints when the skill loads
  • Agent Memory PoisoningWrites attacker-controlled rules into memory that affect later sessions
  • Embedded Malicious CodeShips malicious scripts inside the skill and executes them locally
  • Unauthorized Access and Privilege EscalationObtains permissions beyond the task's legitimate needs
Findings (1)

T03 · Remote Payload Retrieval and Execution

Error
Location
SKILL.md:23
Finding
Arbitrary Code Execution Through Untrusted PyTorch Model Deserialization## Vulnerability Details **File Location**: `SKILL.md`, lines 23-53 **Vulnerability Type**: Unsafe deserialization of a remotely retrieved model **Risk Level**: Critical ### Vulnerable Code ```bash # Prefer git clone git clone {github_url} {project_dir} --depth 1 ``` ```powershell Invoke-WebRequest -Uri "{github_url}/archive/refs/heads/main.zip" -OutFile "outputs/repo.zip" Expand-Archive -Path "outputs/repo.zip" -DestinationPath "outputs/projects" ``` ```python # Find the model file model_file = glob.glob(f"{project_dir}/**/model.pt", recursive=True)[0] # Load the model net = torch.load(model_file, map_location='cpu', weights_only=False) net.eval() ``` The same unsafe loading behavior is reiterated in `SKILL.md`, lines 126-128, and `references/workflow.md`, lines 25-27: ```python net = load(model_file, map_location='cpu', weights_only=False) net.eval() ``` ```text 1. Find model files (model.pt, *.pth) 2. torch.load(weights_only=False) 3. net.eval() ``` ### Technical Analysis The workflow accepts a user-supplied GitHub repository URL, retrieves its contents, recursively locates a `model.pt` file, and loads that file using `torch.load` with `weights_only=False`. PyTorch model files loaded in this mode can use Python pickle-based object deserialization. Pickle is not a safe format for untrusted input because object reconstruction can invoke attacker-controlled callables. Consequently, a malicious model can execute code during `torch.load`; execution occurs before `net.eval()` and does not require the model to be valid or inference to begin. The exposure is compounded by the absence of repository commit pinning, model checksum verification, publisher authentication, or an approved-model allowlist. Recursive selection of the first matching `model.pt` also allows the remote repository to control which artifact is loaded. This creates a direct remote payload retrieval and execution path: t ...[truncated 2017 chars]
Remediation
## Remediation Suggestions 1. **Prohibit unrestricted deserialization** - Remove all instructions recommending `weights_only=False`. - Load only state dictionaries with `torch.load(..., weights_only=True)` where supported. - Do not fall back automatically to unrestricted loading when safe loading fails. 2. **Prefer non-executable model formats** - Require SafeTensors or another format that cannot embed pickle reconstruction logic. - Reconstruct the model architecture from reviewed local code and load only validated tensor weights. 3. **Authenticate remote inputs** - Restrict repositories to trusted owners or an explicit allowlist. - Pin each repository to a reviewed immutable commit hash rather than a mutable branch. - Require an approved cryptographic checksum for every model artifact and verify it before loading. 4. **Validate model selection** - Do not select the first recursive `model.pt` match. - Require an explicit, expected model path. - Reject symbolic links, path traversal, unexpected file types, and model files outside the canonical repository directory. 5. **Isolate unavoidable legacy loading** - If a legacy pickle model must be inspected, deserialize it only in a disposable sandbox or virtual machine. - Disable outbound network access and remove credentials and secrets from the environment. - Mount the host filesystem read-only or expose only a temporary working directory. - Run as a dedicated unprivileged user with strict CPU, memory, process, and execution limits. - Destroy the sandbox after processing. 6. **Fail securely** - Treat safe-loader failures as validation failures. - Record the repository commit and verified model digest in the report. - Never suggest `weights_only=False` as routine troubleshooting guidance.
Vulnerability Patterns
  • Data ExfiltrationExternal Transmission, Env Variable Harvesting, File System Enumeration
  • Excessive AgencyUnrestricted Tool Access, Autonomous Decision Making, Scope Creep
  • Prompt InjectionInstruction Override, Hidden Instructions, Exfiltration Commands
  • Privilege EscalationExcessive Permissions, Sudo/Root Execution, Credential Access
  • Supply ChainUnpinned Dependencies, External Script Fetching, Obfuscated Code
Findings (6)

Context-Inappropriate Capability

High
Confidence
99% confidence
Finding
The skill instructs loading a PyTorch model taken from an arbitrary GitHub repository with `torch.load(..., weights_only=False)`. PyTorch pickle-based deserialization can execute attacker-controlled code during loading, so simply evaluating an untrusted repo can lead to arbitrary code execution on the analyst's machine.

Missing User Warnings

High
Confidence
99% confidence
Finding
The document recommends torch.load(weights_only=False) for model loading. PyTorch model deserialization can invoke Python pickle behavior, and loading an untrusted .pt/.pth file may lead to arbitrary code execution; the skill context makes this especially dangerous because it processes models from arbitrary GitHub repositories.

Missing User Warnings

Medium
Confidence
88% confidence
Finding
The workflow directs cloning/downloading arbitrary repositories, copying local user-supplied files into them, and writing reports to the filesystem without any explicit warning, consent boundary, or isolation guidance. In this context, the skill is specifically designed to handle untrusted third-party code and local data, so omitting safety constraints materially increases the chance of unsafe execution, data exposure, or unintended host modification.

Missing User Warnings

Medium
Confidence
93% confidence
Finding
The workflow explicitly instructs users to clone or download arbitrary GitHub repositories and place external data into the project without any warning, isolation guidance, or trust boundary discussion. In this skill’s context, the natural next step is to run project code against supplied data, so treating untrusted repositories and inputs as routine significantly increases the chance of executing malicious code or mishandling sensitive data.

Natural-Language Policy Violations

Low
Confidence
95% confidence
Finding
The name and operational description instruct the skill in Chinese and do not indicate that users may choose another language. Under the stated policy, forcing a specific language without opt-in is a natural-language policy violation unless the locale restriction is clearly justified.

Natural-Language Policy Violations

Low
Confidence
91% confidence
Finding
All user-facing instructional content in this file is presented only in Chinese, and there is no indication that the skill is region-specific or that users can opt into this locale. Per the policy rule, forcing a specific language without user opt-in is a natural-language policy concern.

Static analysis

No suspicious patterns detected.