Back to skill

Security audit

Arxiv Agentic Verifier

Security checks for vulnerabilities and agentic risk

Overview

This skill does what it claims, but it runs submitted programs directly on the user's machine, so it should be reviewed and sandboxed before installation.

Use this only inside a disposable, locked-down sandbox or container with no personal files, no inherited secrets, restricted network access, and tight CPU, memory, process, and filesystem limits. Do not run it directly on a workstation or shared server, and treat submitted problems and code as data that may be sent to OpenAI when an API key is configured.

Vulnerability Patterns
  • Insecure Skill Coding PracticesFinds exploitable flaws such as hardcoded secrets or command injection
  • Skill Instruction HijackingAlters the agent's session goals or safety constraints when the skill loads
  • Agent Memory PoisoningWrites attacker-controlled rules into memory that affect later sessions
  • Remote Payload Retrieval and ExecutionFetches external code whose behavior can change after review
  • Embedded Malicious CodeShips malicious scripts inside the skill and executes them locally
Findings (2)

T09 · Insecure Skill Coding Practices

Error
Location
index.js:89
Finding

Unsandboxed Execution of Untrusted Candidate Code

Content
View full analysis

Vulnerability Details

File Location: index.js:89-110
Vulnerability Type: Arbitrary code execution without security isolation
Risk Level: High

Vulnerable Code

javascript
runCode(code, input, language = 'python') {
  const filename = `test_run_${Date.now()}.${language === 'python' ? 'py' : 'js'}`;
  const filepath = path.join(this.tempDir, filename);
  
  fs.writeFileSync(filepath, code);

  let command;
  if (language === 'python') {
    command = `python3 ${filepath}`;
  } else {
    command = `node ${filepath}`;
  }

  try {
    // Basic execution with input piping
    // Note: This is simplified. Real competitive coding runners need sandboxing.
    const output = execSync(command, { input: input, timeout: 5000, encoding: 'utf-8' });
    return { success: true, actualOutput: output.trim() };
  } catch (error) {
    return { success: false, error: error.message, actualOutput: error.stdout ? error.stdout.toString() : '' };
  } finally {
    if (fs.existsSync(filepath)) fs.unlinkSync(filepath);
  }
}

Technical Analysis

The code parameter is written directly to a temporary source file and executed using the locally installed Python or Node.js interpreter. No sandbox, container, privilege boundary, filesystem restriction, network restriction, environment sanitization, or syscall policy is applied.

The five-second execSync timeout is only an execution-duration control. It does not prevent the submitted program from reading files, accessing environment variables, making network requests, modifying resources, invoking additional executables, or starting detached child processes. Deleting the source file afterward does not reverse actions already performed by the submitted program.

Although command construction uses an internally generated path rather than directly interpolating attacker-controlled shell text, the intended input to this function is itself executab ...[truncated 1465 chars]

Remediation
View remediation

Remediation Suggestions

  • Execute every candidate program inside a disposable sandbox or container rather than directly on the verifier host.
  • Run the sandbox as a dedicated, unprivileged user with no access to host credentials, application secrets, source files, or sensitive mounted volumes.
  • Remove secrets such as OPENAI_API_KEY from the execution environment by using a strict environment-variable allowlist.
  • Disable outbound and inbound network access unless explicitly required.
  • Use an isolated, size-limited temporary filesystem and mount all other filesystems read-only or make them inaccessible.
  • Enforce CPU, memory, wall-clock, file-size, open-file, and process-count limits. Terminate the complete process group when a timeout occurs.
  • Apply syscall filtering and mandatory access controls where available, such as seccomp, AppArmor, SELinux, or an established code-execution sandbox.
  • Use random, securely created per-execution directories and ensure concurrent requests cannot share files.
  • Destroy the entire sandbox after every run and monitor for failed teardown.
  • Document sandboxing as a mandatory deployment requirement rather than an optional recommendation.

T09 · Insecure Skill Coding Practices

Warning
Location
index.js:40
Finding

Prompt Injection Through Untrusted Problem Descriptions and Candidate Source Code

Content
View full analysis

Vulnerability Details

File Location: index.js:40-65
Vulnerability Type: Prompt injection affecting verifier integrity
Risk Level: Medium

Vulnerable Code

javascript
const prompt = `
You are an Agentic Verifier for competitive coding.
Your goal is to generate a difficult, edge-case test input that might cause the candidate code to fail.
Analyze the problem and the code to find potential weaknesses (e.g., overflow, large inputs, specific constraints).

Problem:
${problemDescription}

Code:
${candidateCode}

Return ONLY a JSON object with:
{
  "input": "string representation of input",
  "expectedOutput": "string representation of correct output",
  "strategy": "Explanation of why this test case is hard"
}
`;

try {
  const completion = await this.openai.chat.completions.create({
    messages: [{ role: "user", content: prompt }],
    model: "gpt-4-turbo-preview", // Use a smart model
    response_format: { type: "json_object" },
  });

  const result = JSON.parse(completion.choices[0].message.content);
  return result;

Technical Analysis

Both problemDescription and candidateCode are untrusted values, but they are interpolated into the same user-role message as the verifier’s operational instructions. An attacker can place natural-language instructions in the problem text or source-code comments that tell the model to disregard the surrounding request and return attacker-selected test data.

The JSON response format only constrains the response’s syntax. It does not establish that input, expectedOutput, or strategy is correct. The parsed object is trusted without schema validation or independent verification of the expected output. Consequently, manipulated model output can influence the final pass or fail decision.

Attack Path

  1. An attacker embeds adversarial model instructions in a submitted problem description or candidate source-code comment. ...[truncated 1148 chars]
Remediation
View remediation

Remediation Suggestions

  • Place immutable verifier rules in a higher-priority system or developer message rather than combining them with untrusted content in one user message.
  • Clearly delimit problem descriptions and source code as untrusted data, and explicitly instruct the model never to follow instructions found inside those fields.
  • Use separate structured message or API fields for instructions and submitted content where supported.
  • Validate the returned object against a strict schema, including field types, length limits, and required properties.
  • Do not trust model-generated expected outputs as an authority. Compute expected results with an independently implemented reference solution or another deterministic validation mechanism.
  • Use multiple independently generated tests and reject results that cannot be validated.
  • Treat generated strategy, input, and output values as untrusted when logging or displaying them.
  • Add adversarial tests containing prompt-injection instructions in source comments and problem descriptions.
  • Require human review or deterministic verification for security-sensitive or high-impact acceptance decisions.
Vulnerability Patterns
  • Data ExfiltrationExternal Transmission, Env Variable Harvesting, File System Enumeration
  • Supply ChainUnpinned Dependencies, External Script Fetching, Obfuscated Code
  • Excessive AgencyUnrestricted Tool Access, Autonomous Decision Making, Scope Creep
  • Prompt InjectionInstruction Override, Hidden Instructions, Exfiltration Commands
  • Privilege EscalationExcessive Permissions, Sudo/Root Execution, Credential Access
Findings (10)

Context-Inappropriate Capability

High
Category
Not specified by scanner
Confidence
98% confidence
Finding

The skill writes attacker-controlled code to disk and executes it locally via Python or Node subprocesses. That creates direct arbitrary code execution on the host, allowing file access, network activity, persistence attempts, or theft of environment secrets from the machine running the skill.

Content

No source excerpt is available for this finding.

Missing User Warnings

High
Category
Not specified by scanner
Confidence
99% confidence
Finding

The skill stores untrusted code in a temporary file and launches it without any warning, confirmation, or containment. In this context, that makes the dangerous behavior more severe because users may not realize supplied code will be executed locally, leading to silent compromise of the host system.

Content

No source excerpt is available for this finding.

Known Vulnerable Dependency: axios==1.13.5 — 16 advisory(ies): CVE-2026-44494 (axios Vulnerable to Full Man-in-the-Middle via Prototype Pollution Gadget in `co); CVE-2026-44495 (axios Vulnerable to Credential Theft and Response Hijacking via Prototype Pollut); CVE-2025-62718 (Axios has a NO_PROXY Hostname Normalization Bypass that Leads to SSRF) +13 more

High
Category
Supply Chain
Confidence
94% confidence
Finding

The lockfile pins axios 1.13.5, and the supplied advisory set includes multiple high-severity issues, including SSRF-related proxy bypass and prototype-pollution-driven request/response manipulation. In an agent skill that likely performs outbound HTTP requests to fetch or verify remote content, a vulnerable HTTP client materially increases the chance of request tampering, credential leakage, or access to unintended internal resources.

Content

No source excerpt is available for this finding.

Known Vulnerable Dependency: form-data==4.0.5 — 1 advisory(ies): CVE-2026-12143 (form-data: CRLF injection in form-data via unescaped multipart field names and f)

High
Category
Supply Chain
Confidence
87% confidence
Finding

form-data 4.0.5 is flagged for CRLF injection via unescaped multipart field names/filenames. If this skill ever builds multipart requests from untrusted input, an attacker could manipulate request structure or inject unintended headers/body parts, which can enable request smuggling-style effects or downstream parser abuse.

Content

No source excerpt is available for this finding.

Known Vulnerable Dependency: axios==1.13.5 — 16 advisory(ies): CVE-2026-44494 (axios Vulnerable to Full Man-in-the-Middle via Prototype Pollution Gadget in `co); CVE-2026-44495 (axios Vulnerable to Credential Theft and Response Hijacking via Prototype Pollut); CVE-2025-62718 (Axios has a NO_PROXY Hostname Normalization Bypass that Leads to SSRF) +13 more

High
Category
Supply Chain
Confidence
97% confidence
Finding

The project allows installation of axios versions in the vulnerable 1.x range via ^1.6.0, and the static finding indicates a resolved install could be axios 1.13.5 with multiple advisories including SSRF-related issues. Because this skill appears to be an agentic verifier that likely performs network requests, a vulnerable HTTP client materially increases risk of request manipulation, SSRF, proxy bypass, credential leakage, or response hijacking depending on usage.

Content

No source excerpt is available for this finding.

Context-Inappropriate Capability

Medium
Category
Not specified by scanner
Confidence
86% confidence
Finding

The constructor silently pulls OPENAI_API_KEY from process.env, which is access to local secrets. Because no manifest or stated purpose is provided, this credential access is not justified by declared intent and expands the skill's effective capabilities.

Content

No source excerpt is available for this finding.

Missing User Warnings

Medium
Category
Not specified by scanner
Confidence
94% confidence
Finding

The skill sends the full problem description and candidate code to an external LLM API without explicit disclosure or opt-in. If the submitted code or prompt contains proprietary, regulated, or secret material, this can cause unintended data exfiltration to a third party.

Content

No source excerpt is available for this finding.

Known Vulnerable Dependency: follow-redirects==1.15.11 — 1 advisory(ies): CVE-2026-40895 (follow-redirects leaks Custom Authentication Headers to Cross-Domain Redirect Ta)

Low
Category
Supply Chain
Confidence
84% confidence
Finding

follow-redirects 1.15.11 is reported as leaking custom authentication headers across cross-domain redirects. Because axios depends on this package and the skill appears network-capable, an attacker controlling a redirect target could potentially exfiltrate bearer tokens, API keys, or other sensitive headers during outbound requests.

Content

No source excerpt is available for this finding.

Unpinned Dependencies

Low
Category
Supply Chain
Confidence
40% confidence
Finding

Dependencies lack version pinning, allowing potential malicious package updates. Consider pinning versions.

Content

Scanner excerpt · package.json (reported line 11)May include surrounding context.

json
"start": "node scripts/run.js"
  },
  "dependencies": {
    "axios": "^1.6.0",
    "openai": "^4.104.0"
  },
  "keywords": [

Unpinned Dependencies

Low
Category
Supply Chain
Confidence
40% confidence
Finding

Dependencies lack version pinning, allowing potential malicious package updates. Consider pinning versions.

Content

Scanner excerpt · package.json (reported line 12)May include surrounding context.

json
},
  "dependencies": {
    "axios": "^1.6.0",
    "openai": "^4.104.0"
  },
  "keywords": [
    "arxiv",

Static analysis

Detected: suspicious.dangerous_exec

Shell command execution detected (child_process).

Critical
Code
suspicious.dangerous_exec
Location
index.js:105