Back to skill

Security audit

Agent Benchmark

Security checks for vulnerabilities and agentic risk

Overview

This benchmark skill has a plausible purpose, but it runs benchmark-supplied code with host privileges and has unsafe file path handling that can overwrite, delete, or persist data outside the expected scope.

Only install or run this skill in a disposable sandbox or container, with no API keys or sensitive files in the environment. Do not run custom task files from untrusted sources, and expect the runner to create reports, write to a memory path, and delete temporary directories. The package should be fixed before normal use: validate task IDs and CLI paths, remove inherited secrets, avoid automatic memory writes, and align the documented PowerShell runner with the shipped executable.

Vulnerability Patterns
  • Agent Memory PoisoningWrites attacker-controlled rules into memory that affect later sessions
  • Insecure Skill Coding PracticesFinds exploitable flaws such as hardcoded secrets or command injection
  • Skill Instruction HijackingAlters the agent's session goals or safety constraints when the skill loads
  • Remote Payload Retrieval and ExecutionFetches external code whose behavior can change after review
  • Embedded Malicious CodeShips malicious scripts inside the skill and executes them locally
Findings (4)

T09 · Insecure Skill Coding Practices

Error
Location
index.js:76
Finding

Untrusted benchmark tasks execute with host privileges and inherited environment secrets

Content
View full analysis
{ const child = spawn(language.command.split(' ')[0], [...language.command.split(' ').slice(1), fullFile], { timeout: CONFIG.timeout, cwd: tempDir, env: { ...process.env, NODE_ENV: 'test' } } ); ``` ### Technical Analysis The benchmark runner treats `task.input.code` and `task.input.test_code` as executable source code and invokes the corresponding host interpreter without an isolation boundary. The spawned process: - Runs with the same operating-system identity and privileges as the benchmark runner. - Inherits all environment variables through `{ ...process.env }`. - Retains normal access to the host filesystem. - Is not prevented from making network connections. - Can create additional child processes. - Is not subject to the declared `CONFIG.memoryLimit`; that value is never enforced. The timeout only attempts to terminate the immediate process. It does not establish filesystem, network, process-count, privilege, or memory isolation. It may also fail to terminate descendants created by the executed task. Although the bundled root tasks are benign, support for custom task files makes task content an executable trust boundary. ### Attack Path 1. An attacker supplies or modifies a custom benchmark task file. 2. The task places a malicious Python, Node.js, or Go payload in `input.co ...[truncated 987 chars]
Remediation
View remediation

T09 · Insecure Skill Coding Practices

Error
Location
index.js:68
Finding

Task ID path traversal can overwrite files and recursively delete directories outside the temporary root

Content
View full analysis
Remediation
View remediation

T09 · Insecure Skill Coding Practices

Warning
Location
index.js:263
Finding

Unvalidated CLI paths allow files outside the project to be read or overwritten

Content
View full analysis
args[i - 1] === '--tasks') || 'tasks.json'; const outputFile = args.find((_, i) => args[i - 1] === '--output') || 'benchmark-report.md'; // Load tasks let tasks; try { const taskPath = path.join(__dirname, taskFile); tasks = JSON.parse(fs.readFileSync(taskPath, 'utf-8')); } catch (error) { console.error('❌ Unable to load task file:', error.message); process.exit(1); } // Run benchmark const results = await runBenchmark(tasks); // Generate report generateReport(results, path.join(__dirname, outputFile)); // Save to memory directory const memoryReport = path.join(__dirname, '../../memory/benchmark-results.md'); generateReport(results, memoryReport); ``` ### Technical Analysis The values supplied through `--tasks` and `--output` are joined with `__dirname`, but they are not restricted to the project directory. Absolute paths and traversal components can therefore select files outside the intended input and output locations. The task input must contain valid JSON, which limits but does not eliminate unauthorized reads. The output path is more dangerous because `fs.writeFileSync()` overwrites an existing writable file with the generated Markdown report. This issue becomes exploitable when an untrusted user, wrapper, automation system, or Agent-generated command can influence the benchmark's CLI arguments. ### Attack Path 1. An attacker gains influence over the arguments used to invoke the benchmark. 2. For unauthorized input access, the attacker passes a traversal or absolute path through `--tasks`. 3. The runner reads the selected file and attempts to parse it as JSON. 4. For file overwrite, ...[truncated 691 chars]
Remediation
View remediation

T02 · Agent Memory Poisoning

Warning
Location
index.js:202
Finding

Attacker-controlled benchmark content is persisted into Agent memory without sanitization

Content
View full analysis
`| ${r.taskId} | ${getStatusIcon(r.status)} | ${r.executionTime}ms | ${r.testsPassed}/${r.testsTotal} | ${r.error ? truncate(r.error, 30) : '-'} |`).join('\n')} --- ## Task Details ${results.map(r => `### ${r.taskId} - **Status**: ${r.status} - **Execution time**: ${r.executionTime}ms - **Exit code**: ${r.exitCode} - **Output**: \`${truncate(r.output, 100)}\` - **Error**: \`${truncate(r.error, 100)}\` --- `).join('\n')} ``` The resulting report is automatically written to a persistent memory path: ```js // Save to memory directory const memoryReport = path.join(__dirname, '../../memory/benchmark-results.md'); generateReport(results, memoryReport); ``` ### Technical Analysis The report generator does not escape Markdown metacharacters in `taskId`, process output, or process errors. A custom task controls its ID and can also print chosen content to standard output or standard error. Truncation limits string length but does not make the content safe. In particular, a malicious task ID can inject line breaks, headings, table cells, links, or instruction-like Markdown outside the intended report structure. The report is then written automatically to a directory explicitly identified as `memory`. If the surrounding Agent loads that file into later contexts, attacker-controlled text can persist beyond the benchmark execution and influence future Agent behavior. The code does not prove that every deployment automatically consumes this memory file. The security impact depends on the surrounding Agent's memory-loading behavior, but writing untrusted content to that location creates a concrete poisoning channel. ### Attack Path 1. A ...[truncated 1134 chars]
Remediation
View remediation
Vulnerability Patterns
  • Prompt InjectionInstruction Override, Hidden Instructions, Exfiltration Commands
  • Data ExfiltrationExternal Transmission, Env Variable Harvesting, File System Enumeration
  • Excessive AgencyUnrestricted Tool Access, Autonomous Decision Making, Scope Creep
  • Trigger AbuseOverly Broad Trigger, Shadow Command Trigger, Keyword Baiting Trigger
  • Behavioral ASTexec() Call, eval() Call, Dynamic Import
Findings (28)

Context-Inappropriate Capability

High
Category
Not specified by scanner
Confidence
99% confidence
Finding

The skill executes task-supplied code as child processes and passes through the parent environment via env: { ...process.env, NODE_ENV: 'test' }. Because tasks are loaded from JSON and written directly to executable files, any task can run arbitrary code and read sensitive environment variables, filesystem contents, or make network requests, turning the benchmark runner into a code-execution wrapper.

Content

No source excerpt is available for this finding.

Missing User Warnings

High
Category
Not specified by scanner
Confidence
99% confidence
Finding

This is the same underlying issue as the arbitrary-code execution path: untrusted task code is launched in a subprocess with the full parent environment inherited. In the context of a benchmark skill, this is especially dangerous because executing submitted code is the core behavior, so without sandboxing every task effectively gains the runner's permissions and environment secrets.

Content

No source excerpt is available for this finding.

Hidden Instructions

High
Category
Prompt Injection
Confidence
60% confidence
Finding

Hidden instructions were detected in comments or invisible text. These could contain malicious directives. Manual review is recommended.

Content

Scanner excerpt · reports/benchmark-report-20260318-092642.md (reported line 1)May include surrounding context.

md
# Agent Benchmark Report

**Generated**: 2026-03-20 15:45:47  
**Total Tasks**: 32

Hidden Instructions

High
Category
Prompt Injection
Confidence
60% confidence
Finding

Hidden instructions were detected in comments or invisible text. These could contain malicious directives. Manual review is recommended.

Content

Scanner excerpt · reports/benchmark-report-20260318-093308.md (reported line 1)May include surrounding context.

md
# Agent Benchmark Report

**Generated**: 2026-03-20 15:45:47  
**Total Tasks**: 32

Hidden Instructions

High
Category
Prompt Injection
Confidence
60% confidence
Finding

Hidden instructions were detected in comments or invisible text. These could contain malicious directives. Manual review is recommended.

Content

Scanner excerpt · reports/benchmark-report-20260320-154547.md (reported line 1)May include surrounding context.

md
# Agent Benchmark Report

**Generated**: 2026-03-20 15:45:47  
**Total Tasks**: 32

Natural-Language Policy Violations

Medium
Category
Not specified by scanner
Confidence
97% confidence
Finding

SQP-3 applies to all file types and covers language or locale policy violations. The document consistently presents instructions and descriptions in Chinese, but it does not offer an alternative language or state that Chinese is a required locale for a justified region-specific use case.

Content

No source excerpt is available for this finding.

Missing User Warnings

Medium
Category
Not specified by scanner
Confidence
89% confidence
Finding

The code forcefully removes the computed temp directory with fs.rmSync(..., recursive: true, force: true). Although cleanup is part of the workflow, there is no visible disclosure to users that filesystem contents under that path will be deleted.

Content

No source excerpt is available for this finding.

Context-Inappropriate Capability

Medium
Category
Not specified by scanner
Confidence
92% confidence
Finding

The skill writes benchmark reports to ../../memory/benchmark-results.md, which is outside the task temp directory and into a shared location. Since reports include task output and error text, untrusted executed code can intentionally emit sensitive data that then gets persisted to a broader shared area, increasing data exposure and cross-run leakage risk.

Content

No source excerpt is available for this finding.

Missing User Warnings

Medium
Category
Not specified by scanner
Confidence
92% confidence
Finding

Persisting benchmark reports containing execution output and errors into a shared memory directory creates a confidentiality risk because untrusted code can print secrets or sensitive host data, which the runner then stores for later access. The absence of any warning exacerbates the issue, but the core security problem is persistence of potentially sensitive execution artifacts into a shared location.

Content

No source excerpt is available for this finding.

Intent-Code Divergence

Medium
Category
Not specified by scanner
Confidence
97% confidence
Finding

The report presents '100% success' and production readiness while the documented optimization explicitly weakens validation from exact expected outputs to loose substring and phrase-based matching. This can allow incorrect, partial, or fabricated outputs to be marked as passing, undermining the integrity of the benchmark and potentially masking broken or unsafe agent behavior.

Content

No source excerpt is available for this finding.

Intent-Code Divergence

Medium
Category
Not specified by scanner
Confidence
96% confidence
Finding

The text claims that matching phrases like 'Sum:' or 'Primes:' verifies that a calculation or function worked, but the shown logic does not validate correctness of the numeric result or returned values. This creates a misleading assurance gap where superficial output tokens can satisfy tests even when core functionality is wrong.

Content

No source excerpt is available for this finding.

Ae4

Medium
Category
analysis-evasion
Confidence
80% confidence
Finding

Suspicious Unicode normalization or mixed-script content

Content

No source excerpt is available for this finding.

Ae4

Medium
Category
analysis-evasion
Confidence
80% confidence
Finding

Suspicious Unicode normalization or mixed-script content

Content

No source excerpt is available for this finding.

Ae4

Medium
Category
analysis-evasion
Confidence
80% confidence
Finding

Suspicious Unicode normalization or mixed-script content

Content

No source excerpt is available for this finding.

Ae4

Medium
Category
analysis-evasion
Confidence
80% confidence
Finding

Suspicious Unicode normalization or mixed-script content

Content

No source excerpt is available for this finding.

Natural-Language Policy Violations

Medium
Category
Not specified by scanner
Confidence
98% confidence
Finding

This JSON manifest contains user-facing natural-language fields such as titles and descriptions in Chinese throughout the file. Because the file does not provide an opt-in, alternative locale, or any justification that the skill is region-specific, it appears to impose a specific language on users, which matches the language/locale policy violation criteria.

Content

No source excerpt is available for this finding.

Vague Triggers

Medium
Category
Not specified by scanner
Confidence
88% confidence
Finding

This JSON manifest describes a benchmark skill in very general terms as evaluating AI agent task completion capabilities, but it does not narrow the activation scope, enumerate specific trigger phrases, or provide exclusion conditions. Because SQP-1 applies to manifest files, this kind of broad description can lead to unintended invocation in contexts unrelated to running this benchmark suite.

Content

No source excerpt is available for this finding.

Missing User Warnings

Medium
Category
Not specified by scanner
Confidence
94% confidence
Finding

The task script writes directly to .\test-output.txt without any user-facing indication that executing the task will modify the filesystem. In an agent setting, undisclosed file creation can violate user expectations, overwrite artifacts in the working directory, and normalize side effects from benchmark content.

Content

No source excerpt is available for this finding.

Missing User Warnings

Medium
Category
Not specified by scanner
Confidence
94% confidence
Finding

This task exports CSV data to .\people.csv, creating a persistent file on disk without any warning to the user. In an agent or skill-execution environment, silent file creation can clutter repositories, overwrite existing files, or be chained with other behaviors to stage unintended data outputs.

Content

No source excerpt is available for this finding.

Vague Triggers

Medium
Category
Not specified by scanner
Confidence
84% confidence
Finding

The manifest describes a large set of benchmark tasks but does not define clear trigger constraints, execution boundaries, or exclusion conditions for potentially state-changing actions. In an agent-skill context, broad task framing can cause an agent to execute local filesystem and system inspection actions in unintended environments, increasing the chance of unsafe or overbroad behavior.

Content

No source excerpt is available for this finding.

Missing User Warnings

Medium
Category
Not specified by scanner
Confidence
91% confidence
Finding

The manifest includes tasks that create, overwrite, and delete files but does not warn users or orchestrators about filesystem side effects. In a real agent runtime, this can lead to unintended modification or deletion of local data if the tasks are executed outside a disposable sandbox or if paths are later generalized from the examples.

Content

No source excerpt is available for this finding.

Missing User Warnings

Low
Category
Not specified by scanner
Confidence
88% confidence
Finding

The skill explicitly instructs users to run a PowerShell benchmark runner and describes tasks involving file creation, writing, directory management, and auto-generated reports, but it does not clearly warn that local files and directories will be modified. In an agent context, this can cause unintended file writes or overwrite artifacts if the user or agent runs it in the wrong workspace, even though the documented purpose appears to be legitimate benchmarking rather than abuse.

Content

No source excerpt is available for this finding.

Natural-Language Policy Violations

Low
Category
Not specified by scanner
Confidence
94% confidence
Finding

This markdown report is entirely presented in Chinese, including headings and summary text, with no indication that the user opted into that language or that the report is region-specific. The policy for this audit flags language or locale constraints when they are imposed without user choice or clear justification.

Content

No source excerpt is available for this finding.

Natural-Language Policy Violations

Low
Category
Not specified by scanner
Confidence
90% confidence
Finding

The file's descriptive comments and user-facing console/report strings are written exclusively in Chinese, indicating a fixed locale choice. There is no opt-in, configuration, or documentation justifying that the skill should always operate in Chinese.

Content

No source excerpt is available for this finding.

Natural-Language Policy Violations

Low
Category
Not specified by scanner
Confidence
89% confidence
Finding

The manifest description is written entirely in Chinese, which imposes a specific language choice in user-facing metadata. There is no indication that the skill is region-specific or that users can choose an alternate language, so this appears to violate the language/locale policy criterion.

Content

No source excerpt is available for this finding.