Back to skill

Security audit

Agent Benchmark

Security checks for vulnerabilities and agentic risk

Overview

This benchmark skill has a coherent purpose, but it can run custom local code with inherited secrets and has unsafe file and persistence behavior that users should review before installing.

Only run this skill in an isolated sandbox with trusted task files, a scrubbed environment, and no valuable workspace or credentials available. It should be fixed to validate task IDs, avoid writing to agent memory, disclose executable custom tasks, and run code in a real container or restricted account before normal use.

Vulnerability Patterns
  • Agent Memory PoisoningWrites attacker-controlled rules into memory that affect later sessions
  • Unauthorized Access and Privilege EscalationObtains permissions beyond the task's legitimate needs
  • Insecure Skill Coding PracticesFinds exploitable flaws such as hardcoded secrets or command injection
  • Skill Instruction HijackingAlters the agent's session goals or safety constraints when the skill loads
  • Remote Payload Retrieval and ExecutionFetches external code whose behavior can change after review
Findings (4)

T09 · Insecure Skill Coding Practices

Error
Location
index.js:73
Finding

Arbitrary Task Code Executes Without Security Isolation

Content
View full analysis
{ const child = spawn(language.command.split(' ')[0], [...language.command.split(' ').slice(1), fullFile], { timeout: CONFIG.timeout, cwd: tempDir, env: { ...process.env, NODE_ENV: 'test' } } ); ``` ### Technical Analysis The benchmark accepts source code from `task.input.code` and `task.input.test_code`, writes that content to a file, and directly invokes the corresponding Python, Node.js, or Go runtime. The child process is not placed in a container, operating-system sandbox, restricted user account, filesystem namespace, or network namespace. It consequently receives the same effective operating-system permissions as the benchmark process. The configured timeout is not a security boundary. It only attempts to terminate the immediate child process after 30 seconds. Task code may access files before termination, launch detached descendants, communicate over the network, or exploit local runtime and operating-system capabilities. In addition, `CONFIG.memoryLimit` is declared but never applied to the spawned process. Although the bundled `tasks.json` code is benign, the CLI explicitly supports selecting another task file. Such a file must therefore be treated as execut ...[truncated 1405 chars]
Remediation
View remediation

T05 · Unauthorized Access and Privilege Escalation

Error
Location
index.js:89
Finding

Inherited Environment Variables Are Exposed to Untrusted Task Processes

Content
View full analysis
Remediation
View remediation

T09 · Insecure Skill Coding Practices

Error
Location
index.js:148
Finding

Task Identifier Path Traversal Can Trigger Arbitrary Recursive Deletion

Content
View full analysis
Remediation
View remediation

T02 · Agent Memory Poisoning

Error
Location
index.js:284
Finding

Untrusted Task Output Is Persisted Into Agent Memory

Content
View full analysis
`| ${r.taskId} | ${getStatusIcon(r.status)} | ${r.executionTime}ms | ${r.testsPassed}/${r.testsTotal} | ${r.error ? truncate(r.error, 30) : '-'} |`).join('\n')} ``` ```js ${results.map(r => `### ${r.taskId} - **状态**: ${r.status} - **执行时间**: ${r.executionTime}ms - **退出码**: ${r.exitCode} - **输出**: \`${truncate(r.output, 100)}\` - **错误**: \`${truncate(r.error, 100)}\` ``` The resulting report is then written outside the project into a directory explicitly designated as memory: ```js // 保存到 memory 目录 const memoryReport = path.join(__dirname, '../../memory/benchmark-results.md'); generateReport(results, memoryReport); ``` ### Technical Analysis A custom task controls its identifier, stdout, and stderr. These values are interpolated directly into a Markdown report. The `truncate()` function limits length but does not escape backticks, pipes, line breaks, headings, HTML, or instruction-like content. A malicious task can therefore alter the report's structure or insert content that appears to be trusted instructions. The report is then persisted to `../../memory/benchmark-results.md`, outside the Skill package. If the surrounding Agent platform ingests Markdown files from that memory directory in later sessions, attacker-controlled task output can become persistent context. This creates a memory-poisoning path in which data emitted by executable benchmark tasks may later be interpreted as instructions or trusted state. ### Attack Path 1. An attacker creates a custom benchmark task with a crafted ID or code that emits malicious Markdown and instruction-like text through stdout or stderr. 2. The benchmark e ...[truncated 1099 chars]
Remediation
View remediation
Vulnerability Patterns
  • Prompt InjectionInstruction Override, Hidden Instructions, Exfiltration Commands
  • Data ExfiltrationExternal Transmission, Env Variable Harvesting, File System Enumeration
  • Excessive AgencyUnrestricted Tool Access, Autonomous Decision Making, Scope Creep
  • Trigger AbuseOverly Broad Trigger, Shadow Command Trigger, Keyword Baiting Trigger
  • MCP Tool PoisoningHidden Instructions, Unicode Deception, Parameter Description Injection
Findings (22)

Context-Inappropriate Capability

High
Category
Not specified by scanner
Confidence
99% confidence
Finding

The benchmark writes task-supplied code to disk and executes it as a subprocess while passing through nearly the entire parent environment via env: { ...process.env, NODE_ENV: 'test' }. This allows untrusted code to read secrets, make network requests, access the filesystem, and abuse any privileges available to the running agent, which is a real sandbox escape / arbitrary code execution risk in an evaluation framework.

Content

No source excerpt is available for this finding.

Missing User Warnings

High
Category
Not specified by scanner
Confidence
95% confidence
Finding

The tool executes untrusted task code with inherited environment variables but does not clearly warn users that arbitrary code will run with the agent's privileges and environment. In a benchmark context this omission is dangerous because operators may assume evaluation is passive, while task code can actively exfiltrate credentials or modify local state.

Content

No source excerpt is available for this finding.

Hidden Instructions

High
Category
Prompt Injection
Confidence
60% confidence
Finding

Hidden instructions were detected in comments or invisible text. These could contain malicious directives. Manual review is recommended.

Content

Scanner excerpt · reports/benchmark-report-20260318-092642.md (reported line 1)May include surrounding context.

md
# Agent Benchmark Report

**Generated**: 2026-03-20 15:45:47  
**Total Tasks**: 32

Hidden Instructions

High
Category
Prompt Injection
Confidence
60% confidence
Finding

Hidden instructions were detected in comments or invisible text. These could contain malicious directives. Manual review is recommended.

Content

Scanner excerpt · reports/benchmark-report-20260318-093308.md (reported line 1)May include surrounding context.

md
# Agent Benchmark Report

**Generated**: 2026-03-20 15:45:47  
**Total Tasks**: 32

Hidden Instructions

High
Category
Prompt Injection
Confidence
60% confidence
Finding

Hidden instructions were detected in comments or invisible text. These could contain malicious directives. Manual review is recommended.

Content

Scanner excerpt · reports/benchmark-report-20260320-154547.md (reported line 1)May include surrounding context.

md
# Agent Benchmark Report

**Generated**: 2026-03-20 15:45:47  
**Total Tasks**: 32

Missing User Warnings

Medium
Category
Not specified by scanner
Confidence
87% confidence
Finding

The documentation explicitly instructs users to run a PowerShell benchmark runner and to supply custom task files, while elsewhere showing that tasks can contain executable PowerShell in a "script" field. Without a clear warning, trust boundary, or restriction on untrusted task content, users may execute task-provided code that writes files or performs arbitrary local actions, making this a real unsafe-execution/documentation issue.

Content

No source excerpt is available for this finding.

Intent-Code Divergence

Medium
Category
Not specified by scanner
Confidence
97% confidence
Finding

The configuration advertises a 512MB memory limit, but no such limit is enforced when spawning child processes. Untrusted code can therefore consume excessive memory and crash or degrade the host process or surrounding agent environment, creating a denial-of-service condition.

Content

No source excerpt is available for this finding.

Missing User Warnings

Medium
Category
Not specified by scanner
Confidence
87% confidence
Finding

The skill writes provided code and test content into a temp directory and later force-deletes that directory with fs.rmSync(..., { recursive: true, force: true }). While these actions are part of the implementation, the file does not include user-facing disclosure that it will create and remove files on disk.

Content

No source excerpt is available for this finding.

Intent-Code Divergence

Medium
Category
Not specified by scanner
Confidence
94% confidence
Finding

The report presents a 100% pass rate and implies reliable validation even though the documented changes intentionally relax verification enough to allow false positives. This can mislead operators into trusting benchmark results that no longer prove correctness, creating a software integrity risk if weakly validated agents are promoted or deployed.

Content

No source excerpt is available for this finding.

Intent-Code Divergence

Medium
Category
Not specified by scanner
Confidence
97% confidence
Finding

The examples claim functionality is 'verified' when the checks only look for minimal phrases like 'Sum:' or 'Primes:', which do not confirm that the underlying computation or extraction was correct. An implementation could emit the expected label while producing wrong, incomplete, or fabricated results and still pass, undermining benchmark integrity.

Content

No source excerpt is available for this finding.

Intent-Code Divergence

Medium
Category
Not specified by scanner
Confidence
95% confidence
Finding

The final assessment states that all capability dimensions were validated and recommends production deployment despite earlier acknowledging that the relaxed verification can miss subtle bugs, regressions, and false positives. This overstatement can drive unsafe operational decisions by overstating assurance and masking limitations in the test methodology.

Content

No source excerpt is available for this finding.

Ae4

Medium
Category
analysis-evasion
Confidence
80% confidence
Finding

Suspicious Unicode normalization or mixed-script content

Content

No source excerpt is available for this finding.

Ae4

Medium
Category
analysis-evasion
Confidence
80% confidence
Finding

Suspicious Unicode normalization or mixed-script content

Content

No source excerpt is available for this finding.

Ae4

Medium
Category
analysis-evasion
Confidence
80% confidence
Finding

Suspicious Unicode normalization or mixed-script content

Content

No source excerpt is available for this finding.

Ae4

Medium
Category
analysis-evasion
Confidence
80% confidence
Finding

Suspicious Unicode normalization or mixed-script content

Content

No source excerpt is available for this finding.

Natural-Language Policy Violations

Medium
Category
Not specified by scanner
Confidence
89% confidence
Finding

This JSON manifest contains multiple natural-language fields such as title and description written only in Chinese, while providing no indication that the skill is region-specific or that users can opt into another language. Under the policy, forcing a specific language without user choice is a natural-language policy violation.

Content

No source excerpt is available for this finding.

Missing User Warnings

Medium
Category
Not specified by scanner
Confidence
92% confidence
Finding

Task task-011 reads and displays environment variables including COMPUTERNAME and USERNAME, which can disclose host and user identity information to the agent output stream. In an agent benchmark context, exposing runtime environment details without disclosure or minimization can leak sensitive metadata to logs, downstream evaluators, or untrusted task consumers.

Content

No source excerpt is available for this finding.

Vague Triggers

Medium
Category
Not specified by scanner
Confidence
83% confidence
Finding

This JSON manifest describes the skill broadly as '30+ benchmark tasks for evaluating AI agent capabilities' without any explicit invocation boundaries, exclusion conditions, or narrow trigger phrases. For manifest files, such generic wording can make it unclear when the skill should activate versus when similar everyday requests about tasks, files, or evaluation should not invoke it.

Content

No source excerpt is available for this finding.

Natural-Language Policy Violations

Low
Category
Not specified by scanner
Confidence
92% confidence
Finding

Natural-language policy violations include forcing a specific language without user opt-in. This skill file presents its operational documentation entirely in Chinese and does not provide an alternative language option or note that the skill is intended only for a Chinese-speaking audience.

Content

No source excerpt is available for this finding.

Natural-Language Policy Violations

Low
Category
Not specified by scanner
Confidence
95% confidence
Finding

The markdown report is written entirely in Chinese, including headings, labels, and summary text, with no indication that language selection was user-configurable or that a Chinese-only locale is required. This can violate a language/locale policy when users are not given an opt-in or alternative language.

Content

No source excerpt is available for this finding.

Natural-Language Policy Violations

Low
Category
Not specified by scanner
Confidence
81% confidence
Finding

Comments, status messages, and the generated report content are written in Chinese with no option for users to choose another language. This can violate language/locale policy when a skill forces a specific language absent explicit opt-in or documented justification.

Content

No source excerpt is available for this finding.

Natural-Language Policy Violations

Low
Category
Not specified by scanner
Confidence
85% confidence
Finding

The manifest description is written entirely in Chinese, which signals a fixed language choice in user-facing metadata. Under the policy, locale or language constraints should either offer user choice or be clearly justified as region-specific; neither is present here.

Content

No source excerpt is available for this finding.

Static analysis

Detected: suspicious.dangerous_exec

Shell command execution detected (child_process).

Critical
Code
suspicious.dangerous_exec
Location
index.js:89