Eval Skills

• Outcome: Review
• Latest audit: Sep 11, 2026 · 2:33 AM UTC
• Version: 0.1.1

Overview

This skill is a coherent evaluation toolkit, but it can run evaluated skills and custom scorers with more host access than users may expect from its sandbox language.

Install only if you are comfortable treating evaluated local skills, MCP stdio servers, and custom scorer files as trusted code unless you force strong container isolation. Prefer Docker mode on a hardened runner without sensitive credentials, avoid untrusted benchmarks with customScorerPath, and do not run this against third-party skills on a workstation or CI account with broad secrets.

A.I.G

Vulnerability Patterns

• Unauthorized Access and Privilege Escalation: Obtains permissions beyond the task's legitimate needs
• Insecure Skill Coding Practices: Finds exploitable flaws such as hardcoded secrets or command injection
• Skill Instruction Hijacking: Alters the agent's session goals or safety constraints when the skill loads
• Agent Memory Poisoning: Writes attacker-controlled rules into memory that affect later sessions
• Remote Payload Retrieval and Execution: Fetches external code whose behavior can change after review

Findings (2)

T09 · Insecure Skill Coding Practices

Error

• Location: packages/core/src/evaluator/scorers/CustomScorer.ts:23

Finding

Arbitrary Host Code Execution Through Unisolated Custom Scorers

Content

View full analysis

((resolve) => { const worker = new Worker(workerCode, { eval: true, workerData: { scorerPath, output, expected }, resourceLimits: { maxOldGenerationSizeMb: 128 }, }); ``` ### Technical Analysis The custom scorer path originates from `expected.customScorerPath`, is converted into an absolute path with `path.resolve()`, and is then dynamically imported. The implementation does not canonicalize the resulting path against an approved scorer directory, require an explicit trust decision, or execute the module inside the project’s Docker sandbox. A Node.js worker thread is a concurrency mechanism, not a security boundary. Code loaded by the worker retains access to Node.js capabilities such as: - Reading and modifying files accessible to the current user - Reading process environment variables - Creating network connections - Starting child processes - Accessing other resources available ...[truncated 1701 chars]

Remediation

View remediation

T05 · Unauthorized Access and Privilege Escalation

Error

• Location: packages/core/src/adapters/SubprocessAdapter.ts:62

Finding

Untrusted Subprocess Skills Execute on the Host with Ineffective Network and Filesystem Policies

Content

View full analysis

Remediation

View remediation

SkillSpector

Vulnerability Patterns

• Privilege Escalation: Excessive Permissions, Sudo/Root Execution, Credential Access
• Excessive Agency: Unrestricted Tool Access, Autonomous Decision Making, Scope Creep
• Tool Misuse: Tool Parameter Abuse, Chaining Abuse, Unsafe Defaults
• Behavioral AST: exec() Call, eval() Call, Dynamic Import
• MCP Tool Poisoning: Hidden Instructions, Unicode Deception, Parameter Description Injection

Findings (181)

Tp4

High

• Category: MCP Tool Poisoning
• Confidence: 99% confidence

Finding

The declared description describes a unit testing and evaluation framework for AI skills, but the supplied code is a simple calculator-style skill. Its primary behavior is handling JSON-RPC requests to evaluate math expressions and perform basic string operations. It does not implement any of the described framework capabilities such as skill discovery, benchmarking, CI/CD quality gates, or report generation. This is a clear material mismatch in primary purpose and exposed capabilities.

Content

No source excerpt is available for this finding.

Tp4

High

• Category: MCP Tool Poisoning
• Confidence: 99% confidence

Finding

The declared description says this skill is a unit testing framework for AI agent skills, with functionality around discovery, scaffolding, selection, evaluation, reporting, and CI/CD quality gates. The supplied code does none of that. Instead, it implements a standalone mock web search skill that reads a JSON-RPC request from stdin, handles 'invoke' and 'healthcheck', and returns canned answers from a hardcoded knowledge base with basic partial-word matching. This is a materially different primary purpose, not just an implementation detail, so the description does not accurately represent the code.

Content

No source excerpt is available for this finding.

Tp4

High

• Category: MCP Tool Poisoning
• Confidence: 96% confidence

Finding

The declared description emphasizes a unit testing/evaluation framework for assessing skill quality, benchmarking, CI/CD quality gates, and reporting. This code chunk instead implements a creation/scaffolding command that generates new skill skeletons from templates or OpenAPI specs and writes them to disk. While scaffolding is mentioned in the broad description, the specific declared purpose centers on evaluation/testing behavior, whereas this code’s primary function is code generation. That is a materially different capability from unit testing/evaluation, so this chunk does not accurately represent the declared description on its own.

Content

No source excerpt is available for this finding.

eval() call detected

High

• Category: Dangerous Code Execution
• Confidence: 94% confidence

Finding

The skill uses Python eval() on user-controlled input. Although it restricts characters to digits, operators, periods, parentheses, and spaces, using eval for parsing remains unsafe-by-design and can still expose the process to denial-of-service through expensive expressions or unexpectedly permissive evaluation semantics; it also creates future risk if the allowlist is later expanded incorrectly.

Content

Scanner excerpt · examples/skills/calculator/skill.py (reported line 31)May include surrounding context.

python

# Safe subset of operations
        allowed = set("0123456789+-*/.() ")
        if all(c in allowed for c in expression):
            result = eval(expression)
            return {"result": str(result)}
        else:
            return {"result": "error: invalid characters in expression"}

Chaining Abuse

High

• Category: Tool Misuse
• Confidence: 75% confidence

Finding

Tool calls are chained to bypass individual safety checks or escalate capabilities beyond what any single tool call would allow.

Content

Scanner excerpt · package.json (reported line 34)May include surrounding context.

json

"lint:fix": "eslint packages/*/src --ext .ts --fix",
    "format": "prettier --write \"packages/*/src/**/*.ts\"",
    "typecheck": "tsc --noEmit -p tsconfig.base.json",
    "clean": "pnpm -r run clean && rm -rf node_modules"
  },
  "devDependencies": {
    "@typescript-eslint/eslint-plugin": "^7.0.0",

Context-Inappropriate Capability

High

• Category: Not specified by scanner
• Confidence: 97% confidence

Finding

The adapter treats any non-HTTP entrypoint as a local command and passes it to StdioClientTransport, which can spawn an arbitrary executable with attacker-controlled arguments. If untrusted skill definitions can supply entrypoints, this becomes direct local code execution in the host environment, which is especially dangerous in an evaluation framework that may process third-party skills.

Content

No source excerpt is available for this finding.

Intent-Code Divergence

High

• Category: Not specified by scanner
• Confidence: 98% confidence

Finding

The file advertises strong syscall filtering, but if the seccomp profile is missing it silently degrades to "unconfined", removing a major containment boundary while continuing execution. In a skill-evaluation sandbox, this materially increases the chance that untrusted code can access dangerous syscalls and exploit kernel/container-escape paths.

Content

No source excerpt is available for this finding.

Docker Socket Access

High

• Category: Privilege Escalation
• Confidence: 96% confidence

Finding

Connecting to /var/run/docker.sock gives this process powerful control over the host Docker daemon, which is effectively host-level privilege in many deployments. If an attacker can influence this component or its configuration, they may create privileged containers, mount the host filesystem, or otherwise escape intended sandbox boundaries completely.

Content

Scanner excerpt · packages/core/src/sandbox/DockerSandbox.ts (reported line 82)May include surrounding context.

ts

constructor(config: SandboxConfig) {
    super(config);
    this.docker = new Docker({
      socketPath: config.docker.socketPath ?? "/var/run/docker.sock",
    });
  }

Tool Parameter Abuse

High

• Category: Tool Misuse
• Confidence: 90% confidence

Finding

Tool parameters are crafted to achieve unintended or unsafe behavior. Parameter abuse can bypass intended safety checks (e.g. shell=True, --force, dangerous glob patterns).

Content

Scanner excerpt · packages/core/src/sandbox/SandboxExecutor.test.ts (reported line 46)May include surrounding context.

ts

it("should reject command with shell injection characters", async () => {
    const sandbox = SandboxFactory.createProcessSandbox(TEST_CONFIG);
    const result = await sandbox.execute(
      "python3; rm -rf /",
      "/tmp",
      { jsonrpc: "2.0", method: "invoke", params: {}, id: 1 },
    );

Tool Parameter Abuse

High

• Category: Tool Misuse
• Confidence: 90% confidence

Finding

Tool parameters are crafted to achieve unintended or unsafe behavior. Parameter abuse can bypass intended safety checks (e.g. shell=True, --force, dangerous glob patterns).

Content

Scanner excerpt · packages/core/src/sandbox/sandbox.integration.test.ts (reported line 327)May include surrounding context.

ts

it("should reject command with shell injection characters", async () => {
    const sandbox = SandboxFactory.createProcessSandbox(TEST_CONFIG);
    const result = await sandbox.execute(
      "python3; rm -rf /",
      "/tmp",
      { jsonrpc: "2.0", method: "invoke", params: {}, id: 1 },
    );

Tool Parameter Abuse

High

• Category: Tool Misuse
• Confidence: 85% confidence

Finding

Tool parameters are crafted to achieve unintended or unsafe behavior. Parameter abuse can bypass intended safety checks (e.g. shell=True, --force, dangerous glob patterns).

Content

Scanner excerpt · packages/core/src/sandbox/SandboxExecutor.test.ts (reported line 46)May include surrounding context.

ts

it("should reject command with shell injection characters", async () => {
    const sandbox = SandboxFactory.createProcessSandbox(TEST_CONFIG);
    const result = await sandbox.execute(
      "python3; rm -rf /",
      "/tmp",
      { jsonrpc: "2.0", method: "invoke", params: {}, id: 1 },
    );

Tool Parameter Abuse

High

• Category: Tool Misuse
• Confidence: 90% confidence

Finding

Tool parameters are crafted to achieve unintended or unsafe behavior. Parameter abuse can bypass intended safety checks (e.g. shell=True, --force, dangerous glob patterns).

Content

Scanner excerpt · packages/core/src/sandbox/SandboxExecutor.test.ts (reported line 46)May include surrounding context.

ts

it("should reject command with shell injection characters", async () => {
    const sandbox = SandboxFactory.createProcessSandbox(TEST_CONFIG);
    const result = await sandbox.execute(
      "python3; rm -rf /",
      "/tmp",
      { jsonrpc: "2.0", method: "invoke", params: {}, id: 1 },
    );

Chaining Abuse

High

• Category: Tool Misuse
• Confidence: 75% confidence

Finding

Tool calls are chained to bypass individual safety checks or escalate capabilities beyond what any single tool call would allow.

Content

Scanner excerpt · packages/core/src/sandbox/SandboxExecutor.test.ts (reported line 46)May include surrounding context.

ts

it("should reject command with shell injection characters", async () => {
    const sandbox = SandboxFactory.createProcessSandbox(TEST_CONFIG);
    const result = await sandbox.execute(
      "python3; rm -rf /",
      "/tmp",
      { jsonrpc: "2.0", method: "invoke", params: {}, id: 1 },
    );

Chaining Abuse

High

• Category: Tool Misuse
• Confidence: 75% confidence

Finding

Tool calls are chained to bypass individual safety checks or escalate capabilities beyond what any single tool call would allow.

Content

Scanner excerpt · packages/core/src/sandbox/sandbox.integration.test.ts (reported line 327)May include surrounding context.

ts

it("should reject command with shell injection characters", async () => {
    const sandbox = SandboxFactory.createProcessSandbox(TEST_CONFIG);
    const result = await sandbox.execute(
      "python3; rm -rf /",
      "/tmp",
      { jsonrpc: "2.0", method: "invoke", params: {}, id: 1 },
    );

Credential Access

High

• Category: Privilege Escalation
• Confidence: 60% confidence

Finding

Code accesses credential files (SSH keys, AWS credentials, etc.). This could indicate credential theft attempts.

Content

Scanner excerpt · packages/core/src/sandbox/SandboxExecutor.test.ts (reported line 60)May include surrounding context.

ts

it("should reject command with path traversal", async () => {
    const sandbox = SandboxFactory.createProcessSandbox(TEST_CONFIG);
    const result = await sandbox.execute(
      "python3 ../../etc/passwd",
      "/tmp/skill",
      { jsonrpc: "2.0", method: "invoke", params: {}, id: 1 },
    );

Credential Access

High

• Category: Privilege Escalation
• Confidence: 60% confidence

Finding

Code accesses credential files (SSH keys, AWS credentials, etc.). This could indicate credential theft attempts.

Content

Scanner excerpt · packages/core/src/sandbox/SandboxExecutor.test.ts (reported line 89)May include surrounding context.

ts

it("should reject command with path traversal", async () => {
    const sandbox = SandboxFactory.createProcessSandbox(TEST_CONFIG);
    const result = await sandbox.execute(
      "python3 ../../etc/passwd",
      "/tmp/skill",
      { jsonrpc: "2.0", method: "invoke", params: {}, id: 1 },
    );

Credential Access

High

• Category: Privilege Escalation
• Confidence: 60% confidence

Finding

Code accesses credential files (SSH keys, AWS credentials, etc.). This could indicate credential theft attempts.

Content

Scanner excerpt · packages/core/src/sandbox/sandbox.integration.test.ts (reported line 168)May include surrounding context.

ts

it("should reject command with path traversal", async () => {
    const sandbox = SandboxFactory.createProcessSandbox(TEST_CONFIG);
    const result = await sandbox.execute(
      "python3 ../../etc/passwd",
      "/tmp/skill",
      { jsonrpc: "2.0", method: "invoke", params: {}, id: 1 },
    );

Credential Access

High

• Category: Privilege Escalation
• Confidence: 60% confidence

Finding

Code accesses credential files (SSH keys, AWS credentials, etc.). This could indicate credential theft attempts.

Content

Scanner excerpt · packages/core/src/sandbox/sandbox.integration.test.ts (reported line 329)May include surrounding context.

ts

it("should reject command with path traversal", async () => {
    const sandbox = SandboxFactory.createProcessSandbox(TEST_CONFIG);
    const result = await sandbox.execute(
      "python3 ../../etc/passwd",
      "/tmp/skill",
      { jsonrpc: "2.0", method: "invoke", params: {}, id: 1 },
    );

Credential Access

High

• Category: Privilege Escalation
• Confidence: 60% confidence

Finding

Code accesses credential files (SSH keys, AWS credentials, etc.). This could indicate credential theft attempts.

Content

Scanner excerpt · packages/core/src/sandbox/sandbox.integration.test.ts (reported line 331)May include surrounding context.

ts

it("should reject command with path traversal", async () => {
    const sandbox = SandboxFactory.createProcessSandbox(TEST_CONFIG);
    const result = await sandbox.execute(
      "python3 ../../etc/passwd",
      "/tmp/skill",
      { jsonrpc: "2.0", method: "invoke", params: {}, id: 1 },
    );

Credential Access

High

• Category: Privilege Escalation
• Confidence: 90% confidence

Finding

Code accesses credential files (SSH keys, AWS credentials, etc.). This could indicate credential theft attempts.

Content

Scanner excerpt · packages/core/src/sandbox/sandbox.integration.test.ts (reported line 168)May include surrounding context.

ts

req = json.loads(sys.stdin.read())
# 尝试读取系统敏感文件
results = {}
sensitive_files = ["/etc/passwd", "/etc/shadow", "/root/.ssh/id_rsa", "/proc/1/environ"]
for f in sensitive_files:
    try:
        with open(f, "r") as fp:

Credential Access

High

• Category: Privilege Escalation
• Confidence: 95% confidence

Finding

Code accesses credential files (SSH keys, AWS credentials, etc.). This could indicate credential theft attempts.

Content

Scanner excerpt · packages/core/src/sandbox/sandbox.integration.test.ts (reported line 168)May include surrounding context.

ts

req = json.loads(sys.stdin.read())
# 尝试读取系统敏感文件
results = {}
sensitive_files = ["/etc/passwd", "/etc/shadow", "/root/.ssh/id_rsa", "/proc/1/environ"]
for f in sensitive_files:
    try:
        with open(f, "r") as fp:

Credential Access

High

• Category: Privilege Escalation
• Confidence: 95% confidence

Finding

Code accesses credential files (SSH keys, AWS credentials, etc.). This could indicate credential theft attempts.

Content

Scanner excerpt · packages/core/src/sandbox/sandbox.integration.test.ts (reported line 330)May include surrounding context.

ts

req = json.loads(sys.stdin.read())
# 尝试读取系统敏感文件
results = {}
sensitive_files = ["/etc/passwd", "/etc/shadow", "/root/.ssh/id_rsa", "/proc/1/environ"]
for f in sensitive_files:
    try:
        with open(f, "r") as fp:

Credential Access

High

• Category: Privilege Escalation
• Confidence: 95% confidence

Finding

Code accesses credential files (SSH keys, AWS credentials, etc.). This could indicate credential theft attempts.

Content

Scanner excerpt · packages/core/src/sandbox/sandbox.integration.test.ts (reported line 567)May include surrounding context.

ts

req = json.loads(sys.stdin.read())
# 尝试读取系统敏感文件
results = {}
sensitive_files = ["/etc/passwd", "/etc/shadow", "/root/.ssh/id_rsa", "/proc/1/environ"]
for f in sensitive_files:
    try:
        with open(f, "r") as fp:

Credential Access

High

• Category: Privilege Escalation
• Confidence: 95% confidence

Finding

Code accesses credential files (SSH keys, AWS credentials, etc.). This could indicate credential theft attempts.

Content

Scanner excerpt · packages/core/src/sandbox/sandbox.integration.test.ts (reported line 575)May include surrounding context.

ts

req = json.loads(sys.stdin.read())
# 尝试读取系统敏感文件
results = {}
sensitive_files = ["/etc/passwd", "/etc/shadow", "/root/.ssh/id_rsa", "/proc/1/environ"]
for f in sensitive_files:
    try:
        with open(f, "r") as fp:

Credential Access

High

• Category: Privilege Escalation
• Confidence: 95% confidence

Finding

Code accesses credential files (SSH keys, AWS credentials, etc.). This could indicate credential theft attempts.

Content

Scanner excerpt · packages/core/src/sandbox/sandbox.integration.test.ts (reported line 576)May include surrounding context.

ts

req = json.loads(sys.stdin.read())
# 尝试读取系统敏感文件
results = {}
sensitive_files = ["/etc/passwd", "/etc/shadow", "/root/.ssh/id_rsa", "/proc/1/environ"]
for f in sensitive_files:
    try:
        with open(f, "r") as fp:

Static analysis

Detected: suspicious.dangerous_exec, suspicious.dynamic_code_execution, suspicious.env_credential_access

Shell command execution detected (child_process).

Critical

• Code: suspicious.dangerous_exec
• Location: packages/cli/src/commands/__tests__/eval.test.ts:44

Shell command execution detected (child_process).

Critical

• Code: suspicious.dangerous_exec
• Location: packages/core/src/sandbox/ProcessSandbox.ts:85

Dynamic code execution detected.

Critical

• Code: suspicious.dynamic_code_execution
• Location: examples/skills/calculator/skill.py:31

Dynamic code execution detected.

Critical

• Code: suspicious.dynamic_code_execution
• Location: packages/core/src/sandbox/sandbox.integration.test.ts:333

Environment variable access combined with network send.

Critical

• Code: suspicious.env_credential_access
• Location: packages/core/src/adapters/HttpAdapter.ts:82