T09 · Insecure Skill Coding Practices
Error
- Location
- packages/core/src/evaluator/scorers/CustomScorer.ts:23
- Finding
- Arbitrary Host Code Execution Through Unisolated Custom Scorers<![CDATA[ ## Vulnerability Details **File Location**: `packages/core/src/evaluator/scorers/CustomScorer.ts:23-53` **Vulnerability Type**: Unrestricted dynamic module loading and execution **Risk Level**: High ### Vulnerable Code ```ts const scorerPath = path.resolve(expected.customScorerPath); const workerCode = ` const { parentPort, workerData } = require('node:worker_threads'); const { pathToFileURL } = require('node:url'); async function run() { try { const { scorerPath, output, expected } = workerData; const modulePath = pathToFileURL(scorerPath).href; const scorerModule = await import(modulePath); const scorerFn = scorerModule.default; if (typeof scorerFn !== 'function') { throw new Error('Default export is not a function'); } const result = await scorerFn(output, expected); parentPort.postMessage({ success: true, result }); } catch (error) { parentPort.postMessage({ success: false, error: error.message }); } } run(); `; return new Promise<ScorerResult>((resolve) => { const worker = new Worker(workerCode, { eval: true, workerData: { scorerPath, output, expected }, resourceLimits: { maxOldGenerationSizeMb: 128 }, }); ``` ### Technical Analysis The custom scorer path originates from `expected.customScorerPath`, is converted into an absolute path with `path.resolve()`, and is then dynamically imported. The implementation does not canonicalize the resulting path against an approved scorer directory, require an explicit trust decision, or execute the module inside the project’s Docker sandbox. A Node.js worker thread is a concurrency mechanism, not a security boundary. Code loaded by the worker retains access to Node.js capabilities such as: - Reading and modifying files accessible to the current user - Reading process environment variables - Creating network connections - Starting child processes - Accessing other resources available ...[truncated 1701 chars]
- Remediation
- <![CDATA[ ## Remediation Suggestions 1. Disable custom scorers by default and require an explicit command-line or configuration opt-in that warns users that scorer code is trusted executable code. 2. Resolve the scorer path using `fs.realpath()` and verify that it remains inside a dedicated, administrator-approved scorer directory. Reject absolute paths, traversal, symlink escapes, and paths outside that root. 3. Execute custom scorers in the Docker sandbox rather than a worker thread. 4. Configure the scorer container with: - No network access - A read-only root filesystem - A narrowly scoped read-only mount for the scorer - No host environment credentials - A non-root user - Dropped Linux capabilities - Resource and execution-time limits - The restrictive seccomp profile 5. Pass only the output and minimum expected scoring fields into the isolated process. 6. Validate the scorer result against a strict schema before accepting it. 7. Document that custom scorer files are executable code and must not be loaded from untrusted benchmarks. ]]>