T08 · Insecure Dependencies
Error
- Location
- evals/run_evals.py:15
- Finding
- Execution of an Unverified External Evaluation Framework## Vulnerability Details **File Location**: `evals/run_evals.py`, lines 15–26 **Vulnerability Type**: Untrusted module search-path modification and external code execution **Risk Level**: High ### Vulnerable Code ```python REPO_ROOT = Path(__file__).resolve().parent.parent.parent.parent sys.path.insert(0, str(REPO_ROOT / "shared")) from eval_framework import main if __name__ == "__main__": # Inject capability path if not provided if "--capability" not in sys.argv: capability_dir = Path(__file__).resolve().parent.parent sys.argv.extend(["--capability", str(capability_dir)]) main() ``` ### Technical Analysis The evaluator prepends a filesystem directory outside the audited project to `sys.path` and then imports `eval_framework` from that location. Given the supplied artifact path, traversing four parent directories from `evals/run_evals.py` resolves `REPO_ROOT` to the filesystem root, causing the runner to search `/shared` first. The implementation of `eval_framework` is not included in the audited project. There is therefore no integrity guarantee, version pinning, or source verification for the code imported and executed by the runner. Python executes module-level code immediately during import, before `main()` is called. This is an unsafe dependency boundary rather than confirmed malicious behavior in the supplied files. Exploitation requires an attacker to create or modify the resolved external module, such as `/shared/eval_framework.py`, or otherwise control the resolved dependency location. ### Attack Path 1. An attacker obtains write access to `/shared`, or compromises the external shared framework installation. 2. The attacker creates or modifies `/shared/eval_framework.py`. 3. A user launches `evals/run_evals.py`. 4. The runner inserts `/shared` at the beginning of `sys.path`. 5. Python imports the attacker-controlled module and executes its module-level code. 6. Th ...[truncated 808 chars]
- Remediation
- ## Remediation Suggestions 1. Remove runtime `sys.path` modification and import the framework as a normal, packaged dependency. 2. Pin the dependency to an exact version and verify it with a lockfile and cryptographic hashes. 3. If the framework must remain local, resolve it relative to a validated repository root and reject paths that escape that root. 4. Verify the resolved module path before import: - Require it to be inside an approved directory. - Reject symbolic-link escapes. - Enforce appropriate ownership and write permissions. 5. Run evaluation code in a restricted environment with minimal filesystem, network, and secret access. 6. Add a startup check that reports the exact framework path and fails closed if the trusted framework is absent. 7. Do not pass secrets to the framework unless they are strictly required, and scope any API key to the minimum necessary permissions and budget.