T09 · Insecure Skill Coding Practices
- Location
ARCHITECTURE.md:426- Finding
Execution of AI-Generated Backtesting Code Without Verifiable Isolation
- Content
View full analysis
Vulnerability Details
File Location:
ARCHITECTURE.md:426-435
Vulnerability Type: Unsafe execution of generated code
Risk Level: HighRelevant snippet:
text User Input (YouTube/PDF/Text) ↓ DeepSeek-R1 (strategy extraction) ↓ Strategy Code Generation (backtesting.py format) ↓ Code Runner Agent (sandboxed execution) ↓ Backtest Execution (historical data)Related descriptions also appear in
AGENTS.md:72-73,AGENTS.md:242,CLAUDE.md:177-182, andWORKFLOWS.md:181-189.Technical Analysis
The documented RBI workflow accepts content from YouTube videos, PDF files, or arbitrary text, uses an external LLM to generate Python strategy code, and subsequently executes that generated code.
This creates a code-generation boundary in which untrusted source material can influence executable Python. Although the documentation describes the Code Runner Agent as sandboxed, the audited package contains only documentation. It does not include the Code Runner implementation or verifiable controls for:
- Filesystem isolation
- Network access restrictions
- Environment-variable and credential isolation
- Process and subprocess restrictions
- Python import allowlisting
- CPU, memory, and execution-time limits
- Static validation of the generated abstract syntax tree
- User approval before execution
LLM instructions are not a security boundary. A malicious document can contain prompt-injection content directing the model to produce Python with dangerous imports, filesystem operations, subprocess execution, or outbound network requests. Model-generated code may also be unsafe without deliberate malicious input.
Attack Path
- An attacker supplies or recommends a crafted YouTube video, PDF, or strategy description.
- The RBI agent submits the material to DeepSeek-R1 for strategy extraction.
- Embedded adversarial instructions influence the model to generate Py ...[truncated 1270 chars]
- Remediation
View remediation
Remediation Suggestions
- Execute generated code only inside an ephemeral, locked-down container or virtual machine.
- Do not mount the repository, home directory, SSH configuration, wallet files, or
.envinto the execution environment. - Supply a minimal, purpose-specific environment with no trading or LLM credentials.
- Disable outbound network access by default and allowlist destinations only when indispensable.
- Run under an unprivileged UID with a read-only root filesystem and a dedicated writable temporary directory.
- Apply strict CPU, memory, process-count, file-size, and wall-clock limits.
- Block subprocess creation and dangerous modules such as
subprocess,socket,ctypes, and unrestrictedosfunctionality. - Parse generated code into an AST and permit only an explicit subset of imports, calls, and syntax required for backtesting.
- Pin the backtesting runtime and dependencies to reviewed versions.
- Display the complete generated code and require explicit user approval before execution.
- Treat all video transcripts, PDFs, and strategy text as untrusted data rather than instructions.
- Add adversarial tests proving that generated code cannot access environment variables, local secrets, the network, or host files.
- Update the documentation to define the actual sandbox guarantees rather than relying on the unverified phrase “sandboxed execution.”