T09 · Insecure Skill Coding Practices
Error
- Location
scripts/lib_grading.py:82- Finding
Arbitrary Code Execution Through Task-Embedded Grading Code
- Content
View full analysis
GradeResult: grading_code = _extract_grading_code(task) if not grading_code: return GradeResult( task_id=task.task_id, score=0.0, max_score=1.0, grading_type="automated", breakdown={}, notes="No automated grading code found", ) namespace: Dict[str, Any] = {} exec(grading_code, namespace) ``` The executed code is extracted directly from a Python code block in a task Markdown file: ```python def _extract_grading_code(task: Task) -> str: if not task.automated_checks: return "" match = re.search(r"```python\s*(.*?)\s*```", task.automated_checks, re.DOTALL) if not match: return "" return match.group(1) ``` ### Technical Analysis Automated grading logic is loaded from `tasks/task_*.md`, extracted as arbitrary Python source, and passed to `exec()` without validation, signing, sandboxing, or process isolation. The empty namespace does not provide meaningful protection because Python automatically exposes built-ins, including `__import__`. A malicious or compromised task can therefore import modules, inspect environment variables, read files available to the benchmark user, modify the filesystem, execute subprocesses, or initiate network connections. The project documentation supports adding custom tasks, so task files constitute an extensibility boundary rather than immutable internal source code. This creates a direct code-execution primitive with all privileges of the user running PinchBench. ### Attack Path 1. An attacker supplies, modifies, or contributes a `tasks/task_*.md` file. 2. The task contains an `Automated Checks` section with a crafted Py ...[truncated 1144 chars]- Remediation
View remediation