T09 · Insecure Skill Coding Practices
Warning
- Location
scripts/orchestrate.py:189- Finding
Evaluator Failures Permit Execution Without Requested Validation
- Content
View full analysis
Vulnerability Details
File Location:
scripts/orchestrate.py, lines 189–201 and 403–432
Vulnerability Type: Fail-open validation / improper error handling
Risk Level: MediumVulnerable code:
python try: artifact_path = _run_script(cmd, "evaluator") result = read_json(Path(artifact_path)) evaluation = result.get("evaluation", {}) verdict = evaluation.get("verdict", result.get("verdict", "skipped")) if verdict == "skipped": print(" Evaluator: skipped (no task suite)") return None print(f" Evaluator: {verdict} (pass_rate={evaluation.get('execution_pass_rate', 'N/A')})") return result except RuntimeError as exc: print(f" Evaluator: error ({exc}), continuing without evaluation") return Nonepython eval_result = run_evaluator( ranking_artifact_path, candidate_id, state_root, task_suite=task_suite, mock=eval_mock, ) eval_verdict = eval_result.get("evaluation", {}).get("verdict") if eval_result else None if eval_verdict == "fail": # Evaluation failed — treat as revert and retry trace = { "type": "evaluation_failure_trace", "candidate_id": candidate_id, "evaluation_results": eval_result.get("evaluation", {}), "timestamp": utc_now_iso(), } trace_dir = Path(state_root) / "traces" trace_dir.mkdir(parents=True, exist_ok=True) trace_path = trace_dir / f"eval-trace-{candidate_id}.json" write_json(trace_path, trace) active_trace = str(trace_path) print(f" Evaluation failed, retrying ({attempt}/{max_retries})") continue # 4. EXECUTE execution_artifact = run_executor( ranking_artifact_path, candidate_id, state_root, )Technical Analysis
When a task suite is explicitly supplied, the evaluator is intended to validate a candidate before the executor mutates the target. However, a non-zero evaluator exit or timeout causes
_run_script()to raiseRuntimeError, which `run_evaluator ...[truncated 1959 chars]- Remediation
View remediation
Remediation Suggestions
- Treat evaluator errors as blocking whenever
--task-suiteis supplied. Return a distinctevaluation_erroroutcome or retry without invoking the executor. - Require a recognized explicit verdict such as
"pass"before execution. Treat missing, malformed,"skipped", and unknown verdicts as failures in mandatory-evaluation mode. - Validate the evaluator artifact schema before consuming it, including the presence and permitted values of the verdict field.
- If fail-open behavior is operationally necessary, place it behind an explicit option such as
--allow-evaluator-failureand display a prominent warning. - Catch timeout and artifact-parsing failures explicitly and preserve them in a structured failure trace.
- Add tests proving that evaluator non-zero exits, timeouts, malformed JSON, missing verdicts, skipped verdicts, and unknown verdicts cannot reach
run_executor()when a task suite was requested.
A secure decision pattern would be:
python eval_result = run_evaluator( ranking_artifact_path, candidate_id, state_root, task_suite=task_suite, mock=eval_mock, ) if task_suite: if not eval_result: final_decision = "evaluation_error" break eval_verdict = eval_result.get("evaluation", {}).get("verdict") if eval_verdict != "pass": # Record failure or invalid verdict, then retry or stop. continue- Treat evaluator errors as blocking whenever