T09 · Insecure Skill Coding Practices
- Location
scripts/orchestrate.py:188- Finding
Evaluator Failure Allows Candidate Execution Without Requested Validation
- Content
View full analysis
Vulnerability Details
File Location:
scripts/orchestrate.py:188-197andscripts/orchestrate.py:397-421
Vulnerability Type: Fail-open validation and safety-control bypass
Risk Level: HighVulnerable Code
python try: artifact_path = _run_script(cmd, "evaluator") result = read_json(Path(artifact_path)) verdict = result.get("verdict", "skipped") if verdict == "skipped": print(" Evaluator: skipped (no task suite)") return None print(f" Evaluator: {verdict} (pass_rate={result.get('evaluation_results', {}).get('execution_pass_rate', 'N/A')})") return result except RuntimeError as exc: print(f" Evaluator: error ({exc}), continuing without evaluation") return NoneThe returned
Nonevalue permits the pipeline to continue into execution:python eval_result = None if skip_eval else run_evaluator( ranking_artifact_path, candidate_id, state_root, task_suite=task_suite, eval_threshold=cat_threshold, mock=eval_mock, ) eval_artifact_path: str | None = None if eval_result: eval_artifact_path = eval_result.get("truth_anchor") if eval_result and eval_result.get("verdict") == "fail": # Evaluation failed — treat as revert and retry trace = { "type": "evaluation_failure_trace", "candidate_id": candidate_id, "evaluation_results": eval_result.get("evaluation_results", {}), "timestamp": utc_now_iso(), } trace_dir = Path(state_root) / "traces" trace_dir.mkdir(parents=True, exist_ok=True) trace_path = trace_dir / f"eval-trace-{candidate_id}.json" write_json(trace_path, trace) active_sources.append(str(trace_path)) print(f" Evaluation failed, retrying ({attempt}/{max_retries})") continue # 4. EXECUTE execution_artifact = run_executor( ranking_artifact_path, candidate_id, state_root, )Technical Analysis
When evaluation is required, an explicit
failverdict prevents execution an ...[truncated 2099 chars]- Remediation
View remediation
Remediation Suggestions
-
Fail closed when evaluation was requested but cannot complete:
- Stop the pipeline;
- Mark the candidate as rejected;
- Or create a failure trace and retry without executing the candidate.
-
Return a structured evaluator status instead of overloading
None, for example:
python { "status": "error", "verdict": "blocked", "reason": str(exc), }- Track whether evaluation is mandatory:
python evaluation_required = bool(task_suite) and not skip_evalIf
evaluation_requiredis true, require all of the following before execution:- A valid evaluation artifact;
- A recognized verdict;
- A passing verdict;
- A valid
truth_anchor.
-
Make the gate reject missing evaluator evidence whenever evaluation was required. This provides defense in depth if the orchestration layer is bypassed.
-
Add tests covering:
- Evaluator timeout;
- Evaluator non-zero exit;
- Malformed evaluator artifact;
- Missing
truth_anchor; - Confirmation that the executor is not called after any mandatory-evaluation error.
-