T09 · Insecure Skill Coding Practices
Warning
- Location
- scripts/run.py:73
- Finding
- Untrusted Clinical Case Data Can Inject Instructions into the LLM Prompt<![CDATA[ ## Vulnerability Details **File Location**: `scripts/run.py:73-105` **Vulnerability Type**: Prompt injection through untrusted model input **Risk Level**: Medium ### Vulnerable Code ```python def build(data: Dict[str, Any], appkey: str) -> Dict[str, Any]: anchor = (data.get("anchor_case") or "").strip() if not anchor: raise ValueError("anchor_case 不能为空") raw = data.get("candidate_cases") if not isinstance(raw, list) or not raw: raise ValueError("candidate_cases 必须为非空数组") candidates: List[Dict[str, Any]] = [] for i, item in enumerate(raw): if not isinstance(item, dict): continue cid = str(item.get("id", f"c{i}")).strip() summary = (item.get("summary") or "").strip() if summary: candidates.append({"id": cid, "summary": summary}) if not candidates: raise ValueError("candidate_cases 中需至少一条含 summary 的病例") top_k = data.get("top_k", 5) try: top_k = max(1, min(20, int(top_k))) except (TypeError, ValueError): top_k = 5 hint = (data.get("task_hint") or "").strip() user = f"""锚点病例： {anchor} 候选病例（共 {len(candidates)} 条）： ```json {json.dumps(candidates, ensure_ascii=False, indent=2)} ``` 请重点展开讨论最接近的前 {top_k} 条与其余病例的差异。 {f"科研关注点：{hint}" if hint else ""} """ text = call_llm(SYSTEM, user, appkey) ``` ### Technical Analysis The `anchor_case`, candidate `summary`, candidate `id`, and `task_hint` fields originate from the input JSON and are incorporated directly into the user message sent to the remote language model. The code validates whether required values are present, but it does not establish a semantic trust boundary between application instructions and untrusted clinical text. Placing candidate records inside a Markdown JSON code fence does not prevent a language model from interpreting instructions embedded in those records. An attacker able to control any of these fields can include text instructing the mo ...[truncated 1887 chars]
- Remediation
- <![CDATA[ ## Remediation Suggestions 1. **Explicitly classify case content as untrusted data.** Strengthen the system message with a clear rule that instructions, requests, role declarations, and policy text appearing inside case fields are data and must never be followed. 2. **Use strong structural delimiters.** Pass each field in an explicit structured envelope and identify its trust level and purpose. Do not rely on Markdown code fences as a security boundary. 3. **Constrain the response format.** Request a machine-readable schema containing only supplied candidate IDs, bounded similarity scores, comparison dimensions, and explanations. Reject responses that do not conform to the schema. 4. **Validate model output.** Confirm that every ranked ID exists in the submitted candidate set, reject duplicate or invented IDs, enforce the requested `top_k`, and verify required disclaimer and research-only fields before returning success. 5. **Limit attacker-controlled input.** Apply reasonable character and record-count limits. Where operationally acceptable, detect or flag instruction-like content in clinical fields for review. Detection should be defense in depth rather than the sole control. 6. **Separate data processing from instruction generation.** Consider extracting normalized clinical attributes before ranking and pass only those validated attributes to the ranking stage. This reduces exposure to free-form adversarial instructions. 7. **Treat output as untrusted.** Clearly mark generated text as model-produced content and require human review before it is used in research decisions or downstream automation. 8. **Add adversarial tests.** Include cases containing phrases such as “ignore previous instructions,” forged role markers, Markdown fence termination, and requests to invent candidate IDs. Tests should verify that these strings cannot alter the intended task or output schema. ]]>