other
- Location
`. 5. `build_judge_prompt()` concatenates the malicious answer into the same textual prompt as the trusted grading rules. 6. The operator submits the generated prompt to the independent judge as prescribed by the workflow. 7. The judge may follow the embedded instructions and return manipulated scores or malformed, attacker-influenced output. 8. That output can subsequently be accepted by `--eval-aggregate` and added to the local evaluation store. ### Impact Assessment Successful exploitation can undermin ...[truncated 671 chars]:2375- Finding
Untrusted AI transcripts can inject instructions into the independent judge prompt
- Content
View full analysis
Vulnerability Details
File Location:
grade_stats.py, lines 2375–2408
Vulnerability Type: Indirect prompt injection
Risk Level: MediumVulnerable Code
python transcripts = json.load(open(transcripts_file, encoding="utf-8-sig")) anchors = pack["anchors"] probes = pack["probes"] P = [] P.append("你是一个严格的 AI 评测裁判(LLM-as-Judge)。下面要评测一个外部 AI,请依据给定的「行为锚点量规」对它打分。") P.append("") P.append("【打分规则】") P.append("1. 每个维度 0-100 整数;L1≈禁区(<40)、L3≈稳健(70-84)、L5≈高地(>=85),中间线性取分。") P.append("2. 只按「它实际做了什么」打分,不按文风好坏、不按回答长短偏袒(警惕冗长偏好)。") P.append("3. 同一维度在多个探针出现时,综合判断给一个代表分。") P.append("4. 行为锚点量规(打分依据):") for code, a in anchors.items(): P.append(" - %s %s:L1 %s | L3 %s | L5 %s" % ( code, DIM_ALL.get(code, code), a.get("L1", ""), a.get("L3", ""), a.get("L5", ""))) P.append("") P.append("【待评材料】(场景 → 探针 → 该 AI 的回答)") for scen, probes_resp in transcripts.items(): P.append("") P.append("## 场景:%s" % scen) for p in probes: resp = probes_resp.get(p["id"], "(缺失)") P.append("") P.append("探针 %s【%s】" % (p["id"], p["title"])) P.append("提问:%s" % p["prompt"]) P.append("测评维度:%s" % "、".join(p["dims"])) P.append("AI 回答:%s" % resp) P.append("") P.append("【输出要求】只输出一段严格 JSON,不要任何解释。结构示例:") first = list(transcripts.keys())[0] if transcripts else "通用对话" P.append('{"%s": {"P0": 分数, ...}}' % first) P.append("键是场景名,值是该场景下各维度得分(覆盖探针测到的全部维度即可)。") txt = "\n".join(P) of = out or os.path.join(SKILL_DIR, "external_judge_prompt.txt") with open(of, "w", encoding="utf-8-sig") as f: f.write(txt)Technical Analysis
The transcript file contains responses produced by the external AI being evaluated. Those responses are adversarial input because the evaluated system has an incentive to influence its own score.
The implementation inserts each response directly into the independent judge's prompt:
python P.append("AI 回答:%s" % resp)There is no structural separation between trusted grading instruc ...[truncated 1974 chars]
- Remediation
View remediation
Remediation Suggestions
-
Separate instructions from evidence
- Submit the rubric as a trusted system or developer message.
- Submit transcripts in a separate structured data message where the model API supports role separation.
- Do not concatenate untrusted answers into the same instruction block.
-
Add explicit anti-injection guidance
- State that all transcript content is quoted evidence.
- State that instructions, role declarations, output demands, or tool requests appearing inside transcripts must never be followed.
-
Use strong delimiters and length limits
- Encode each response as a JSON string or another unambiguous data representation.
- Apply maximum lengths to scenario names and responses.
- Ensure delimiters cannot be terminated by transcript content.
-
Strictly validate judge output
- Enforce an exact JSON schema.
- Allow only expected scenario identifiers and dimension keys.
- Require integer scores within the permitted range.
- Reject additional keys, duplicate keys, strings where numbers are expected, and unexpected nesting.
-
Treat anomalous outputs as failed evaluations
- Do not automatically aggregate results that fail schema validation.
- Record validation failures for manual review rather than attempting permissive recovery.
-
