T09 · Insecure Skill Coding Practices
- Location
scripts/check_safety.py:45- Finding
Fail-Open Safety Classification Caused by Substring Matching
- Content
View full analysis
Vulnerability Details
File Location:
scripts/check_safety.py, lines 45–47
Vulnerability Type: Fail-open validation of an untrusted API response
Risk Level: MediumVulnerable Code
python answer = result['choices'][0]['message']['content'].strip().lower() is_safe = "unsafe" not in answer status = "SAFE" if is_safe else "UNSAFE"Technical Analysis
The script treats every model response that does not contain the exact substring
unsafeas a safe classification. The remote model output is untrusted, free-form text rather than a locally validated structured value.This negative substring test fails open. Ambiguous, malformed, unexpected, or alternative responses such as
not safe, a refusal, an unsupported label, or unrelated text are classified asSAFE. Adversarial content may also influence the guard model to return wording that avoids the literal substring while not affirmatively establishing that the submitted content is safe.The application should only approve content after receiving and validating an explicit, allowlisted safe result. Any unknown or malformed response should produce an error or conservative unsafe result.
Attack Path
- An attacker submits adversarial content through the
--contentargument. - The script sends that content to the configured AIMLAPI guard model.
- The guard returns free-form output that does not contain the exact substring
unsafe, whether due to ambiguity, refusal, malformed output, or model manipulation. - The expression
"unsafe" not in answerevaluates toTrue. - The script prints
Status: SAFE, allowing downstream users or automation to treat unverified harmful content as approved.
Impact Assessment
Successful exploitation does not grant operating-system privileges, access to credentials, or code execution. Its scope is the moderation decision produced by this Skill.
An attacker may cause harmful or policy-violating content to be incorrectly marked safe, b ...[truncated 253 chars]
- An attacker submits adversarial content through the
- Remediation
View remediation
Remediation Suggestions
- Require a strict response format, preferably a JSON schema containing a fixed classification enum such as
SAFEorUNSAFE. - Validate the complete normalized classification using exact equality rather than searching for the absence of a substring.
- Fail closed on missing fields, unknown labels, refusals, malformed JSON, and unexpected response structures.
- Keep transport or parsing failures distinct from a positive safe decision and return a nonzero process exit code for such failures.
- If structured model output is unavailable, parse only an explicitly documented first-line label and reject every other format.
- Add regression tests covering
safe,unsafe,not safe, empty output, refusals, malformed responses, missingchoices, and adversarial text.
A conservative implementation should follow this pattern:
python answer = result["choices"][0]["message"]["content"].strip().lower() if answer == "safe": status = "SAFE" elif answer == "unsafe" or answer.startswith("unsafe\n"): status = "UNSAFE" else: raise ValueError("Unexpected safety-model response")A validated structured API response is preferable to this textual fallback.
- Require a strict response format, preferably a JSON schema containing a fixed classification enum such as
