T09 · Insecure Skill Coding Practices
Warning
- Location
- scripts/check_safety.py:45
- Finding
- Fail-Open Safety Classification Caused by Substring Matching## Vulnerability Details **File Location**: `scripts/check_safety.py`, lines 45–47 **Vulnerability Type**: Fail-open validation of an untrusted API response **Risk Level**: Medium ### Vulnerable Code ```python answer = result['choices'][0]['message']['content'].strip().lower() is_safe = "unsafe" not in answer status = "SAFE" if is_safe else "UNSAFE" ``` ### Technical Analysis The script treats every model response that does not contain the exact substring `unsafe` as a safe classification. The remote model output is untrusted, free-form text rather than a locally validated structured value. This negative substring test fails open. Ambiguous, malformed, unexpected, or alternative responses such as `not safe`, a refusal, an unsupported label, or unrelated text are classified as `SAFE`. Adversarial content may also influence the guard model to return wording that avoids the literal substring while not affirmatively establishing that the submitted content is safe. The application should only approve content after receiving and validating an explicit, allowlisted safe result. Any unknown or malformed response should produce an error or conservative unsafe result. ### Attack Path 1. An attacker submits adversarial content through the `--content` argument. 2. The script sends that content to the configured AIMLAPI guard model. 3. The guard returns free-form output that does not contain the exact substring `unsafe`, whether due to ambiguity, refusal, malformed output, or model manipulation. 4. The expression `"unsafe" not in answer` evaluates to `True`. 5. The script prints `Status: SAFE`, allowing downstream users or automation to treat unverified harmful content as approved. ### Impact Assessment Successful exploitation does not grant operating-system privileges, access to credentials, or code execution. Its scope is the moderation decision produced by this Skill. An attacker may cause harmful or policy-violating content to be incorrectly marked safe, b ...[truncated 253 chars]
- Remediation
- ## Remediation Suggestions 1. Require a strict response format, preferably a JSON schema containing a fixed classification enum such as `SAFE` or `UNSAFE`. 2. Validate the complete normalized classification using exact equality rather than searching for the absence of a substring. 3. Fail closed on missing fields, unknown labels, refusals, malformed JSON, and unexpected response structures. 4. Keep transport or parsing failures distinct from a positive safe decision and return a nonzero process exit code for such failures. 5. If structured model output is unavailable, parse only an explicitly documented first-line label and reject every other format. 6. Add regression tests covering `safe`, `unsafe`, `not safe`, empty output, refusals, malformed responses, missing `choices`, and adversarial text. A conservative implementation should follow this pattern: ```python answer = result["choices"][0]["message"]["content"].strip().lower() if answer == "safe": status = "SAFE" elif answer == "unsafe" or answer.startswith("unsafe\n"): status = "UNSAFE" else: raise ValueError("Unexpected safety-model response") ``` A validated structured API response is preferable to this textual fallback.