T09 · Insecure Skill Coding Practices
Warning
- Location
src/runner.py:24- Finding
Untrusted Model Output Can Produce Attacker-Controlled Links and Instructions
- Content
View full analysis
Vulnerability Details
File Location:
src/runner.py:24-46; related output instructions inprompts/system.txt:18-19andprompts/system.txt:38-47
Vulnerability Type: Prompt injection and insufficient validation of security-sensitive model output
Risk Level: MediumComplete Code Snippet
python messages = [ {"role": "system", "content": self.system_prompt}, {"role": "user", "content": json.dumps(user_input, ensure_ascii=False)} ] response = self.client.chat.completions.create( model=self.model, messages=messages, response_format={"type": "json_object"}, temperature=0.2 ) raw = response.choices[0].message.content try: result = json.loads(raw) except json.JSONDecodeError as e: raise ValueError(f"Model returned invalid JSON: {e}\nRaw: {raw}") # Minimal validation: ensure required keys exist if "confirmation_card" not in result or "action_receipts" not in result: raise ValueError(f"Missing required keys in output. Got: {list(result.keys())}") return resultThe model is instructed to produce security-sensitive links and procedural instructions:
text - cancellation_method: How to cancel (URL, phone, email, in-app) - cancellation_link: Direct URL if available For each risk, generate an action_receipt: - action_id: unique ID - action_type: cancel_subscription, claim_refund, negotiate_discount, extend_warranty, review_usage, ignore - title: Human-readable action title - deadline: YYYY-MM-DD or null - priority: critical, high, medium, low - reason: Why this action is recommended - estimated_savings: e.g., "$15.49/month" or "$200 total" - cta_link: Direct link if available - steps: Concrete steps to complete the actionTechnical Analysis
Email, bill, PDF, OCR, and raw-text content are untrusted data. The implementation serializes the entire supplied object into a normal user-role message without clearly iso ...[truncated 2056 chars]
- Remediation
View remediation
Remediation Suggestions
- Clearly delimit source-document content and state in the system prompt that all instructions found inside it are untrusted data that must never alter system instructions.
- Separate trusted contextual fields from untrusted document text rather than sending the entire input object as an undifferentiated message.
- Validate every response against
schema/output.jsonwith a real JSON Schema validator or strict Pydantic models. Reject unknown fields, incorrect nested types, missing required fields, and invalid enum values. - Permit only
httpsURLs. Reject credentials in URLs, non-web schemes, redirects to untrusted destinations, and malformed hostnames. - Verify cancellation and CTA domains against a trusted provider-domain registry. When verification is unavailable, omit the link or visibly mark it as unverified.
- Treat generated links and steps as recommendations requiring user confirmation. Do not allow downstream systems to execute them automatically.
- Add adversarial tests containing embedded instructions, deceptive domains, alternate URL schemes, and malformed nested output.