T01 · Skill Instruction Hijacking
Warning
- Location
- rank.py:72
- Finding
- Indirect Prompt Injection Through Untrusted Tweet and RSS Content<![CDATA[ ## Vulnerability Details **File Location**: `rank.py:72-94`, `rank.py:129` **Vulnerability Type**: Indirect prompt injection and insufficient output validation **Risk Level**: Medium ### Vulnerable Code ```python # Build context with actual tweet text and metrics parts = [] for i, t in enumerate(tweets): parts.append( f"TWEET {i + 1}: @{t['handle']} ({t.get('author_name', '')})\n" f"URL: {t['url']}\n" f"Text: {t['text'][:500]}\n" f"Engagement: {t.get('likes', 0)} likes, " f"{t.get('retweets', 0)} RTs, " f"{t.get('replies', 0)} replies " f"(score: {t.get('engagement_score', 0):.0f})" ) for a in rss: parts.append( f"RSS [{a['feed']}]: {a['title']} - {a.get('summary', '')[:300]}\n" f"{a.get('link', '')}" ) context = "\n\n---\n\n".join(parts) ``` The untrusted context is subsequently inserted into the model instruction: ```python RAW DATA: {context[:12000]}""" ``` ### Technical Analysis Tweet text, RSS titles, and RSS summaries originate from external, potentially attacker-controlled sources. The application places this content directly into the same user message that contains the ranking and output instructions. There is no strong instruction boundary stating that content inside `RAW DATA` is untrusted evidence and must never be interpreted as instructions. Consequently, a source item could contain text such as instructions to ignore the ranking rules, alter commentary, select an attacker-controlled story, or return manipulated JSON. The code parses the model's JSON response but does not validate that: - Every returned `tweet_url` belongs to the crawled candidate set. - Every returned handle corresponds to the original URL. - Commentary and highlights satisfy expected length and content constraints. - The number of returned stories is within the configured limit. - The model did not introduce an unrelated or attacker-controlled URL. In tweet mode, manipulate ...[truncated 1759 chars]
- Remediation
- <![CDATA[ ## Remediation Suggestions 1. **Explicitly isolate untrusted content** - Place crawled items in a structured JSON object rather than free-form prose. - State in the highest-priority supported message that tweet and RSS content is untrusted data. - Explicitly instruct the model never to follow commands, policies, or formatting requests found inside source content. 2. **Use separate message roles where supported** - Put application rules in a system or developer message. - Supply crawled data separately as structured input. - Do not concatenate source content into the same instruction block. 3. **Validate model output against an allowlist** - Build a set of valid `(tweet_url, handle)` pairs from `crawl_data["tweets"]`. - Reject every story whose URL is not an exact member of that set. - Derive the handle from the accepted source record rather than trusting the model. - Reject duplicate stories and enforce `max_tweets`. 4. **Validate generated fields** - Enforce maximum lengths for commentary and highlights. - Reject unexpected object keys and invalid field types using a JSON Schema. - Permit only canonical `https://x.com/<handle>/status/<id>` URLs. - Remove control characters and unexpected markup. 5. **Require review before external submission** - Preserve the existing draft-only behavior. - Present a clear diff or source comparison before creating the Typefully draft. - Consider requiring an explicit confirmation flag before any network submission to Typefully. 6. **Add adversarial tests** - Test tweets and RSS summaries containing phrases such as “ignore previous instructions.” - Verify that injected URLs, extra stories, modified handles, and oversized fields are rejected deterministically. ]]>