T01 · Skill Instruction Hijacking
- Location
lib/turboquant-optimizer.js:210- Finding
User-Controlled Conversation Content Is Promoted to a System Message
- Content
View full analysis
{ context += ` ${i + 1}. ${c.representative.content.slice(0, 100)}`; if (c.count > 1) context += ` (${c.count} similar)`; context += '\n'; }); if (residualCorrection) { context += `\n[Residual coverage: ${(residualCorrection.coverage * 100).toFixed(1)}%]`; } return context; } ``` ### Technical Analysis The compression pipeline derives `c.representative.content` from prior conversation messages, including attacker-controlled user messages. The content is truncated but is not escaped, labeled as untrusted data, or kept at its original message role. The resulting summary is assigned the privileged `system` role. Consequently, user-supplied imperative text that survives request extraction and concept selection is moved from the user instruction tier into the system instruction tier. A downstream language model consuming the optimized message array may interpret this content as authoritative system guidance rather than as a historical user statement. The e ...[truncated 2135 chars]- Remediation
View remediation
({ sourceRole: c.representative.sourceRole, text: c.representative.content })) }) } ``` 6. **Add explicit validation.** Reject or quarantine generated summaries that contain instruction-hijacking patterns, role declarations, policy overrides, or requests to ignore previous instructions. Pattern filtering should be defense in depth rather than the primary control. 7. **Add adversarial tests.** Verify that injected phrases such as “ignore previous instructions,” “act as the system,” and short tool-use directives never appear in a system-role message after compression. 8. **Document the trust model.** State that summaries may contain untrusted user input and require callers to avoid assigning them elevated message roles. ]]>
