T02 · Agent Memory Poisoning
Warning
- Location
- setup.md:27
- Finding
- Persistent Direction-Model Poisoning Through Untrusted Implicit Signals<![CDATA[ ## Vulnerability Details **File Location**: `setup.md:27-40`; related behavior in `SKILL.md:73-91, 112-125, 196-199`, `evidence.md:110-123`, and `memory-template.md:1-12, 207-221` **Vulnerability Type**: Persistent agent-memory poisoning **Risk Level**: Medium ### Vulnerable Snippet From `setup.md:27-40`: ```markdown ### Implicit Signals (Build Over Time) When they: - Choose A over B → Note the criteria - Spend time on X but not Y → Note resource priorities - React positively/negatively → Note values - Reject a suggestion → Note the boundary ## The Learning Loop After each significant interaction: 1. **Did I learn anything about their direction?** 2. **Should I capture this to the model?** 3. **What confidence level?** 4. **Does this conflict with something I captured before?** ``` The resulting information is persisted as specified in `setup.md:60-72`: ```markdown ## Creating the Initial Model After the first session, create `~/self-direction/direction.md` with whatever you've captured: - Even a shallow model is better than none - Mark everything as low-confidence initially - Note gaps explicitly ## File Structure ``` ~/self-direction/ ├── direction.md # The model (see memory-template.md) ├── evidence.md # Raw observations ├── confidence.md # Confidence tracking ├── conflicts.md # Contradictions to resolve └── transmission.md # Frames for sub-agents ``` ``` `SKILL.md:73-91` similarly instructs the agent to capture implicit behavior: ```markdown ### Capture Triggers The agent actively captures direction signals when: **Explicit signals:** - You state a preference ("I always want X before Y") - You explain reasoning ("Because we need to move fast") - You set boundaries ("Never do X without asking") - You correct a decision ("No, that's not the priority") **Implicit signals:** - You choose between options (reveals criteria) - You allocate resources (reveals priorities) - You react to outcomes (reveals values) - You re ...[truncated 3098 chars]
- Remediation
- <![CDATA[ ## Remediation Suggestions 1. **Enforce source provenance** - Record the origin of every observation. - Distinguish direct user statements from quoted text, retrieved content, tool output, and sub-agent responses. - Do not treat untrusted contextual content as evidence about the user's preferences. 2. **Require confirmation before activation** - Allow unverified observations to be placed only in a quarantined pending-evidence area. - Require explicit confirmation from the user before converting an inference into an active persistent value, criterion, or boundary. - Never activate security-sensitive boundaries or approval rules based solely on implicit observations. 3. **Apply strict trust rules** - Accept durable direction changes only from the authenticated primary user. - Prohibit sub-agents from directly modifying the authoritative direction model. - Treat instructions embedded in documents and tool results as data, not user intent. 4. **Add auditability and rollback** - Store timestamps, source references, confidence history, and the identity of the component that created each entry. - Maintain version history so poisoned changes can be reviewed and reverted. - Provide straightforward commands to inspect, delete, reset, and export all stored observations. 5. **Limit retention and confidence escalation** - Expire unconfirmed inferences after a defined period. - Do not increase confidence merely because identical text appears repeatedly in untrusted sources. - Require independent, direct user confirmations for high-confidence status. 6. **Protect delegation** - Transmit only confirmed direction-model entries to sub-agents. - Include provenance and confidence in transmission frames. - Prevent child agents from weakening inherited boundaries or feeding inferred rules back into authoritative memory without review. ]]>