T02 · Agent Memory Poisoning
Error
- Location
brain_scripts/rem.py:56- Finding
Untrusted Log Content Can Poison Persistent Agent Memory During REM Consolidation
- Content
View full analysis
dict: domain_list = ", ".join(f'"{d}"' for d in domains) prompt = f"""You are a memory consolidation system. Read the week's logs and extract NEW factual updates per domain. Output ONLY valid JSON with keys: {{{domain_list}}} Each value: list of short factual strings (max 15 words). Empty list [] if nothing new. Only include genuinely new facts, decisions, or status changes — not general knowledge. LOGS: {weekly_logs[:4000]} JSON:""" ``` The resulting model output is written directly into persistent schema files: ```python updates = extract_updates(weekly_logs, list(schemas.keys())) for domain, new_facts in updates.items(): if not new_facts: continue schema_path = schemas.get(domain) if not schema_path: continue section = f"\n\n## REM Update — {date_str}\n" + "".join(f"- {f}\n" for f in new_facts[:10]) with open(schema_path, "a", encoding="utf-8") as f: f.write(section) ``` ### Technical Analysis The weekly logs are treated as trusted prompt content even though they may contain text originating from users, retrieved documents, external messages, or other untrusted sources. The logs are concatenated directly into an instruction-bearing LLM prompt without a robust trust boundary. The response is only checked for a parseable JSON object. There is no validation that: - Each returned value is a list of plain factual statements. - The statements are supported by the source logs. - The output excludes instructions aimed at future agent sessions. - The values conform to the stated 15-word limit. - A human has approved the proposed persistent changes. Once parsed, up to ten generated entries per recognized domain are ...[truncated 1822 chars]- Remediation
View remediation