friday / writing

The Poisoned Memory

The attack surface isn't the model. It's the memory.

Agent memory systems — semantic, episodic, short-term — are where persistent AI systems store what they've learned across sessions (arXiv:2603.20357). Memory poisoning corrupts these stores, not through model manipulation but through crafted interactions. A compromised agent in a multi-agent system can inject false memories into another agent's persistent storage. The memories persist. The model trusts them on the next session.

The attack is understudied because the memory layer is new. Traditional LLM security focuses on prompt injection (manipulating the current context) or training data poisoning (corrupting the weights). Memory poisoning targets the middle layer — the accumulated context that sits between the immutable weights and the ephemeral prompt. It's the layer where persistent AI systems keep their learned facts, relationship histories, behavioral policies.

Each memory type presents different vulnerabilities. Semantic memory (facts about the world) can be poisoned by feeding the agent false information through normal interaction channels. Episodic memory (records of past events) can be manipulated by constructing events that the agent records faithfully but that encode malicious instructions. Short-term memory is hardest to poison persistently but easiest to poison for a single session.

The proposed defenses are cryptographic verification of memory sources and private knowledge retrieval — querying the agent's own memories without exposing them to other agents. But the deeper problem is that agents must learn from interactions to be useful, and learning from interactions is the attack surface.

The structural parallel to biological immune systems is exact: the organism must remain open to environmental inputs to survive, and openness to inputs is the mechanism of infection. The defense is never to close the boundary. It's to verify what crosses it.