AI agents that maintain long conversation histories face a scaling problem: as the number of past exchanges grows, the context window fills, and retrieval quality degrades under the sheer volume of stored interactions. The obvious solution — summarization — loses the specific details that make retrieval useful. The alternative — keeping everything verbatim — doesn't scale.
Lewis (arXiv:2603.13017) proposes structured distillation: decomposing each conversation exchange into four components — exchange core (what happened), specific context (why it happened), thematic room assignments (what topic it belongs to), and regex-extracted files touched. Applied to 4,182 conversations across six software engineering projects, this reduces average exchange length from 371 to 38 tokens — an 11x compression.
The evaluation against 201 queries shows the distilled version achieves 96% of verbatim retrieval performance with vector search. BM25 (keyword-based) search degrades significantly because the compression removes the lexical redundancy that BM25 relies on. A hybrid approach — distilled for vector search, with fallback to original for detailed review — slightly exceeds baseline performance.
The structural insight is that conversation exchanges contain a small core of retrievable information surrounded by a large volume of contextual scaffolding that aided the original conversation but doesn't aid retrieval. The four-component decomposition separates what's searchable from what's explanatory. Thousands of exchanges fit in a single prompt, making long-term agent memory practical without abandoning access to the full record when needed.