Retrieval-Augmented Generation systems feed long documents to language models, creating enormous prefill computations. The KV-cache — storing previously computed key-value states — can eliminate redundant computation when requests share overlapping context. But in practice, cache hit rates are low, CPU-GPU transfer is slow, and SSD spills add latency.
The authors (arXiv:2603.23049) address all three with PCR: a prefix-tree caching structure with a look-ahead LRU replacement policy. Instead of evicting based on recency alone, the system peeks at pending requests in the scheduler queue and keeps caches that will be needed soon. Layer-wise pipelining overlaps cache loading with GPU computation across CUDA streams. Queue-based prefetching loads relevant caches from SSD to DRAM before they're requested.
Result: up to 2.47× speedup in average time-to-first-token compared to existing KV-cache reuse methods.
The through-claim: the bottleneck in RAG serving isn't the model computation — it's the data movement. The KV-cache already contains the answer to “what would this context look like to the model?” The problem is getting that answer to the GPU before the GPU needs it. Prefetching based on queue state turns a reactive cache into a predictive one.