LLM serving batches prefill requests together. A prefill computes the key-value cache for a prompt before generation begins. The standard optimization: batch multiple prefills to saturate GPU compute. All prefills look the same to the scheduler — a block of tokens to process before the real work starts.
But not all prefills are equal. In multi-turn conversations, a new turn reuses most of the KV cache from prior turns. Only the new user message needs prefilling. This “incremental prefill” is tiny — a few hundred tokens — compared to a fresh conversation's prefill of thousands. Batching them together wastes the GPU: the small incremental prefills complete instantly while the large fresh prefills hold the batch open.
PPD (Prefill-Phase Disaggregation) segregates prefills by their characteristics. Fresh-conversation prefills go to one pool. Incremental multi-turn prefills go to another. The pools are scheduled independently, with different batch sizes and different resource allocations. Small incremental prefills complete in microseconds and free their resources immediately, rather than waiting for large prefills to finish.
The throughput improvement is substantial — not from faster hardware or better algorithms but from the recognition that a category treated as uniform was actually bimodal. The scheduler that treats all prefills identically pays the cost of the slowest prefill in every batch. The scheduler that segregates pays only the cost of each category's own workload.
The structural lesson: batching helps until the variance within a batch dominates the benefit. Uniform treatment of non-uniform work is an implicit averaging that hides inefficiency. The fix is not a better batching strategy but a finer classification — recognizing that “prefill” is not one operation but two, and scheduling accordingly. The optimization was in the taxonomy, not the algorithm.