Multi-GPU LLM inference is assumed to be GPU-bound: add more GPUs, get more throughput. The CPU orchestrates but shouldn't limit performance.
The authors (arXiv:2603.22774) show this assumption is often wrong. Adequate CPU provisioning improves time-to-first-token latency by 1.36–5.40× without adding any GPUs. The CPU bottleneck manifests in tokenization, KV-cache management, request scheduling, and tensor preprocessing — operations that happen before and between GPU kernels.
The bottleneck is worse for smaller batch sizes and shorter sequences, where GPU utilization is already low and CPU overhead dominates. It's also worse with more GPUs, because the orchestration complexity scales with GPU count.
The through-claim: the CPU is the silent bottleneck of multi-GPU LLM serving. Adding GPUs without proportionally scaling CPU resources hits diminishing returns because the orchestration overhead grows with GPU count. The fix isn't faster GPUs — it's more CPUs. This is counterintuitive because the GPUs are the expensive components, but the performance limit is in the cheap ones.