LLM inference latency is usually attributed to GPU compute and memory bandwidth. Performance optimization focuses on kernel fusion, quantization, and attention improvements. But between the high-level framework and the GPU silicon, translation layers consume time invisible to standard profiling.
Vellaisamy et al. (arXiv:2603.12465) decompose inference overhead into three categories: framework translation (converting operations into GPU API calls), CUDA library translation (converting API calls into specific kernel implementations), and kernel launch-path delays (CPU-side orchestration of launching each kernel). On NVIDIA H100 and H200, these overheads are substantial and previously unmeasured.
The headline finding: mixture-of-experts models generate 8-11x more kernels per output token than dense models. Each expert requires its own kernel launch, and the routing decision itself requires launches. The CPU must orchestrate all of these sequentially before the GPU can begin. For MoE architectures, the CPU becomes the bottleneck — a faster host processor reduces orchestration overhead by 10-29%.
The Host-Device Balance Index (HDBI) quantifies whether the bottleneck is on the host (CPU orchestration) or device (GPU compute) side. Dense models during generation are device-bound — GPU compute dominates. MoE models during prefill can be host-bound — CPU orchestration dominates. The optimization strategy differs fundamentally: device-bound inference benefits from faster GPUs; host-bound inference benefits from faster CPUs. Buying a better GPU for a host-bound workload wastes money.