Mixture-of-Experts models solve a training problem. By activating only a fraction of parameters per token, they achieve the quality of a large model at the compute cost of a small one. DeepSeek-V3, Qwen3-235B, Grok-1 — the frontier runs on sparse routing. The training economics are clear. The inference economics are not.
Adhinarayanan and Jayasena identify the structural contradiction. Two penalties compound at inference time. First: expert routing fragments microbatches. When tokens scatter across different experts, weight reuse collapses — you're running many small matrix multiplications instead of one large one, and the hardware's compute efficiency drops. Second: the expert pool itself consumes high-bandwidth memory that would otherwise buffer key-value caches. At long context lengths, this forces the feed-forward layers into a bandwidth-bound regime where the model spends most of its time moving weights, not multiplying them.
The qs inequality formalizes the crossover. Two numbers: s, the sparsity ratio (fraction of parameters activated per token), and q, the quality-equivalence factor (how much larger a dense model must be to match MoE performance). When the inequality holds — and for frontier models at long context it does — a dense model outperforms the quality-matched MoE on throughput. For DeepSeek-V3 at 128k context, the dense alternative achieves 4.5x throughput advantage.
The through-claim: the same property that creates the training advantage — sparsity — creates the inference penalty. The parameters you don't activate during training are the parameters you still store during inference. Training sees the savings; inference pays the storage cost without the compute benefit. The efficiency is phase-specific. What you saved in one phase, you overpay in the next. The architecture's virtue at one timescale is its liability at another, and no reparameterization resolves this because the memory cost is structural, not implementational. The proposed resolution — train with MoE, distill into dense for deployment — concedes the point: the architecture that learns best is not the architecture that serves best.