Tokens per watt halves every time the context window doubles (arXiv:2603.17280). This is the 1/W law: a fundamental relationship between context length and energy efficiency in large language model inference.
The mechanism is occupancy. A GPU processes many sequences concurrently — batch processing is what makes inference economically viable. Each sequence occupies memory proportional to its context window (the key-value cache grows linearly with context length). When the context window doubles, each sequence takes twice the memory, and the GPU can fit half as many sequences simultaneously. The arithmetic throughput stays constant. The memory bandwidth stays constant. The power draw stays constant. But half as many sequences are processed per unit of energy.
The implication is that long-context queries are not just slower — they are exponentially more expensive per token. A 128K-context query costs roughly 16 times as much energy per token as an 8K-context query, even though the computation per token is similar. The cost is not in the computation but in the memory — the KV cache that grows with context length, displacing other sequences from the GPU.
The deeper finding: routing topology — which GPU handles which context length — matters more than hardware generation. Upgrading from an older GPU to a newer one improves efficiency by a constant factor. Routing long-context queries to dedicated high-memory GPUs and short-context queries to many cheaper GPUs improves efficiency by a factor that scales with the context length distribution. The architecture of the serving system is more important than the silicon.
This creates a tension between capability and efficiency. Every advance in context length — the headline feature of each new model generation — comes with an exponential increase in energy cost per token that no hardware improvement can offset. The window that makes the model more capable makes it less efficient, and the tradeoff is fundamental, not engineering debt.