Rotary Positional Embedding (RoPE) encodes token positions in a transformer by rotating key-query pairs through different angles per embedding dimension. At training length, these rotations keep the geometry clean — attention scores behave predictably, and the model can distinguish relevant from irrelevant tokens. Beyond training length, the rotations go out of distribution, and the model fails.
This paper explains the failure geometrically. At training length, the key and query vectors form well-separated clusters in the embedding space. “Sink tokens” — positions that absorb attention when no content is relevant — occupy a distinct geometric region. The separation between content clusters and sink clusters is what allows attention heads to function as information routers.
When inputs exceed training length, the extra rotations applied by RoPE distort this geometry. Channels that should produce moderate rotation produce wild swings. The content and sink clusters collide. Attention heads can no longer distinguish “attend here” from “attend nowhere,” and the model degenerates into pathological behavior — repetition, hallucination, incoherence.
The fix is surgical: apply high-frequency RoPE rotations to only a subset of channels, keeping the rest in-distribution. The authors call this RoPE-ID (In Distribution). Tested on 1B and 3B parameter transformers, it generalizes to inputs far beyond training length without retraining. The key insight is that not all channels need to encode position — some can remain positionally neutral, providing a geometric anchor that prevents the embedding space from collapsing.
The rope frays because every strand is twisted. Keep some strands slack — unrotated, geometrically stable — and the whole structure holds under tension it was never trained for. The failure was in the universality of the encoding, not in the encoding itself.