No matrices to memorise, no layout bookkeeping. Just one geometric idea, three small toys to play with, and the single line of algebra that makes it click.
Assumes only: a vector has a length and a direction, and the dot product a·b = |a||b|cos(angle between them). Everything else is built up below.
Keypoint 1 — the foundation
Attention scores a query against a key with a dot product. Geometrically that number isn't about where the arrows are — only the angle between them. Same direction → big positive score. Perpendicular → zero. Opposite → negative. Lengths only scale it.
Hold onto this: if we can control the angle between query and key, we control their attention score. RoPE's whole job is to make that angle carry position.
Keypoint 2 — the move
Give every token a position number. Then spin its arrow by (position × a fixed rate θ). The query at position m turns by mθ; the key at position n turns by nθ. Because the rate is shared, the angle between them shifts by exactly (m − n)·θ — it depends only on the gap in their positions, never on where they sit absolutely.
Try this: tick lock the gap, then drag. Both arrows whirl around the circle as the absolute positions climb — but the score number never moves. That frozen number is the point of RoPE: the model sees only relative position.
Why — one line of algebra
A 2D rotation by angle φ is the matrix R(φ). The only fact we need is the one your eyes already saw in Keypoint 2: rotating by α then by β is the same as rotating by α+β — angles just add. A consequence: undoing a rotation is rotating backward, so R(α)⊤ = R(−α).
The absolute positions m and n have vanished. The score is a function of (n − m) alone — a built-in, relative position code that the model never had to learn and that keeps working at distances it never saw during training.
Toy example — see it in numbers
Take a query and key that point the same way (identical content). With rate θ = 0.4, their score is just cos((m − n)·θ). Watch it depend only on the gap — the two highlighted rows have different absolute positions but the same gap, and land on the same score:
| m | n | gap | (gap)·θ | score = cos(·) |
|---|---|---|---|---|
| 0 | 0 | 0 | 0.00 | +1.000 |
| 3 | 4 | −1 | −0.40 | +0.921 |
| 3 | 5 | −2 | −0.80 | +0.697 |
| 10 | 12 | −2 | −0.80 | +0.697 |
| 2 | 5 | −3 | −1.20 | +0.362 |
Identical tokens score highest when they're at the same position and gently fade apart as the gap grows — a soft, automatic "nearby matters more," handed to the model for free by geometry.
Keypoint 3 — scaling up
One rotating pair is a single clock hand — fine for small gaps, but it wraps around and repeats. So RoPE splits the full vector into many 2D pairs and spins each at its own rate θ0 > θ1 > θ2 > …. Fast hands separate neighbouring tokens; slow hands keep marching to encode long distances without wrapping. Like a real clock telling seconds, minutes, and hours at once.
Slide m up: the leftmost hand whirls through many turns while the rightmost barely creeps. Read together, the set of hand-positions is a unique, multi-scale fingerprint of where the token sits — and pairwise it still resolves to pure relative angle, exactly as in Keypoint 2.
The whole mechanism in four breaths
1. Attention scores live in the angle between query and key. 2. Spin each by position × rate, and that angle becomes the position gap. 3. Algebra confirms the absolute positions cancel — only (n−m) remains. 4. Use many rates so the encoding works from one token away to thousands.
That's all of RoPE. Everything in the implementations — interleaved vs split-half pairing, partial rotation, the inverse-frequency table — is just bookkeeping for which dimensions form each pair and how fast each one spins. The geometry above is the part that actually makes attention care about position.