Δm multiplied by the selected angular frequency.
RoPE rotates query and key vector pairs according to token position. Explore frequency bands, token distance, and context scaling, then watch relative phase change attention compatibility along a 3D helix.
Open the phase labInstead of adding a position vector, rotary embeddings multiply pairs of features by a position-dependent rotation. When a rotated query meets a rotated key, their dot product depends on relative displacement, giving attention direct access to how far apart two tokens are.
Δm multiplied by the selected angular frequency.
Contribution of this rotated feature pair to q·k.
Tokens required for one full rotation in this band.
Distance after optional context interpolation.
Two scalar features become a 2D vector. Position m rotates that vector by m times the band's angular frequency.
R(mω) [x₂ᵢ, x₂ᵢ₊₁]ᵀRotation matrices are orthogonal. The query rotation transposes against the key rotation, leaving a single rotation by their relative position.
(R(mω)q)ᵀ(R(nω)k) = qᵀR((n−m)ω)kLinear interpolation divides positions before rotation, preserving phase longer but compressing distinctions. Other methods alter frequencies nonuniformly or fine-tune on longer sequences.
m′ = m / scaleRapid rotation distinguishes nearby positions but aliases after relatively short wavelengths.
Useful for local token order.Slow rotation changes gently across long spans, retaining coarse position relationships.
Useful for long-range structure.Positions far beyond training can enter unfamiliar phase combinations even though the formula still computes.
Scaling trades resolution for range.