RoVE: Rotary Value Embeddings Attention for Relative Position-dependent Value Pathways
For LLM practitioners, RoVE offers a simple, parameter-free improvement to RoPE that enhances long-range dependency handling.
RoVE modifies Rotary Position Embeddings to make value vectors position-sensitive, unifying attention and convolution. It achieves consistent gains over RoPE in few-shot learning, OOD perplexity, and long-context retrieval on 124M and 354M GPT-2 models.
Rotary Position Embeddings (RoPE) make attention scores position-relative but leave the value pathway position-blind: the message sent by a value token is the same regardless of its distance from the query. We propose RoVE, a parameter-free modification that makes values position-sensitive by rotating them simultaneously with keys, and show that it turns RoPE attention into attentive convolution. This new perspective unifies several independent formulations of the same operation across computer vision, robotics, and modern LLM architectures. Trained 124M and 354M GPT-2 models show consistent empirical gains over RoPE on few-shot in-context learning, out-of-distribution perplexity, and long-context retrieval, with the clearest improvements on tasks that require long-range aggregation.