Bifocal Attention: Harmonizing Geometric and Spectral Positional Embeddings for Algorithmic Generalization
This addresses a specific limitation in positional encodings for large language models, particularly for tasks involving recursive logic, but appears incremental as it builds on existing RoPE methods.
The paper tackles the problem of standard Rotary Positional Embeddings (RoPE) failing to capture long-range, periodic structures in recursive logic and algorithmic reasoning, leading to poor generalization on deeper recursive steps, and introduces Bifocal Attention with a Spectral Evolution training protocol to address this, resulting in improved algorithmic generalization.
Rotary Positional Embeddings (RoPE) have become the standard for Large Language Models (LLMs) due to their ability to encode relative positions through geometric rotation. However, we identify a significant limitation we term ''Spectral Rigidity'': standard RoPE utilizes a fixed geometric decay ($θ^{-i}$) optimized for local syntactic coherence, which fails to capture the long-range, periodic structures inherent in recursive logic and algorithmic reasoning. This results in a ''Structure Gap'', where models trained on shallow reasoning chains fail to extrapolate to deeper recursive steps. In this work, we introduce Bifocal Attention, an architectural paradigm that decouples positional encoding into two distinct modalities: Geometric Eyes (Standard RoPE) for precise token-level manipulation, and Spectral Eyes (Learnable Harmonic Operators) for tracking long-range recursive depth. We propose a novel training protocol, Spectral Evolution, which initializes positional frequencies as static geometric parameters but allows them to evolve via gradient descent into a harmonic basis optimized for the specific algorithmic topology of the task.