YuriiFormer: A Suite of Nesterov-Accelerated Transformers
This work addresses performance improvements for transformer models, offering a novel optimization-theoretic perspective that yields practical gains, though it is incremental in applying classical ideas to a known architecture.
The authors tackled the problem of improving transformer performance by interpreting transformer layers as optimization algorithm iterations, leading to the development of a Nesterov-accelerated transformer that outperformed a nanoGPT baseline on TinyStories and OpenWebText.
We propose a variational framework that interprets transformer layers as iterations of an optimization algorithm acting on token embeddings. In this view, self-attention implements a gradient step of an interaction energy, while MLP layers correspond to gradient updates of a potential energy. Standard GPT-style transformers emerge as vanilla gradient descent on the resulting composite objective, implemented via Lie--Trotter splitting between these two energy functionals. This perspective enables principled architectural design using classical optimization ideas. As a proof of concept, we introduce a Nesterov-style accelerated transformer that preserves the same attention and MLP oracles. The resulting architecture consistently outperforms a nanoGPT baseline on TinyStories and OpenWebText, demonstrating that optimization-theoretic insights can translate into practical gains.