Deriving Transformer Architectures as Implicit Multinomial Regression
This provides a theoretical foundation for attention in transformers, addressing a core problem in machine learning interpretability.
The paper tackles the lack of rigorous mathematical justification for attention mechanisms by showing that optimizing latent features in a fixed multinomial regression setting yields solutions aligning with attention block dynamics, interpreting transformer representation evolution as a trajectory recovering optimal features for classification.
While attention has been empirically shown to improve model performance, it lacks a rigorous mathematical justification. This short paper establishes a novel connection between attention mechanisms and multinomial regression. Specifically, we show that in a fixed multinomial regression setting, optimizing over latent features yields solutions that align with the dynamics induced on features by attention blocks. In other words, the evolution of representations through a transformer can be interpreted as a trajectory that recovers the optimal features for classification.