Not a nuisance but a useful heuristic: Outlier dimensions favor frequent tokens in language models
This addresses the problem of understanding internal mechanisms in language models for researchers, but it is incremental as it builds on existing knowledge of outlier dimensions.
The paper investigates outlier dimensions in language models, showing they function as a heuristic for predicting frequent tokens, and demonstrates how models can counteract this when inappropriate by reallocating weight to other dimensions.
We study last-layer outlier dimensions, i.e. dimensions that display extreme activations for the majority of inputs. We show that outlier dimensions arise in many different modern language models, and trace their function back to the heuristic of constantly predicting frequent words. We further show how a model can block this heuristic when it is not contextually appropriate, by assigning a counterbalancing weight mass to the remaining dimensions, and we investigate which model parameters boost outlier dimensions and when they arise during training. We conclude that outlier dimensions are a specialized mechanism discovered by many distinct models to implement a useful token prediction heuristic.