Mitigating Frequency Bias and Anisotropy in Language Model Pre-Training with Syntactic Smoothing
This addresses a fundamental issue in language model pre-training that affects generalization and representation quality, though it is incremental as it builds on known problems with maximum likelihood training.
The paper tackled the problem of language models' reliance on token frequency, which leads to poor generalization on infrequent tokens and anisotropy in representations, by introducing Syntactic Smoothing to adjust the training objective, resulting in better performance on infrequent English tokens and reduced anisotropy.
Language models strongly rely on frequency information because they maximize the likelihood of tokens during pre-training. As a consequence, language models tend to not generalize well to tokens that are seldom seen during training. Moreover, maximum likelihood training has been discovered to give rise to anisotropy: representations of tokens in a model tend to cluster tightly in a high-dimensional cone, rather than spreading out over their representational capacity. Our work introduces a method for quantifying the frequency bias of a language model by assessing sentence-level perplexity with respect to token-level frequency. We then present a method for reducing the frequency bias of a language model by inducing a syntactic prior over token representations during pre-training. Our Syntactic Smoothing method adjusts the maximum likelihood objective function to distribute the learning signal to syntactically similar tokens. This approach results in better performance on infrequent English tokens and a decrease in anisotropy. We empirically show that the degree of anisotropy in a model correlates with its frequency bias.