EigenNoise: A Contrastive Prior to Warm-Start Representations
This is an incremental improvement for natural language processing researchers, offering a competitive initialization scheme that may reduce data dependency.
The paper tackles the problem of initializing word vectors without pre-training data by proposing EigenNoise, a method based on a dense, independent co-occurrence model, and shows it can approach GloVe's performance as measured by MDL probing.
In this work, we present a naive initialization scheme for word vectors based on a dense, independent co-occurrence model and provide preliminary results that suggest it is competitive and warrants further investigation. Specifically, we demonstrate through information-theoretic minimum description length (MDL) probing that our model, EigenNoise, can approach the performance of empirically trained GloVe despite the lack of any pre-training data (in the case of EigenNoise). We present these preliminary results with interest to set the stage for further investigations into how this competitive initialization works without pre-training data, as well as to invite the exploration of more intelligent initialization schemes informed by the theory of harmonic linguistic structure. Our application of this theory likewise contributes a novel (and effective) interpretation of recent discoveries which have elucidated the underlying distributional information that linguistic representations capture from data and contrast distributions.