Neural Language Priors
This work addresses the problem of understanding architectural biases in NLP for researchers, showing that prior assumptions in encoders are largely ineffective without learning, which is incremental as it builds on existing encoder analysis.
The study investigated how different sentence encoder architectures, when randomly initialized and fixed, perform as language priors on downstream tasks, finding they generally fail to leverage additional task-relevant information and that uninformative priors perform similarly to informative ones, indicating learning is essential.
The choice of sentence encoder architecture reflects assumptions about how a sentence's meaning is composed from its constituent words. We examine the contribution of these architectures by holding them randomly initialised and fixed, effectively treating them as as hand-crafted language priors, and evaluating the resulting sentence encoders on downstream language tasks. We find that even when encoders are presented with additional information that can be used to solve tasks, the corresponding priors do not leverage this information, except in an isolated case. We also find that apparently uninformative priors are just as good as seemingly informative priors on almost all tasks, indicating that learning is a necessary component to leverage information provided by architecture choice.