Language Through a Prism: A Spectral Approach for Multiscale Language Representations
This addresses the challenge of multiscale representation learning in NLP, with potential applications in other domains like images and audio, but it is incremental as it builds on existing models like BERT.
The paper tackled the problem of capturing multiscale language structure in deep models by using spectral filters on neuron activations, resulting in filtered embeddings that excel at specific tasks like part-of-speech tagging and topic classification, and a BERT + Prism model that improves masked token prediction and performance on utterance- and document-level tasks.
Language exhibits structure at different scales, ranging from subwords to words, sentences, paragraphs, and documents. To what extent do deep models capture information at these scales, and can we force them to better capture structure across this hierarchy? We approach this question by focusing on individual neurons, analyzing the behavior of their activations at different timescales. We show that signal processing provides a natural framework for separating structure across scales, enabling us to 1) disentangle scale-specific information in existing embeddings and 2) train models to learn more about particular scales. Concretely, we apply spectral filters to the activations of a neuron across an input, producing filtered embeddings that perform well on part of speech tagging (word-level), dialog speech acts classification (utterance-level), or topic classification (document-level), while performing poorly on the other tasks. We also present a prism layer for training models, which uses spectral filters to constrain different neurons to model structure at different scales. Our proposed BERT + Prism model can better predict masked tokens using long-range context and produces multiscale representations that perform better at utterance- and document-level tasks. Our methods are general and readily applicable to other domains besides language, such as images, audio, and video.