SenseBERT: Driving Some Sense into BERT
This addresses the problem of lexical-semantic understanding in natural language processing for researchers and practitioners, representing a novel method rather than an incremental improvement.
The paper tackles the limitation of existing self-supervision techniques operating at the word form level by proposing SenseBERT, a model pre-trained to predict masked words and their WordNet supersenses without human annotation. It achieves significantly improved lexical understanding, attaining state-of-the-art results on the Word in Context task.
The ability to learn from large unlabeled corpora has allowed neural language models to advance the frontier in natural language understanding. However, existing self-supervision techniques operate at the word form level, which serves as a surrogate for the underlying semantic content. This paper proposes a method to employ weak-supervision directly at the word sense level. Our model, named SenseBERT, is pre-trained to predict not only the masked words but also their WordNet supersenses. Accordingly, we attain a lexical-semantic level language model, without the use of human annotation. SenseBERT achieves significantly improved lexical understanding, as we demonstrate by experimenting on SemEval Word Sense Disambiguation, and by attaining a state of the art result on the Word in Context task.