Large Scale Substitution-based Word Sense Induction
This addresses the problem of scalable and accurate word-sense induction for NLP applications, offering a data-driven approach that can handle large corpora and domain-specific senses, though it is incremental as it builds on existing MLM techniques.
The authors tackled word-sense induction by using pre-trained masked language models to scale efficiently, resulting in high-quality sense-tagging on English Wikipedia that outperforms methods like Babelfy and produces senseful embeddings that beat existing ones on datasets such as WiC.
We present a word-sense induction method based on pre-trained masked language models (MLMs), which can cheaply scale to large vocabularies and large corpora. The result is a corpus which is sense-tagged according to a corpus-derived sense inventory and where each sense is associated with indicative words. Evaluation on English Wikipedia that was sense-tagged using our method shows that both the induced senses, and the per-instance sense assignment, are of high quality even compared to WSD methods, such as Babelfy. Furthermore, by training a static word embeddings algorithm on the sense-tagged corpus, we obtain high-quality static senseful embeddings. These outperform existing senseful embeddings methods on the WiC dataset and on a new outlier detection dataset we developed. The data driven nature of the algorithm allows to induce corpora-specific senses, which may not appear in standard sense inventories, as we demonstrate using a case study on the scientific domain.