SensePOLAR: Word sense aware interpretability for pre-trained contextual word embeddings
This addresses a fundamental limitation for researchers and practitioners in NLP by enabling sense-aware interpretations of contextual word embeddings, though it is incremental as it builds on existing POLAR methods.
The authors tackled the problem of polysemy in interpretable word embeddings by extending the POLAR framework to create SensePOLAR, which provides word-sense aware interpretability for pre-trained contextual embeddings, achieving performance comparable to original embeddings on benchmarks like GLUE and SQuAD.
Adding interpretability to word embeddings represents an area of active research in text representation. Recent work has explored thepotential of embedding words via so-called polar dimensions (e.g. good vs. bad, correct vs. wrong). Examples of such recent approaches include SemAxis, POLAR, FrameAxis, and BiImp. Although these approaches provide interpretable dimensions for words, they have not been designed to deal with polysemy, i.e. they can not easily distinguish between different senses of words. To address this limitation, we present SensePOLAR, an extension of the original POLAR framework that enables word-sense aware interpretability for pre-trained contextual word embeddings. The resulting interpretable word embeddings achieve a level of performance that is comparable to original contextual word embeddings across a variety of natural language processing tasks including the GLUE and SQuAD benchmarks. Our work removes a fundamental limitation of existing approaches by offering users sense aware interpretations for contextual word embeddings.