Rotations and Interpretability of Word Embeddings: the Case of the Russian Language
This work addresses interpretability issues in NLP for Russian language applications, but it is incremental as it builds on existing SVD techniques.
The authors tackled the problem of improving interpretability and stability of word embeddings by applying canonical orthogonal transformations from SVD, demonstrating increased meaning and stability of components for Russian language models like RusVectores, fastText, and RDT.
Consider a continuous word embedding model. Usually, the cosines between word vectors are used as a measure of similarity of words. These cosines do not change under orthogonal transformations of the embedding space. We demonstrate that, using some canonical orthogonal transformations from SVD, it is possible both to increase the meaning of some components and to make the components more stable under re-learning. We study the interpretability of components for publicly available models for the Russian language (RusVectores, fastText, RDT).