FLiP: Towards understanding and interpreting multimodal multilingual sentence embeddings
Provides a diagnostic tool for practitioners to understand biases in sentence embeddings without downstream tasks, addressing interpretability in multimodal multilingual models.
FLiP models recover over 75% of lexical content from multilingual and multimodal sentence embeddings, outperforming non-factorized baselines, and reveal modality and language biases in encoders like LaBSE, SONAR, and Gemini.
This paper presents factorized linear projection (FLiP) models for understanding pretrained sentence embedding spaces. We train FLiP models to recover the lexical content from multilingual (LaBSE), multimodal (SONAR) and API-based (Gemini) sentence embedding spaces in several high- and mid-resource languages. We show that FLiP can recall more than 75% of lexical content from the embeddings, significantly outperforming existing non-factorized baselines. Using this as a diagnostic tool, we uncover the modality and language biases across the selected sentence encoders and provide practitioners with intrinsic insights about the encoders without relying on conventional downstream evaluation tasks. Our implementation is public https://github.com/BUTSpeechFIT/FLiP.