iBERT: Interpretable Style Embeddings via Sense Decomposition
This work addresses the need for interpretable embeddings in NLP, particularly for style-focused tasks, though it is incremental as it builds on existing BERT-style models with added modularity.
The authors tackled the problem of producing interpretable and controllable embeddings for language by introducing iBERT, which decomposes tokens into sparse, non-negative sense vectors, resulting in an ~8-point improvement in style representation effectiveness on the STEL benchmark while maintaining competitive authorship verification performance.
We present iBERT (interpretable-BERT), an encoder to produce inherently interpretable and controllable embeddings - designed to modularize and expose the discriminative cues present in language, such as stylistic and semantic structure. Each input token is represented as a sparse, non-negative mixture over k context-independent sense vectors, which can be pooled into sentence embeddings or used directly at the token level. This enables modular control over representation, before any decoding or downstream use. To demonstrate our model's interpretability, we evaluate it on a suite of style-focused tasks. On the STEL benchmark, it improves style representation effectiveness by ~8 points over SBERT-style baselines, while maintaining competitive performance on authorship verification. Because each embedding is a structured composition of interpretable senses, we highlight how specific style attributes - such as emoji use, formality, or misspelling can be assigned to specific sense vectors. While our experiments center on style, iBERT is not limited to stylistic modeling. Its structural modularity is designed to interpretably decompose whichever discriminative signals are present in the data - enabling generalization even when supervision blends stylistic and semantic factors.