IDIOLEX: Unified and Continuous Representations for Idiolectal and Stylistic Variation
This addresses the need for models sensitive to stylistic differences, such as in developing diverse and accessible LLMs, though it is incremental as it builds on existing representation learning methods.
The paper tackles the problem of learning sentence representations that capture style and dialect separately from semantic content, introducing IDIOLEX, a framework that combines supervision from sentence provenance and linguistic features. The results show that the learned representations capture meaningful variation and transfer across domains for analysis and classification, with applications in stylistically aligning language models.
Existing sentence representations primarily encode what a sentence says, rather than how it is expressed, even though the latter is important for many applications. In contrast, we develop sentence representations that capture style and dialect, decoupled from semantic content. We call this the task of idiolectal representation learning. We introduce IDIOLEX, a framework for training models that combines supervision from a sentence's provenance with linguistic features of a sentence's content, to learn a continuous representation of each sentence's style and dialect. We evaluate the approach on dialects of both Arabic and Spanish. The learned representations capture meaningful variation and transfer across domains for analysis and classification. We further explore the use of these representations as training objectives for stylistically aligning language models. Our results suggest that jointly modeling individual and community-level variation provides a useful perspective for studying idiolect and supports downstream applications requiring sensitivity to stylistic differences, such as developing diverse and accessible LLMs.