A Subword Embedding Approach for Variation Detection in Luxembourgish User Comments
This provides a reproducible framework for studying language variety in multilingual and small-language contexts, though it is incremental as it builds on existing distributional modeling techniques.
The paper tackled the problem of detecting linguistic variation in noisy, low-resource text by developing a subword embedding approach that groups related forms without prior normalization, using a corpus of Luxembourgish user comments to uncover extensive lexical and orthographic variation aligning with dialectal patterns.
This paper presents an embedding-based approach to detecting variation without relying on prior normalisation or predefined variant lists. The method trains subword embeddings on raw text and groups related forms through combined cosine and n-gram similarity. This allows spelling and morphological diversity to be examined and analysed as linguistic structure rather than treated as noise. Using a large corpus of Luxembourgish user comments, the approach uncovers extensive lexical and orthographic variation that aligns with patterns described in dialectal and sociolinguistic research. The induced families capture systematic correspondences and highlight areas of regional and stylistic differentiation. The procedure does not strictly require manual annotation, but does produce transparent clusters that support both quantitative and qualitative analysis. The results demonstrate that distributional modelling can reveal meaningful patterns of variation even in ''noisy'' or low-resource settings, offering a reproducible methodological framework for studying language variety in multilingual and small-language contexts.