ALEE: Any-Language Evaluation of Embeddings via English-Centric Minimal Pairs
For NLP practitioners and researchers, ALEE provides a scalable, cross-lingual evaluation benchmark that exposes persistent weaknesses in embedding models for low-resource languages.
ALEE is a framework for evaluating text embeddings in any language by generating English minimal pairs with controlled semantic shifts via AMR and translating them into target languages. Experiments across 275+ languages reveal that embedding performance varies substantially by language, text length, and linguistic phenomenon, with gaps tied to training data prevalence and tokenization.
Text embeddings are standard for semantic similarity tasks, yet their evaluation remains an open challenge. Current benchmarks are static, cover only a limited set of languages, are often domain-specific, susceptible to overfitting, and poorly representative of low-resource languages. To address these limitations, we introduce ALEE, a framework that extends Sentence Smith (Li et al., 2025) to the cross-lingual and paragraph level. ALEE uses Abstract Meaning Representations (AMR) to generate English minimal pairs with controlled, fine-grained semantic shifts, which are paired with translations in target languages. This approach enables targeted diagnostics for models in any language with English parallel data. We conduct a large-scale empirical study across a diverse set of embedding models and 275+ languages spanning three parallel datasets. On ALEE, performance varies substantially across languages, text lengths, and linguistic phenomena, exposing persistent gaps in cross-lingual semantic representation that track language prevalence in training resources and subword tokenization. We release ALEE at https://github.com/Andrian0s/any-lang-embed-eval