The Harder Text Embedding Benchmark (HTEB): Beyond One-dimensional Static Robustness
For practitioners and researchers evaluating text embeddings, this work challenges static benchmarks by revealing that robustness is not a single scalar property, but the findings are incremental as they extend existing evaluation methodology.
The authors argue that embedding robustness is multidimensional and propose HTEB, a dynamic evaluation framework that tests models along Lexical/Stylistic, Length, and Language axes. Evaluating 16 models on 32 datasets, they find that scale improves absolute scores but not robustness gaps, and English datasets are more sensitive to transformations.
Embedding benchmarks like MTEB report a single score per model, implicitly treating robustness as a static, scalar property. We argue that embedding robustness is multidimensional, since models respond differently to different types of variation, and requires dynamic evaluation to expose failures hidden by static benchmarks. We introduce the Harder Text Embedding Benchmark (HTEB), a dynamic evaluation framework that challenges model robustness along three practically interpretable axes (Lexical/Stylistic, Length and Language) by stochastically transforming inputs at evaluation time with an LLM. Evaluating 16 open-weight embedding models on 32 datasets covering 42 languages under transformations validated by 4,800 human ratings on an English subsample, we find three patterns: (1) Models exhibit specific, partly decoupled robustness profiles across axes. (2) Across three model families, scale increases absolute scores but does not close the gap between original and transformed evaluations. Here, scaling tends to improve specifically the Language axis. (3) English datasets are more sensitive to HTEB transformations than multilingual datasets. This demonstrates that HTEB identifies strengths and weaknesses of models along deployment-relevant axes, challenging current embedding benchmarks and arguing for multidimensional, dynamic robustness evaluation.