Skip to content
conftrace_
2026 EMNLP EMNLP 2026

The Harder Text Embedding Benchmark (HTEB): Beyond One-dimensional Static Robustness

AbstractEmbedding benchmarks like MTEB report a single score per model, implicitly treating robustness as a static, scalar property. We argue that robustness is multidimensional, since models respond differently to different types of variation, and requires dynamic evaluation to expose failures hidden by static benchmarks. We introduce the Harder Text Embedding Benchmark (HTEB), a dynamic evaluation framework that challenges model robustness along three practically interpretable axes (Lexical/Stylistic, Length and Language) by stochastically transforming inputs at evaluation time with an LLM. Evaluating 16 open-weight embedding models on 32 datasets covering 42 languages under transformations validated by 4,800 individual human ratings on an English subsample, supplemented by a Spanish-source evaluation and an exploratory STS-B study of pair-level label preservation, we find three patterns: (1) Models exhibit partly decoupled robustness profiles across axes. (2) Across three model families, scale increases absolute scores but does not close the gap between original and transformed evaluations. Scaling tends to improve specifically the Language axis. (3) English datasets are more sensitive to HTEB transformations than multilingual datasets. This demonstrates that HTEB identifies strengths and weaknesses of models along deployment-relevant axes, challenging embedding benchmarks and arguing for multidimensional, dynamic robustness evaluation. We make HTEB’s code publicly available.

More from EMNLP 2026

All EMNLP 2026 papers →