STEB: Style Text Embedding Benchmark
This benchmark standardizes evaluation of style embeddings for researchers working on stylistic text analysis.
The authors introduce STEB, a benchmark for evaluating style embeddings across 96 datasets and 7 languages, finding that semantic embeddings fail on stylistic tasks and no single style embedding is universally superior.
While semantic embeddings are rigorously evaluated on the Massive Text Embedding Benchmark, the evaluation of style embeddings remains fragmented, with each work relying on their own set of tasks and datasets. To bridge this gap, we introduce the Style Text Embedding Benchmark, a comprehensive open-source benchmark intended to standardize the evaluation of style embeddings. STEB encompasses 96 datasets across 7 languages, spanning applications such as authorship verification, authorship retrieval, AI-text detection, probing of linguistic features, and others. We find that semantic embeddings consistently fail in stylistic tasks, and that there is no style embedding that is universally superior across all tasks evaluated. We open-source the STEB code base at: https://github.com/rrivera1849/STEB.