FLEURS-ASL: Including American Sign Language in Massively Multilingual Multitask Evaluation
It addresses the problem of excluding sign languages from mainstream AI evaluation for the deaf community and researchers, representing an incremental step by extending existing benchmarks.
The paper tackles the lack of sign language integration in machine translation by introducing FLEURS-ASL, a benchmark for American Sign Language (ASL) translation to 200 text or 102 speech languages, with baselines showing competitive performance using a unified model with timestamp tokens and context windows.
Sign language translation has historically been peripheral to mainstream machine translation research. In order to help converge the fields, we introduce FLEURS-ASL, an extension of the multiway parallel benchmarks FLORES (for text) and FLEURS (for speech) to support their first sign language (as video), American Sign Language, translated by 5 Certified Deaf Interpreters. FLEURS-ASL can be used to evaluate a variety of tasks -- primarily sentence- and discourse-level translation -- between ASL and 200 other languages as text, or 102 languages as speech. We provide baselines for tasks from ASL to English text using a unified modeling approach that incorporates timestamp tokens and previous text tokens in a 34-second context window, trained on random video clips from YouTube-ASL. This model meets or exceeds the performance of phrase-level baselines while supporting a multitude of new tasks. We also use FLEURS-ASL to show that multimodal frontier models have virtually no understanding of ASL, underscoring the importance of including sign languages in standard evaluation suites.