The 2022 NIST Language Recognition Evaluation
This evaluation provides a benchmark for language recognition technology, particularly for low-resource African languages, but is incremental as part of an ongoing series.
The 2022 NIST Language Recognition Evaluation tackled the problem of language recognition in conversational and broadcast speech, introducing new features like a focus on African languages and variable-duration test segments, with results showing that Oromo and Tigrinya were easier to detect while Xhosa and Zulu were more challenging, and performance improved with longer speech durations up to a point of diminishing returns.
In 2022, the U.S. National Institute of Standards and Technology (NIST) conducted the latest Language Recognition Evaluation (LRE) in an ongoing series administered by NIST since 1996 to foster research in language recognition and to measure state-of-the-art technology. Similar to previous LREs, LRE22 focused on conversational telephone speech (CTS) and broadcast narrowband speech (BNBS) data. LRE22 also introduced new evaluation features, such as an emphasis on African languages, including low resource languages, and a test set consisting of segments containing between 3s and 35s of speech randomly sampled and extracted from longer recordings. A total of 21 research organizations, forming 16 teams, participated in this 3-month long evaluation and made a total of 65 valid system submissions to be evaluated. This paper presents an overview of LRE22 and an analysis of system performance over different evaluation conditions. The evaluation results suggest that Oromo and Tigrinya are easier to detect while Xhosa and Zulu are more challenging. A greater confusability is seen for some language pairs. When speech duration increased, system performance significantly increased up to a certain duration, and then a diminishing return on system performance is observed afterward.