STAB: Speech Tokenizer Assessment Benchmark
This work addresses the need for scalable evaluation of speech tokenizers in AI, though it is incremental as it builds on existing tokenizer methods by offering a new benchmark.
The authors tackled the problem of evaluating speech tokenizers for downstream tasks by introducing STAB, a systematic benchmark that assesses tokenizer characteristics and correlates them with task performance, providing a standardized resource for comparative analysis.
Representing speech as discrete tokens provides a framework for transforming speech into a format that closely resembles text, thus enabling the use of speech as an input to the widely successful large language models (LLMs). Currently, while several speech tokenizers have been proposed, there is ambiguity regarding the properties that are desired from a tokenizer for specific downstream tasks and its overall generalizability. Evaluating the performance of tokenizers across different downstream tasks is a computationally intensive effort that poses challenges for scalability. To circumvent this requirement, we present STAB (Speech Tokenizer Assessment Benchmark), a systematic evaluation framework designed to assess speech tokenizers comprehensively and shed light on their inherent characteristics. This framework provides a deeper understanding of the underlying mechanisms of speech tokenization, thereby offering a valuable resource for expediting the advancement of future tokenizer models and enabling comparative analysis using a standardized benchmark. We evaluate the STAB metrics and correlate this with downstream task performance across a range of speech tasks and tokenizer choices.