Highly Efficient Real-Time Streaming and Fully On-Device Speaker Diarization with Multi-Stage Clustering
This addresses the problem of efficient speaker diarization for on-device applications, such as mobile or embedded systems, where CPU, memory, and battery are limited, representing an incremental improvement focused on optimization.
The paper tackles the challenge of improving efficiency for on-device speaker diarization by proposing a multi-stage clustering strategy that uses different algorithms based on input length, achieving real-time streaming with computational complexity bounds to adapt to resource-constrained devices.
While recent research advances in speaker diarization mostly focus on improving the quality of diarization results, there is also an increasing interest in improving the efficiency of diarization systems. In this paper, we demonstrate that a multi-stage clustering strategy that uses different clustering algorithms for input of different lengths can address multi-faceted challenges of on-device speaker diarization applications. Specifically, a fallback clusterer is used to handle short-form inputs; a main clusterer is used to handle medium-length inputs; and a pre-clusterer is used to compress long-form inputs before they are processed by the main clusterer. Both the main clusterer and the pre-clusterer can be configured with an upper bound of the computational complexity to adapt to devices with different resource constraints. This multi-stage clustering strategy is critical for streaming on-device speaker diarization systems, where the budgets of CPU, memory and battery are tight.