VibeVoice-ASR-BitNet Technical Report
This work addresses the need for efficient ASR on edge CPUs, offering a practical compression technique for real-time deployment.
VibeVoice-ASR-BitNet applies heterogeneous quantization (INT8 and ternary weights) to compress an ASR model for edge CPU inference, achieving real-time recognition (RTF<1) with 3 threads and 1.6-2.3x speedup over Whisper.cpp at ~1.6 GB model size, with modest accuracy loss.
We present VibeVoice-ASR-BitNet, a compressed variant of VibeVoice-ASR optimized for real-time inference on edge CPUs. We apply heterogeneous quantization tailored to the computational characteristics of each stage: the VAE acoustic tokenizer uses full-pipeline INT8 quantization (I8_S) with kernel fusion and SIMD optimization, while the autoregressive language model adopts BitNet-style ternary weights (I2_S). To preserve accuracy under aggressive compression, we employ a progressive quantization-aware training strategy. For inference, we implement custom SIMD kernels and fused operators within the ggml framework targeting both ARM and x86 platforms, achieving real-time recognition with RTF < 1 using as few as 3 CPU threads. VibeVoice-ASR-BitNet is 1.6-2.3x faster than Whisper.cpp at comparable model sizes (~1.6 GB), with only modest accuracy degradation compared to the FP16 baseline.