SparseVSR: Lightweight and Noise Robust Visual Speech Recognition
This work addresses the deployment gap for visual speech recognition in resource-constrained environments, though it is incremental as it builds on existing pruning techniques.
The paper tackled the problem of deploying visual speech recognition models on resource-constrained devices by developing a lightweight model using magnitude-based pruning, achieving state-of-the-art results at 10% sparsity on the LRS3 dataset and a 2% WER improvement under visual noise.
Recent advances in deep neural networks have achieved unprecedented success in visual speech recognition. However, there remains substantial disparity between current methods and their deployment in resource-constrained devices. In this work, we explore different magnitude-based pruning techniques to generate a lightweight model that achieves higher performance than its dense model equivalent, especially under the presence of visual noise. Our sparse models achieve state-of-the-art results at 10% sparsity on the LRS3 dataset and outperform the dense equivalent up to 70% sparsity. We evaluate our 50% sparse model on 7 different visual noise types and achieve an overall absolute improvement of more than 2% WER compared to the dense equivalent. Our results confirm that sparse networks are more resistant to noise than dense networks.