Emotion Detection in Speech Using Lightweight and Transformer-Based Models: A Comparative and Ablation Study
This work addresses the problem of real-time speech emotion recognition for edge devices, offering an incremental improvement by benchmarking and optimizing lightweight models.
The paper tackled emotion recognition from speech by comparing lightweight transformer models DistilHuBERT and PaSST on the CREMA-D dataset, finding that DistilHuBERT achieved superior accuracy of 70.64% and F1 score of 70.36% with a very small model size of 0.02 MB, outperforming PaSST and a CNN-LSTM baseline.
Emotion recognition from speech plays a vital role in the development of empathetic human-computer interaction systems. This paper presents a comparative analysis of lightweight transformer-based models, DistilHuBERT and PaSST, by classifying six core emotions from the CREMA-D dataset. We benchmark their performance against a traditional CNN-LSTM baseline model using MFCC features. DistilHuBERT demonstrates superior accuracy (70.64%) and F1 score (70.36%) while maintaining an exceptionally small model size (0.02 MB), outperforming both PaSST and the baseline. Furthermore, we conducted an ablation study on three variants of the PaSST, Linear, MLP, and Attentive Pooling heads, to understand the effect of classification head architecture on model performance. Our results indicate that PaSST with an MLP head yields the best performance among its variants but still falls short of DistilHuBERT. Among the emotion classes, angry is consistently the most accurately detected, while disgust remains the most challenging. These findings suggest that lightweight transformers like DistilHuBERT offer a compelling solution for real-time speech emotion recognition on edge devices. The code is available at: https://github.com/luckymaduabuchi/Emotion-detection-.