Evaluating Open-Source Vision Language Models for Facial Emotion Recognition against Traditional Deep Learning Models
This work provides practical insights for deploying emotion recognition systems in noisy environments, though it is incremental as it benchmarks existing models rather than proposing new ones.
This study compared open-source vision-language models (VLMs) against traditional deep learning models for facial emotion recognition on the FER-2013 dataset, finding that traditional models like EfficientNet-B0 (86.44% accuracy) significantly outperformed VLMs such as CLIP (64.07%) and Phi-3.5 Vision (51.66%).
Facial Emotion Recognition (FER) is crucial for applications such as human-computer interaction and mental health diagnostics. This study presents the first empirical comparison of open-source Vision-Language Models (VLMs), including Phi-3.5 Vision and CLIP, against traditional deep learning models VGG19, ResNet-50, and EfficientNet-B0 on the challenging FER-2013 dataset, which contains 35,887 low-resolution grayscale images across seven emotion classes. To address the mismatch between VLM training assumptions and the noisy nature of FER data, we introduce a novel pipeline that integrates GFPGAN-based image restoration with FER evaluation. Results show that traditional models, particularly EfficientNet-B0 (86.44%) and ResNet-50 (85.72%), significantly outperform VLMs like CLIP (64.07%) and Phi-3.5 Vision (51.66%), highlighting the limitations of VLMs in low-quality visual tasks. In addition to performance evaluation using precision, recall, F1-score, and accuracy, we provide a detailed computational cost analysis covering preprocessing, training, inference, and evaluation phases, offering practical insights for deployment. This work underscores the need for adapting VLMs to noisy environments and provides a reproducible benchmark for future research in emotion recognition.