Contrastive Language-Image Learning with Augmented Textual Prompts for 3D/4D FER Using Vision-Language Model
This work addresses facial emotion recognition for applications in human-computer interaction, but it appears incremental as it builds on existing vision-language models with specific enhancements.
The paper tackles the problem of understanding facial emotions from 3D/4D data by introducing AffectVLM, a vision-language model that integrates multiviews and uses augmented textual prompts, achieving superior performance across multiple benchmarks.
In this paper, we introduce AffectVLM, a vision-language model designed to integrate multiviews for a semantically rich and visually comprehensive understanding of facial emotions from 3D/4D data. To effectively capture visual features, we propose a joint representation learning framework paired with a novel gradient-friendly loss function that accelerates model convergence towards optimal feature representation. Additionally, we introduce augmented textual prompts to enhance the model's linguistic capabilities and employ mixed view augmentation to expand the visual dataset. We also develop a Streamlit app for a real-time interactive inference and enable the model for distributed learning. Extensive experiments validate the superior performance of AffectVLM across multiple benchmarks.