Efficient Labelling of Affective Video Datasets via Few-Shot & Multi-Task Contrastive Learning
This addresses the tedious and error-prone task of manual labeling for emotion prediction in videos, offering an efficient automated solution for researchers and practitioners in affective computing.
The paper tackles the problem of labeling affective video datasets by proposing MT-CLAR, a few-shot learning method that uses multi-task contrastive learning to infer valence and arousal from facial expressions, achieving results comparable to state-of-the-art with a support-set only about 6% the size of the video dataset.
Whilst deep learning techniques have achieved excellent emotion prediction, they still require large amounts of labelled training data, which are (a) onerous and tedious to compile, and (b) prone to errors and biases. We propose Multi-Task Contrastive Learning for Affect Representation (\textbf{MT-CLAR}) for few-shot affect inference. MT-CLAR combines multi-task learning with a Siamese network trained via contrastive learning to infer from a pair of expressive facial images (a) the (dis)similarity between the facial expressions, and (b) the difference in valence and arousal levels of the two faces. We further extend the image-based MT-CLAR framework for automated video labelling where, given one or a few labelled video frames (termed \textit{support-set}), MT-CLAR labels the remainder of the video for valence and arousal. Experiments are performed on the AFEW-VA dataset with multiple support-set configurations; moreover, supervised learning on representations learnt via MT-CLAR are used for valence, arousal and categorical emotion prediction on the AffectNet and AFEW-VA datasets. The results show that valence and arousal predictions via MT-CLAR are very comparable to the state-of-the-art (SOTA), and we significantly outperform SOTA with a support-set $\approx$6\% the size of the video dataset.