PersEmoN: A Deep Network for Joint Analysis of Apparent Personality, Emotion and Their Relationship
This addresses the problem of integrating high-level affect traits for affective computing researchers, but it is incremental as it builds on existing individual analysis methods.
The paper tackles the joint analysis of apparent personality and emotion from face images by introducing PersEmoN, a deep Siamese-like network that uses multi-task learning and adversarial loss, achieving effectiveness in experiments.
Apparent personality and emotion analysis are both central to affective computing. Existing works solve them individually. In this paper we investigate if such high-level affect traits and their relationship can be jointly learned from face images in the wild. To this end, we introduce PersEmoN, an end-to-end trainable and deep Siamese-like network. It consists of two convolutional network branches, one for emotion and the other for apparent personality. Both networks share their bottom feature extraction module and are optimized within a multi-task learning framework. Emotion and personality networks are dedicated to their own annotated dataset. Furthermore, an adversarial-like loss function is employed to promote representation coherence among heterogeneous dataset sources. Based on this, we also explore the emotion-to-apparent-personality relationship. Extensive experiments demonstrate the effectiveness of PersEmoN.