Multimodal Video-based Apparent Personality Recognition Using Long Short-Term Memory and Convolutional Neural Networks
This work addresses personality recognition for applications in affective computing, but it is incremental as it builds on existing multimodal and neural network approaches.
The paper tackles the problem of recognizing the Big Five personality traits from videos by developing a multimodal system that integrates face, environment, audio, and transcription features using modality-specific neural networks and a two-stage training process. It achieves the best overall mean accuracy score on the ChaLearn First Impressions V2 challenge dataset compared to state-of-the-art methods.
Personality computing and affective computing, where the recognition of personality traits is essential, have gained increasing interest and attention in many research areas recently. We propose a novel approach to recognize the Big Five personality traits of people from videos. Personality and emotion affect the speaking style, facial expressions, body movements, and linguistic factors in social contexts, and they are affected by environmental elements. We develop a multimodal system to recognize apparent personality based on various modalities such as the face, environment, audio, and transcription features. We use modality-specific neural networks that learn to recognize the traits independently and we obtain a final prediction of apparent personality with a feature-level fusion of these networks. We employ pre-trained deep convolutional neural networks such as ResNet and VGGish networks to extract high-level features and Long Short-Term Memory networks to integrate temporal information. We train the large model consisting of modality-specific subnetworks using a two-stage training process. We first train the subnetworks separately and then fine-tune the overall model using these trained networks. We evaluate the proposed method using ChaLearn First Impressions V2 challenge dataset. Our approach obtains the best overall "mean accuracy" score, averaged over five personality traits, compared to the state-of-the-art.