TNTC: two-stream network with transformer-based complementarity for gait-based emotion recognition
This work addresses a specific bottleneck in gait-based emotion recognition for intelligent applications, representing an incremental improvement.
The paper tackled the problem of insufficiently exploiting complementary relationships between skeleton joint and affective features in gait-based emotion recognition by proposing a two-stream network with a transformer-based complementarity module, resulting in improved accuracy over state-of-the-art methods on the latest dataset.
Recognizing the human emotion automatically from visual characteristics plays a vital role in many intelligent applications. Recently, gait-based emotion recognition, especially gait skeletons-based characteristic, has attracted much attention, while many available methods have been proposed gradually. The popular pipeline is to first extract affective features from joint skeletons, and then aggregate the skeleton joint and affective features as the feature vector for classifying the emotion. However, the aggregation procedure of these emerged methods might be rigid, resulting in insufficiently exploiting the complementary relationship between skeleton joint and affective features. Meanwhile, the long range dependencies in both spatial and temporal domains of the gait sequence are scarcely considered. To address these issues, we propose a novel two-stream network with transformer-based complementarity, termed as TNTC. Skeleton joint and affective features are encoded into two individual images as the inputs of two streams, respectively. A new transformer-based complementarity module (TCM) is proposed to bridge the complementarity between two streams hierarchically via capturing long range dependencies. Experimental results demonstrate TNTC outperforms state-of-the-art methods on the latest dataset in terms of accuracy.