Ordering Matters: Rank-Aware Selective Fusion for Blended Emotion Recognition
For researchers in affective computing, this work provides a method to improve fine-grained blended emotion recognition by selectively fusing multimodal encoders, though the improvement is incremental over existing fusion approaches.
The paper tackles blended emotion recognition by proposing a rank-aware multi-encoder framework that selectively fuses top-n informative encoders and decouples prediction into presence and salience heads. The system achieved 2nd place in the BlEmoRE challenge, outperforming individual encoders and naïve fusion baselines.
Blended emotion recognition is challenging because emotions are often expressed as mixtures of subtle and overlapping multimodal cues rather than a single dominant signal. We propose a rank-aware multi-encoder framework that selectively combines complementary representations from diverse pre-extracted video and audio encoders. Our method projects heterogeneous encoder features into a shared latent space, estimates sample-wise encoder importance through an attention-based gating module, and fuses only the top-n most informative encoders. To better model blended emotions, we decouple prediction into presence and salience heads and align them through probability-level fusion. We further incorporate feature-level unsupervised domain adaptation without pseudo-labeling to improve robustness under distribution shift. Experiments on the BlEmoRE challenge show that the proposed framework outperforms strong individual encoders and naïve multi-encoder fusion baselines. Our final system ranked 2nd in the competition, supporting the effectiveness of rank-aware selective fusion for fine-grained blended emotion recognition.