Multimodal Joint Emotion and Game Context Recognition in League of Legends Livestreams
This work addresses the challenge of analyzing player behavior in uncontrolled livestreams for gaming and affective computing communities, but it is incremental as it builds on existing fusion techniques.
The authors tackled the problem of jointly recognizing player emotion and game context in League of Legends livestreams by creating the first annotated dataset and proposing a tensor decomposition method for multimodal fusion, achieving improved performance over baseline fusion approaches.
Video game streaming provides the viewer with a rich set of audio-visual data, conveying information both with regards to the game itself, through game footage and audio, as well as the streamer's emotional state and behaviour via webcam footage and audio. Analysing player behaviour and discovering correlations with game context is crucial for modelling and understanding important aspects of livestreams, but comes with a significant set of challenges - such as fusing multimodal data captured by different sensors in uncontrolled ('in-the-wild') conditions. Firstly, we present, to our knowledge, the first data set of League of Legends livestreams, annotated for both streamer affect and game context. Secondly, we propose a method that exploits tensor decompositions for high-order fusion of multimodal representations. The proposed method is evaluated on the problem of jointly predicting game context and player affect, compared with a set of baseline fusion approaches such as late and early fusion.