A Multimodal Transformer for Live Streaming Highlight Prediction
This work addresses the challenge of real-time highlight detection for live streaming platforms, which is incremental as it builds on existing multimodal methods but adapts them to the constraints of live content.
The paper tackles the problem of predicting highlights in live streaming by developing a multimodal transformer that processes images, audio, and text comments without future frames, and it outperforms strong baselines in experiments on real-world and public datasets.
Recently, live streaming platforms have gained immense popularity. Traditional video highlight detection mainly focuses on visual features and utilizes both past and future content for prediction. However, live streaming requires models to infer without future frames and process complex multimodal interactions, including images, audio and text comments. To address these issues, we propose a multimodal transformer that incorporates historical look-back windows. We introduce a novel Modality Temporal Alignment Module to handle the temporal shift of cross-modal signals. Additionally, using existing datasets with limited manual annotations is insufficient for live streaming whose topics are constantly updated and changed. Therefore, we propose a novel Border-aware Pairwise Loss to learn from a large-scale dataset and utilize user implicit feedback as a weak supervision signal. Extensive experiments show our model outperforms various strong baselines on both real-world scenarios and public datasets. And we will release our dataset and code to better assess this topic.