LiveChat: Video Comment Generation from Audio-Visual Multimodal Contexts
This addresses the need for AI agents to engage in live video interactions, though it is incremental as it builds on existing multimodal generation with a new dataset and model.
The paper tackled the problem of generating live comments for videos by creating a large-scale audio-visual multimodal dialogue dataset from Twitch, with 438 hours of video and 3.2 million comments, and proposed a model that effectively generates comments aligned with temporal and spatial events.
Live commenting on video, a popular feature of live streaming platforms, enables viewers to engage with the content and share their comments, reactions, opinions, or questions with the streamer or other viewers while watching the video or live stream. It presents a challenging testbed for AI agents, which involves the simultaneous understanding of audio-visual multimodal contexts from live streams and the ability to interact with human viewers through dialogue. As existing live streaming-based comments datasets contain limited categories and lack a diversity, we create a large-scale audio-visual multimodal dialogue dataset to facilitate the development of live commenting technologies. The data is collected from Twitch, with 11 different categories and 575 streamers for a total of 438 hours of video and 3.2 million comments. Moreover, we propose a novel multimodal generation model capable of generating live comments that align with the temporal and spatial events within the video, as well as with the ongoing multimodal dialogue context. Our initial results have demonstrated the effectiveness of the proposed model, providing a robust foundation for further research and practical applications in the field of live video interaction.