Semantics-Consistent Cross-domain Summarization via Optimal Transport Alignment
This work addresses the need for automated, semantics-aware summarization in real-world applications like news and video platforms, but it appears incremental as it builds on existing multimodal summarization techniques.
The paper tackled the problem of generating multimodal summaries (cover images and titles) from videos and articles by addressing the neglect of structural semantics in existing methods, and the proposed SCCS model achieved high-quality results on three multimodal datasets.
Multimedia summarization with multimodal output (MSMO) is a recently explored application in language grounding. It plays an essential role in real-world applications, i.e., automatically generating cover images and titles for news articles or providing introductions to online videos. However, existing methods extract features from the whole video and article and use fusion methods to select the representative one, thus usually ignoring the critical structure and varying semantics. In this work, we propose a Semantics-Consistent Cross-domain Summarization (SCCS) model based on optimal transport alignment with visual and textual segmentation. In specific, our method first decomposes both video and article into segments in order to capture the structural semantics, respectively. Then SCCS follows a cross-domain alignment objective with optimal transport distance, which leverages multimodal interaction to match and select the visual and textual summary. We evaluated our method on three recent multimodal datasets and demonstrated the effectiveness of our method in producing high-quality multimodal summaries.