Video-guided Machine Translation with Global Video Context
This work addresses the challenge of capturing global narrative context in long videos for machine translation, which is incremental as it builds on existing methods by enhancing their ability to handle broader video contexts.
The paper tackled the problem of video-guided multimodal translation by addressing the limitation of existing methods that rely on locally aligned video segments, proposing a framework that leverages global video context to improve translation accuracy in long videos, achieving significant performance improvements over baseline models.
Video-guided Multimodal Translation (VMT) has advanced significantly in recent years. However, most existing methods rely on locally aligned video segments paired one-to-one with subtitles, limiting their ability to capture global narrative context across multiple segments in long videos. To overcome this limitation, we propose a globally video-guided multimodal translation framework that leverages a pretrained semantic encoder and vector database-based subtitle retrieval to construct a context set of video segments closely related to the target subtitle semantics. An attention mechanism is employed to focus on highly relevant visual content, while preserving the remaining video features to retain broader contextual information. Furthermore, we design a region-aware cross-modal attention mechanism to enhance semantic alignment during translation. Experiments on a large-scale documentary translation dataset demonstrate that our method significantly outperforms baseline models, highlighting its effectiveness in long-video scenarios.