ViDove: A Translation Agent System with Multimodal Context and Memory-Augmented Reasoning
This work addresses the problem of improving translation quality for multimodal inputs, such as video subtitling, by integrating visual and contextual information, though it is incremental as it builds on existing LLM-based translation agents.
The paper tackles the limitation of text-only inputs in LLM-based translation agents by introducing ViDove, a system that uses multimodal context and memory-augmented reasoning to enhance translation, achieving a 28% improvement in BLEU scores and a 15% improvement in SubER over previous state-of-the-art baselines.
LLM-based translation agents have achieved highly human-like translation results and are capable of handling longer and more complex contexts with greater efficiency. However, they are typically limited to text-only inputs. In this paper, we introduce ViDove, a translation agent system designed for multimodal input. Inspired by the workflow of human translators, ViDove leverages visual and contextual background information to enhance the translation process. Additionally, we integrate a multimodal memory system and long-short term memory modules enriched with domain-specific knowledge, enabling the agent to perform more accurately and adaptively in real-world scenarios. As a result, ViDove achieves significantly higher translation quality in both subtitle generation and general translation tasks, with a 28% improvement in BLEU scores and a 15% improvement in SubER compared to previous state-of-the-art baselines. Moreover, we introduce DoveBench, a new benchmark for long-form automatic video subtitling and translation, featuring 17 hours of high-quality, human-annotated data. Our code is available here: https://github.com/pigeonai-org/ViDove