LITTA: Late-Interaction and Test-Time Alignment for Visually-Grounded Multimodal Retrieval
This work addresses the problem of evidence retrieval from complex multimodal documents for users in fields like education and industry, but it is incremental as it builds on existing retrieval methods without retraining.
The paper tackled the challenge of retrieving relevant pages from visually rich documents with long context and weak lexical overlap by proposing LITTA, a query-expansion-centric retrieval framework that uses a large language model to generate query variants and aggregates results through reciprocal rank fusion. The result showed significant improvements in top-k accuracy, recall, and MRR across domains like computer science and pharmaceuticals, with gains particularly large in high-variability domains.
Retrieving relevant evidence from visually rich documents such as textbooks, technical reports, and manuals is challenging due to long context, complex layouts, and weak lexical overlap between user questions and supporting pages. We propose LITTA, a query-expansion-centric retrieval framework for evidence page retrieval that improves multimodal document retrieval without retriever retraining. Given a user query, LITTA generates complementary query variants using a large language model and retrieves candidate pages for each variant using a frozen vision retriever with late-interaction scoring. Candidates from expanded queries are then aggregated through reciprocal rank fusion to improve evidence coverage and reduce sensitivity to any single phrasing. This simple test-time strategy significantly improves retrieval robustness while remaining compatible with existing multimodal embedding indices. We evaluate LITTA on visually grounded document retrieval tasks across three domains: computer science, pharmaceuticals, and industrial manuals. Multi-query retrieval consistently improves top-k accuracy, recall, and MRR compared to single-query retrieval, with particularly large gains in domains with high visual and semantic variability. Moreover, the accuracy-efficiency trade-off is directly controllable by the number of query variants, making LITTA practical for deployment under latency constraints. These results demonstrate that query expansion provides a simple yet effective mechanism for improving visually grounded multimodal retrieval.