Sinkhorn Transformations for Single-Query Postprocessing in Text-Video Retrieval
This work addresses the need for efficient postprocessing in multimodal retrieval, particularly for scenarios with limited test queries, offering incremental improvements over existing methods.
The paper tackles the problem of postprocessing in text-video retrieval by introducing a Sinkhorn transformation method that outperforms the dual-softmax loss, achieving a new state-of-the-art on several datasets in both full-test-set and single-query settings.
A recent trend in multimodal retrieval is related to postprocessing test set results via the dual-softmax loss (DSL). While this approach can bring significant improvements, it usually presumes that an entire matrix of test samples is available as DSL input. This work introduces a new postprocessing approach based on Sinkhorn transformations that outperforms DSL. Further, we propose a new postprocessing setting that does not require access to multiple test queries. We show that our approach can significantly improve the results of state of the art models such as CLIP4Clip, BLIP, X-CLIP, and DRL, thus achieving a new state-of-the-art on several standard text-video retrieval datasets both with access to the entire test set and in the single-query setting.