MarineVRS: Marine Video Retrieval System with Explainability via Semantic Understanding
This addresses the problem of processing dense and repetitive marine video data for researchers and scientists, though it is incremental as it builds on existing methods with added explainability.
The authors tackled the challenge of video retrieval in marine environments by developing MarineVRS, a system that integrates visual and linguistic object representation for efficient search and includes an explainability module for object segmentation, achieving accurate and scalable analysis of underwater video data.
Building a video retrieval system that is robust and reliable, especially for the marine environment, is a challenging task due to several factors such as dealing with massive amounts of dense and repetitive data, occlusion, blurriness, low lighting conditions, and abstract queries. To address these challenges, we present MarineVRS, a novel and flexible video retrieval system designed explicitly for the marine domain. MarineVRS integrates state-of-the-art methods for visual and linguistic object representation to enable efficient and accurate search and analysis of vast volumes of underwater video data. In addition, unlike the conventional video retrieval system, which only permits users to index a collection of images or videos and search using a free-form natural language sentence, our retrieval system includes an additional Explainability module that outputs the segmentation masks of the objects that the input query referred to. This feature allows users to identify and isolate specific objects in the video footage, leading to more detailed analysis and understanding of their behavior and movements. Finally, with its adaptability, explainability, accuracy, and scalability, MarineVRS is a powerful tool for marine researchers and scientists to efficiently and accurately process vast amounts of data and gain deeper insights into the behavior and movements of marine species.