Composing Open-domain Vision with RAG for Ocean Monitoring and Conservation
This work addresses scalable species identification for ocean monitoring, which is crucial for communities and economies reliant on healthy oceans, though it appears incremental as it builds on existing methods.
The paper tackles the challenge of applying computer vision to ocean conservation by proposing an open-domain learning framework using pretrained vision-language models with retrieval-augmented generation, demonstrating emergent capabilities in fish classification from video without domain-specific training.
Climate change's destruction of marine biodiversity is threatening communities and economies around the world which rely on healthy oceans for their livelihoods. The challenge of applying computer vision to niche, real-world domains such as ocean conservation lies in the dynamic and diverse environments where traditional top-down learning struggle with long-tailed distributions, generalization, and domain transfer. Scalable species identification for ocean monitoring is particularly difficult due to the need to adapt models to new environments and identify rare or unseen species. To overcome these limitations, we propose leveraging bottom-up, open-domain learning frameworks as a resilient, scalable solution for image and video analysis in marine applications. Our preliminary demonstration uses pretrained vision-language models (VLMs) combined with retrieval-augmented generation (RAG) as grounding, leaving the door open for numerous architectural, training and engineering optimizations. We validate this approach through a preliminary application in classifying fish from video onboard fishing vessels, demonstrating impressive emergent retrieval and prediction capabilities without domain-specific training or knowledge of the task itself.