Who Needs Labels? Adapting Vision Foundation Models With the Metadata You Already Have
For practitioners in scientific domains with scarce labels, this method enables effective model adaptation using readily available metadata, reducing reliance on expensive annotations.
The authors propose FINO, a label-free method that uses metadata to adapt vision foundation models to specialized scientific domains, outperforming both unsupervised and fully supervised adaptation across four domains without task labels.
We propose a label-free approach to adapt powerful but generic vision foundation models to specialized scientific domains. Standard supervised fine-tuning is often ill-suited to these settings: labels are scarce, and task-specific training can collapse the model's generality and hurt robustness. We instead leverage metadata to adapt representations to new domains in a self-supervised manner. Our method, FINO, combines a standard self-supervised objective with flexible metadata guidance that handles both highly granular discrete metadata and continuous metadata. It encourages the representation to preserve informative factors while suppressing spurious ones. Across subcellular fluorescence microscopy, Earth observation, wildlife monitoring, and medical imaging, FINO consistently outperforms standard unsupervised domain adaptation and fully supervised adaptation. It also exceeds highly-specialized domain-specific state of the art, while using no task labels for backbone adaptation and only lightweight probes for supervision.