DIET-CP: Lightweight and Data Efficient Self Supervised Continued Pretraining
This addresses the problem of data-efficient domain adaptation for specialized domains, offering a practical solution for scenarios with small datasets, though it is incremental as it builds on existing continued pretraining methods.
The paper tackles the challenge of adapting foundation models to new domains with limited data by proposing DIET-CP, a lightweight continued pretraining strategy that requires no labels and minimal hyperparameters, achieving significant performance improvements for models like DINOv3 using only 1000 images.
Continued pretraining offers a promising solution for adapting foundation models to a new target domain. However, in specialized domains, available datasets are often very small, limiting the applicability of SSL methods developed for large-scale pretraining and making hyperparameter search infeasible. In addition, pretrained models are usually released as backbone-weights only, lacking important information to continue pretraining. We propose to bridge this gap with DIET-CP, a simple continued pretraining strategy, where any strong foundation model can be steered towards the new data distribution of interest. DIET-CP relies on a very simple objective, requires no labels, and introduces no more hyperparameters than supervised finetuning. It is stable across data modalities and backbone choices, while providing a significant performance boost for state-of-the-art models such as DINOv3 using only 1000 images.