Personalizing Pre-trained Models
This work addresses the need for efficient and privacy-preserving personalization of pre-trained models for various learning tasks, though it is incremental as it builds on existing CLIP representations.
The paper tackled the problem of adapting pre-trained models like CLIP for downstream tasks such as few-shot, multi-label, and continual learning, resulting in a lightweight technique called CLIPPER that sets new benchmarks on 15 datasets with robust and competitive performance.
Self-supervised or weakly supervised models trained on large-scale datasets have shown sample-efficient transfer to diverse datasets in few-shot settings. We consider how upstream pretrained models can be leveraged for downstream few-shot, multilabel, and continual learning tasks. Our model CLIPPER (CLIP PERsonalized) uses image representations from CLIP, a large-scale image representation learning model trained using weak natural language supervision. We developed a technique, called Multi-label Weight Imprinting (MWI), for multi-label, continual, and few-shot learning, and CLIPPER uses MWI with image representations from CLIP. We evaluated CLIPPER on 10 single-label and 5 multi-label datasets. Our model shows robust and competitive performance, and we set new benchmarks for few-shot, multi-label, and continual learning. Our lightweight technique is also compute-efficient and enables privacy-preserving applications as the data is not sent to the upstream model for fine-tuning.