Preserving Domain Generalization in Fine-Tuning via Joint Parameter Selection
This work addresses the challenge of maintaining domain generalization for machine learning practitioners when fine-tuning models, representing an incremental improvement over existing parameter-efficient adaptation strategies.
The paper tackles the problem of domain generalization in fine-tuning pre-trained vision models by introducing Joint Parameter Selection (JPS), a method that selectively updates a sparse subset of parameters to preserve generalization capabilities, achieving superior performance compared to state-of-the-art methods in benchmark experiments.
Domain generalization seeks to develop models trained on a limited set of source domains that are capable of generalizing effectively to unseen target domains. While the predominant approach leverages large-scale pre-trained vision models as initialization, recent studies have highlighted that full fine-tuning can compromise the intrinsic generalization capabilities of these models. To address this limitation, parameter-efficient adaptation strategies have emerged, wherein only a subset of model parameters is selectively fine-tuned, thereby balancing task adaptation with the preservation of generalization. Motivated by this paradigm, we introduce Joint Parameter Selection (JPS), a novel method that restricts updates to a small, sparse subset of parameters, thereby retaining and harnessing the generalization strength of pre-trained models. Theoretically, we establish a generalization error bound that explicitly accounts for the sparsity of parameter updates, thereby providing a principled justification for selective fine-tuning. Practically, we design a selection mechanism employing dual operators to identify and update parameters exhibiting consistent and significant gradients across all source domains. Extensive benchmark experiments demonstrate that JPS achieves superior performance compared to state-of-the-art domain generalization methods, substantiating both the efficiency and efficacy of the proposed approach.