Towards Optimization and Model Selection for Domain Generalization: A Mixup-guided Solution
This work solves domain generalization issues for machine learning models facing distribution shifts, but it is incremental as it builds on existing methods.
The paper tackles the problem of domain generalization by addressing optimization and model selection challenges, proposing Mixup-guided techniques that improve existing algorithms and achieve new state-of-the-art results.
The distribution shifts between training and test data typically undermine the performance of models. In recent years, lots of work pays attention to domain generalization (DG) where distribution shifts exist, and target data are unseen. Despite the progress in algorithm design, two foundational factors have long been ignored: 1) the optimization for regularization-based objectives, and 2) the model selection for DG since no knowledge about the target domain can be utilized. In this paper, we propose Mixup guided optimization and selection techniques for DG. For optimization, we utilize an adapted Mixup to generate an out-of-distribution dataset that can guide the preference direction and optimize with Pareto optimization. For model selection, we generate a validation dataset with a closer distance to the target distribution, and thereby it can better represent the target data. We also present some theoretical insights behind our proposals. Comprehensive experiments demonstrate that our model optimization and selection techniques can largely improve the performance of existing domain generalization algorithms and even achieve new state-of-the-art results.