Generating Synthetic Datasets by Interpolating along Generalized Geodesics
This approach addresses the need for domain-specific pretraining data in machine learning, offering a novel way to generate synthetic datasets on-demand, though it is incremental in building on existing optimal transport methods.
The paper tackles the problem of synthesizing datasets for targeted pretraining by interpolating between heterogeneous datasets using generalized geodesics from optimal transport theory, resulting in a scalable and efficient method that can interpolate between datasets with unrelated labels and shows promise in transfer learning experiments.
Data for pretraining machine learning models often consists of collections of heterogeneous datasets. Although training on their union is reasonable in agnostic settings, it might be suboptimal when the target domain -- where the model will ultimately be used -- is known in advance. In that case, one would ideally pretrain only on the dataset(s) most similar to the target one. Instead of limiting this choice to those datasets already present in the pretraining collection, here we explore extending this search to all datasets that can be synthesized as `combinations' of them. We define such combinations as multi-dataset interpolations, formalized through the notion of generalized geodesics from optimal transport (OT) theory. We compute these geodesics using a recent notion of distance between labeled datasets, and derive alternative interpolation schemes based on it: using either barycentric projections or optimal transport maps, the latter computed using recent neural OT methods. These methods are scalable, efficient, and -- notably -- can be used to interpolate even between datasets with distinct and unrelated label sets. Through various experiments in transfer learning in computer vision, we demonstrate this is a promising new approach for targeted on-demand dataset synthesis.