Backbone Augmented Training for Adaptations
This addresses data scarcity in adaptation training for large models like diffusion and transformers, though it appears incremental as it builds on existing adaptation techniques.
The paper tackles the problem of limited adaptation data for large backbone models by proposing Backbone Augmented Training (BAT), which leverages backbone pre-training data to augment adaptation datasets, and demonstrates its effectiveness in personalization and language generation tasks with scarce data.
Adaptations facilitate efficient training of large backbone models, including diffusion models for image generation and transformer-based language models. While various adaptation techniques enhance performance with minimal computational resources, limited adaptation data often leads to challenges in training. To address this, we focus on the enormous amount of backbone data used to pre-train the backbone models. We propose Backbone Augmented Training (BAT), a method that leverages backbone data to augment the adaptation dataset. First, we formulate and prove two mathematical key propositions: one establishes the validity of BAT, while the other identifies a condition under which BAT benefits adaptation. Furthermore, we introduce an advanced data selection scheme that satisfies these propositions and present ALBAT algorithm to implement this approach. ALBAT efficiently enhances adaptation training in both personalization and language generation tasks with scarce data.