Piggyback GAN: Efficient Lifelong Learning for Image Conditioned Generation
This addresses the problem of efficient lifelong learning for AI systems in image generation, though it is incremental as it builds on existing GAN and lifelong learning methods.
The paper tackles catastrophic forgetting in deep neural networks for image-conditioned generation by proposing Piggyback GAN, a parameter-efficient framework that factorizes filters from previous tasks to learn new ones, achieving high generation quality comparable to standalone models with fewer parameters while preserving performance on old tasks.
Humans accumulate knowledge in a lifelong fashion. Modern deep neural networks, on the other hand, are susceptible to catastrophic forgetting: when adapted to perform new tasks, they often fail to preserve their performance on previously learned tasks. Given a sequence of tasks, a naive approach addressing catastrophic forgetting is to train a separate standalone model for each task, which scales the total number of parameters drastically without efficiently utilizing previous models. In contrast, we propose a parameter efficient framework, Piggyback GAN, which learns the current task by building a set of convolutional and deconvolutional filters that are factorized into filters of the models trained on previous tasks. For the current task, our model achieves high generation quality on par with a standalone model at a lower number of parameters. For previous tasks, our model can also preserve generation quality since the filters for previous tasks are not altered. We validate Piggyback GAN on various image-conditioned generation tasks across different domains, and provide qualitative and quantitative results to show that the proposed approach can address catastrophic forgetting effectively and efficiently.