Continual Learning of Personalized Generative Face Models with Experience Replay
This addresses the challenge of maintaining personalized face models over time for applications like digital avatars or photo editing, but it is incremental as it builds on existing continual learning and generative model techniques.
The paper tackles the problem of catastrophic forgetting in continually updating personalized generative face models as new photos are captured, and demonstrates that a novel experience replay algorithm using StyleGAN's latent space to represent the buffer as an optimal convex hull is more effective at preventing forgetting than random sampling baselines.
We introduce a novel continual learning problem: how to sequentially update the weights of a personalized 2D and 3D generative face model as new batches of photos in different appearances, styles, poses, and lighting are captured regularly. We observe that naive sequential fine-tuning of the model leads to catastrophic forgetting of past representations of the individual's face. We then demonstrate that a simple random sampling-based experience replay method is effective at mitigating catastrophic forgetting when a relatively large number of images can be stored and replayed. However, for long-term deployment of these models with relatively smaller storage, this simple random sampling-based replay technique also forgets past representations. Thus, we introduce a novel experience replay algorithm that combines random sampling with StyleGAN's latent space to represent the buffer as an optimal convex hull. We observe that our proposed convex hull-based experience replay is more effective in preventing forgetting than a random sampling baseline and the lower bound.