How good are variational autoencoders at transfer learning?
This addresses a practical issue for researchers and practitioners using VAEs in domains like music generation or medical image analysis, but it is incremental as it builds on existing methods for analysis.
The paper tackled the problem of assessing which components of variational autoencoders (VAEs) to retrain for transfer learning and whether it is likely to help, showing through representational similarity analysis that encoders' representations are generic but decoders' specific.
Variational autoencoders (VAEs) are used for transfer learning across various research domains such as music generation or medical image analysis. However, there is no principled way to assess before transfer which components to retrain or whether transfer learning is likely to help on a target task. We propose to explore this question through the lens of representational similarity. Specifically, using Centred Kernel Alignment (CKA) to evaluate the similarity of VAEs trained on different datasets, we show that encoders' representations are generic but decoders' specific. Based on these insights, we discuss the implications for selecting which components of a VAE to retrain and propose a method to visually assess whether transfer learning is likely to help on classification tasks.