On The Distribution of Penultimate Activations of Classification Networks
This work addresses the challenge of domain shift in knowledge distillation and enhances generative modeling for image synthesis, though it appears incremental as it builds on existing classification and generative frameworks.
The paper tackles the problem of understanding the distribution of penultimate activations in classification networks, showing that training with cross-entropy loss leads to a generative model parameterized by the final layer weights. This model enables stable knowledge distillation under domain shift and facilitates class-conditional image generation with VAEs and GANs.
This paper studies probability distributions of penultimate activations of classification networks. We show that, when a classification network is trained with the cross-entropy loss, its final classification layer forms a Generative-Discriminative pair with a generative classifier based on a specific distribution of penultimate activations. More importantly, the distribution is parameterized by the weights of the final fully-connected layer, and can be considered as a generative model that synthesizes the penultimate activations without feeding input data. We empirically demonstrate that this generative model enables stable knowledge distillation in the presence of domain shift, and can transfer knowledge from a classifier to variational autoencoders and generative adversarial networks for class-conditional image generation.