Efficient Neural Task Adaptation by Maximum Entropy Initialization
This addresses a bottleneck in transfer learning for tasks with few examples, offering a robust and architecture-independent improvement, though it appears incremental as it builds on existing fine-tuning methods.
The paper tackles the problem of knowledge contamination in neural network fine-tuning for transfer learning, proposing a method that maximizes output entropy initially and stalls the first back-propagation error, resulting in outperforming traditional fine-tuning and speeding up convergence.
Transferring knowledge from one neural network to another has been shown to be helpful for learning tasks with few training examples. Prevailing fine-tuning methods could potentially contaminate pre-trained features by comparably high energy random noise. This noise is mainly delivered from a careless replacement of task-specific parameters. We analyze theoretically such knowledge contamination for classification tasks and propose a practical and easy to apply method to trap and minimize the contaminant. In our approach, the entropy of the output estimates gets maximized initially and the first back-propagated error is stalled at the output of the last layer. Our proposed method not only outperforms the traditional fine-tuning, but also significantly speeds up the convergence of the learner. It is robust to randomness and independent of the choice of architecture. Overall, our experiments show that the power of transfer learning has been substantially underestimated so far.