Wider Networks Learn Better Features
This addresses the problem of efficient transfer learning for practitioners by showing that wider networks can improve feature reuse, though it is incremental in understanding neural network design.
The paper investigates how network width affects feature learning and transferability, finding that wider networks learn more informative features despite similar test accuracy, leading to significant performance gains when fine-tuning on new tasks.
Transferability of learned features between tasks can massively reduce the cost of training a neural network on a novel task. We investigate the effect of network width on learned features using activation atlases --- a visualization technique that captures features the entire hidden state responds to, as opposed to individual neurons alone. We find that, while individual neurons do not learn interpretable features in wide networks, groups of neurons do. In addition, the hidden state of a wide network contains more information about the inputs than that of a narrow network trained to the same test accuracy. Inspired by this observation, we show that when fine-tuning the last layer of a network on a new task, performance improves significantly as the width of the network is increased, even though test accuracy on the original task is independent of width.