The Foes of Neural Network's Data Efficiency Among Unnecessary Input Dimensions
This addresses a data efficiency problem for deep learning practitioners, but it is incremental as it focuses on a specific aspect of input layer robustness.
The study investigated how unnecessary input dimensions, such as background in object recognition, degrade the data efficiency of deep neural networks, showing that task-unrelated dimensions substantially increase the amount of examples needed to achieve generalization performance.
Datasets often contain input dimensions that are unnecessary to predict the output label, e.g. background in object recognition, which lead to more trainable parameters. Deep Neural Networks (DNNs) are robust to increasing the number of parameters in the hidden layers, but it is unclear whether this holds true for the input layer. In this letter, we investigate the impact of unnecessary input dimensions on a central issue of DNNs: their data efficiency, ie. the amount of examples needed to achieve certain generalization performance. Our results show that unnecessary input dimensions that are task-unrelated substantially degrade data efficiency. This highlights the need for mechanisms that remove {task-unrelated} dimensions to enable data efficiency gains.