Non-discriminative data or weak model? On the relative importance of data and model resolution
This work addresses a fundamental trade-off in neural network design for computer vision, offering a novel architecture that improves efficiency without sacrificing accuracy.
The paper investigates the impact of input versus internal resolution on neural network performance, finding that internal resolution is the key driver of model quality up to a point. It introduces Isometric Neural Networks, which maintain fixed internal resolution, achieving high accuracy with low activation footprint and parameter count.
We explore the question of how the resolution of the input image ("input resolution") affects the performance of a neural network when compared to the resolution of the hidden layers ("internal resolution"). Adjusting these characteristics is frequently used as a hyperparameter providing a trade-off between model performance and accuracy. An intuitive interpretation is that the reduced information content in the low-resolution input causes decay in the accuracy. In this paper, we show that up to a point, the input resolution alone plays little role in the network performance, and it is the internal resolution that is the critical driver of model quality. We then build on these insights to develop novel neural network architectures that we call \emph{Isometric Neural Networks}. These models maintain a fixed internal resolution throughout their entire depth. We demonstrate that they lead to high accuracy models with low activation footprint and parameter count.