Deep Neural Networks for Rotation-Invariance Approximation and Learning
This addresses a foundational issue in neural network design for handling rotational invariance, though it appears incremental as it builds on existing tree architectures.
The paper tackles the problem of approximating rotation-invariant functions in high-dimensional spaces using deep neural networks, showing that deep nets achieve near-optimal rates while shallow nets cannot.
Based on the tree architecture, the objective of this paper is to design deep neural networks with two or more hidden layers (called deep nets) for realization of radial functions so as to enable rotational invariance for near-optimal function approximation in an arbitrarily high dimensional Euclidian space. It is shown that deep nets have much better performance than shallow nets (with only one hidden layer) in terms of approximation accuracy and learning capabilities. In particular, for learning radial functions, it is shown that near-optimal rate can be achieved by deep nets but not by shallow nets. Our results illustrate the necessity of depth in neural network design for realization of rotation-invariance target functions.