Universality of Gradient Descent Neural Network Training
This provides theoretical orientation on meta-learning possibilities, but it is incremental as it focuses on a universality result not intended for practical computations.
The paper tackles the problem of whether neural networks can always be redesigned to train well with gradient descent, proving that if any algorithm can find good weights for a classification task, there exists an extension of the network that reproduces these weights and outputs via gradient descent.
It has been observed that design choices of neural networks are often crucial for their successful optimization. In this article, we therefore discuss the question if it is always possible to redesign a neural network so that it trains well with gradient descent. This yields the following universality result: If, for a given network, there is any algorithm that can find good network weights for a classification task, then there exists an extension of this network that reproduces these weights and the corresponding forward output by mere gradient descent training. The construction is not intended for practical computations, but it provides some orientation on the possibilities of meta-learning and related approaches.