Infinite-dimensional Folded-in-time Deep Neural Networks
This work provides a more mathematically rigorous and flexible framework for physically implemented neural networks, which could benefit researchers working on hardware-efficient AI.
This paper generalizes a recently introduced deep neural network architecture that uses a single nonlinear element and delayed feedback. The generalization allows for weights to be described by Lebesgue integrable functions, offering greater flexibility and enabling a functional back-propagation algorithm for gradient descent training.
The method recently introduced in arXiv:2011.10115 realizes a deep neural network with just a single nonlinear element and delayed feedback. It is applicable for the description of physically implemented neural networks. In this work, we present an infinite-dimensional generalization, which allows for a more rigorous mathematical analysis and a higher flexibility in choosing the weight functions. Precisely speaking, the weights are described by Lebesgue integrable functions instead of step functions. We also provide a functional back-propagation algorithm, which enables gradient descent training of the weights. In addition, with a slight modification, our concept realizes recurrent neural networks.