Learn Like The Pro: Norms from Theory to Size Neural Computation
This work addresses the challenge of designing efficient neural architectures for applications needing to emulate dynamical systems, offering a theoretical, data-free approach that is incremental in nature.
The paper tackles the problem of optimally sizing neural networks by deriving structural bounds from dynamical systems theory, without requiring training data. It provides exact sizing for networks with multiplicative nodes and tight lower bounds for classical feed-forward networks, as validated by simulations.
The optimal design of neural networks is a critical problem in many applications. Here, we investigate how dynamical systems with polynomial nonlinearities can inform the design of neural systems that seek to emulate them. We propose a Learnability metric and its associated features to quantify the near-equilibrium behavior of learning dynamics. Equating the Learnability of neural systems with equivalent parameter estimation metric of the reference system establishes bounds on network structure. In this way, norms from theory provide a good first guess for neural structure, which may then further adapt with data. The proposed approach neither requires training nor training data. It reveals exact sizing for a class of neural networks with multiplicative nodes that mimic continuous- or discrete-time polynomial dynamics. It also provides relatively tight lower size bounds for classical feed-forward networks that is consistent with simulated assessments.