Expressivity of Quadratic Neural ODEs
This work provides theoretical insights into the role of depth in deep learning architectures, addressing a foundational problem for researchers in machine learning theory.
The paper derived quantitative approximation error bounds for neural ODEs with quadratic nonlinearities, showing that expressivity arises from iterative composition of simple operations rather than complex dynamics.
This work focuses on deriving quantitative approximation error bounds for neural ordinary differential equations having at most quadratic nonlinearities in the dynamics. The simple dynamics of this model form demonstrates how expressivity can be derived primarily from iteratively composing many basic elementary operations, versus from the complexity of those elementary operations themselves. Like the analog differential analyzer and universal polynomial DAEs, the expressivity is derived instead primarily from the "depth" of the model. These results contribute to our understanding of what depth specifically imparts to the capabilities of deep learning architectures.