Neural Piecewise-Constant Delay Differential Equations
This work addresses the problem of improving modeling capability in continuous-depth neural networks for machine learning and data science applications, though it appears incremental as it builds upon Neural Delay Differential Equations.
The authors introduced Neural Piecewise-Constant Delay Differential Equations (PCDDEs), a new continuous-depth neural network that transforms single delays into piecewise-constant ones to enhance modeling capability without increasing network dimension, and demonstrated that it outperforms existing frameworks on datasets like MNIST, CIFAR10, and SVHN.
Continuous-depth neural networks, such as the Neural Ordinary Differential Equations (ODEs), have aroused a great deal of interest from the communities of machine learning and data science in recent years, which bridge the connection between deep neural networks and dynamical systems. In this article, we introduce a new sort of continuous-depth neural network, called the Neural Piecewise-Constant Delay Differential Equations (PCDDEs). Here, unlike the recently proposed framework of the Neural Delay Differential Equations (DDEs), we transform the single delay into the piecewise-constant delay(s). The Neural PCDDEs with such a transformation, on one hand, inherit the strength of universal approximating capability in Neural DDEs. On the other hand, the Neural PCDDEs, leveraging the contributions of the information from the multiple previous time steps, further promote the modeling capability without augmenting the network dimension. With such a promotion, we show that the Neural PCDDEs do outperform the several existing continuous-depth neural frameworks on the one-dimensional piecewise-constant delay population dynamics and real-world datasets, including MNIST, CIFAR10, and SVHN.