Framing RNN as a kernel method: A neural ODE approach
This work offers theoretical insights for researchers in machine learning, though it is incremental as it builds on existing neural ODE interpretations.
The paper connects recurrent neural networks (RNNs) to kernel methods by interpreting them as neural ODEs and showing their solutions are linear functions of input signatures, providing theoretical guarantees on generalization and stability.
Building on the interpretation of a recurrent neural network (RNN) as a continuous-time neural differential equation, we show, under appropriate conditions, that the solution of a RNN can be viewed as a linear function of a specific feature set of the input sequence, known as the signature. This connection allows us to frame a RNN as a kernel method in a suitable reproducing kernel Hilbert space. As a consequence, we obtain theoretical guarantees on generalization and stability for a large class of recurrent networks. Our results are illustrated on simulated datasets.