On the Neural Tangent Kernel of Equilibrium Models
This provides theoretical insights for researchers working on infinite-depth neural network architectures, though it is incremental as it builds on existing DEQ and NTK frameworks.
The paper tackles the problem of understanding the neural tangent kernel (NTK) in deep equilibrium models, showing that unlike fully-connected networks, these models have a deterministic NTK even when width and depth go to infinity simultaneously, and it can be computed efficiently via root-finding.
This work studies the neural tangent kernel (NTK) of the deep equilibrium (DEQ) model, a practical ``infinite-depth'' architecture which directly computes the infinite-depth limit of a weight-tied network via root-finding. Even though the NTK of a fully-connected neural network can be stochastic if its width and depth both tend to infinity simultaneously, we show that contrarily a DEQ model still enjoys a deterministic NTK despite its width and depth going to infinity at the same time under mild conditions. Moreover, this deterministic NTK can be found efficiently via root-finding.