JFB: Jacobian-Free Backpropagation for Implicit Networks
This addresses a key bottleneck for researchers and practitioners using implicit networks, offering a more efficient training method without sacrificing performance.
The paper tackles the high computational cost of backpropagation in implicit networks by proposing Jacobian-Free Backpropagation (JFB), which eliminates the need to solve Jacobian-based equations, resulting in faster training and easier implementation while maintaining competitive test accuracy with feedforward and prior implicit networks.
A promising trend in deep learning replaces traditional feedforward networks with implicit networks. Unlike traditional networks, implicit networks solve a fixed point equation to compute inferences. Solving for the fixed point varies in complexity, depending on provided data and an error tolerance. Importantly, implicit networks may be trained with fixed memory costs in stark contrast to feedforward networks, whose memory requirements scale linearly with depth. However, there is no free lunch -- backpropagation through implicit networks often requires solving a costly Jacobian-based equation arising from the implicit function theorem. We propose Jacobian-Free Backpropagation (JFB), a fixed-memory approach that circumvents the need to solve Jacobian-based equations. JFB makes implicit networks faster to train and significantly easier to implement, without sacrificing test accuracy. Our experiments show implicit networks trained with JFB are competitive with feedforward networks and prior implicit networks given the same number of parameters.