Structured Inverse-Free Natural Gradient: Memory-Efficient & Numerically-Stable KFAC
This addresses a gap for practitioners needing efficient and stable second-order optimization in modern mixed-precision training, though it is incremental as it builds on KFAC.
The paper tackled the memory inefficiency and numerical instability of second-order methods like KFAC in low-precision neural net training by proposing SINGD, which uses an inverse-free update and structured Kronecker factors, showing it outperforms AdamW in half precision.
Second-order methods such as KFAC can be useful for neural net training. However, they are often memory-inefficient since their preconditioning Kronecker factors are dense, and numerically unstable in low precision as they require matrix inversion or decomposition. These limitations render such methods unpopular for modern mixed-precision training. We address them by (i) formulating an inverse-free KFAC update and (ii) imposing structures in the Kronecker factors, resulting in structured inverse-free natural gradient descent (SINGD). On modern neural networks, we show that SINGD is memory-efficient and numerically robust, in contrast to KFAC, and often outperforms AdamW even in half precision. Our work closes a gap between first- and second-order methods in modern low-precision training.