MinBackProp -- Backpropagating through Minimal Solvers
This addresses the instability and inefficiency in training networks that rely on minimal solvers, such as in 3D point registration and image matching, though it is incremental as it builds on existing implicit differentiation methods.
The paper tackles the problem of backpropagating through minimal problem solvers in neural network training by using the Implicit Function Theorem, resulting in 100% stability and a 10x speed improvement over autograd and Deep Declarative Networks.
We present an approach to backpropagating through minimal problem solvers in end-to-end neural network training. Traditional methods relying on manually constructed formulas, finite differences, and autograd are laborious, approximate, and unstable for complex minimal problem solvers. We show that using the Implicit function theorem (IFT) to calculate derivatives to backpropagate through the solution of a minimal problem solver is simple, fast, and stable. We compare our approach to (i) using the standard autograd on minimal problem solvers and relate it to existing backpropagation formulas through SVD-based and Eig-based solvers and (ii) implementing the backprop with an existing PyTorch Deep Declarative Networks (DDN) framework. We demonstrate our technique on a toy example of training outlier-rejection weights for 3D point registration and on a real application of training an outlier-rejection and RANSAC sampling network in image matching. Our method provides $100\%$ stability and is 10 times faster compared to autograd, which is unstable and slow, and compared to DDN, which is stable but also slow.