Deep Loss Convexification for Learning Iterative Models
This addresses a fundamental challenge in nonconvex optimization for iterative models, benefiting applications in computer vision and machine learning, though it appears incremental as it builds on existing network architectures.
The paper tackles the problem of bad local optima in iterative methods like ICP for point cloud registration by proposing Deep Loss Convexification (DLC), which learns to reshape loss landscapes into convex-like shapes using adversarial training and star-convexity constraints, achieving state-of-the-art performance in tasks such as training RNNs, 3D point cloud registration, and multimodel image alignment.
Iterative methods such as iterative closest point (ICP) for point cloud registration often suffer from bad local optimality (e.g. saddle points), due to the nature of nonconvex optimization. To address this fundamental challenge, in this paper we propose learning to form the loss landscape of a deep iterative method w.r.t. predictions at test time into a convex-like shape locally around each ground truth given data, namely Deep Loss Convexification (DLC), thanks to the overparametrization in neural networks. To this end, we formulate our learning objective based on adversarial training by manipulating the ground-truth predictions, rather than input data. In particular, we propose using star-convexity, a family of structured nonconvex functions that are unimodal on all lines that pass through a global minimizer, as our geometric constraint for reshaping loss landscapes, leading to (1) extra novel hinge losses appended to the original loss and (2) near-optimal predictions. We demonstrate the state-of-the-art performance using DLC with existing network architectures for the tasks of training recurrent neural networks (RNNs), 3D point cloud registration, and multimodel image alignment.