Eliminating all bad Local Minima from Loss Landscapes without even adding an Extra Unit
This addresses the issue of optimization challenges in machine learning by providing a theoretical method to simplify loss landscapes, though it appears incremental as it builds on prior work.
The paper tackles the problem of eliminating all bad local minima from loss landscapes, showing that this can be achieved without adding extra units, as long as the global minimum has zero loss, by converting bad local minima into minima at infinity.
Recent work has noted that all bad local minima can be removed from neural network loss landscapes, by adding a single unit with a particular parameterization. We show that the core technique from these papers can be used to remove all bad local minima from any loss landscape, so long as the global minimum has a loss of zero. This procedure does not require the addition of auxiliary units, or even that the loss be associated with a neural network. The method of action involves all bad local minima being converted into bad (non-local) minima at infinity in terms of auxiliary parameters.