Gradient descent in higher codimension
This is an incremental study that addresses a theoretical problem in optimization for researchers in machine learning and mathematics.
The paper investigates the behavior of noisy gradient descent in settings where minima have higher codimension, building on prior work focused on codimension 1, and reports observations from computer experiments.
We consider the behavior of gradient flow and of discrete and noisy gradient descent. It is commonly noted that the addition of noise to the process of discrete gradient descent can affect the trajectory of gradient descent. In previous work, we observed such effects. There, we considered the case where the minima had codimension 1. In this note, we do some computer experiments and observe the behavior of noisy gradient descent in the more complex setting of minima of higher codimension.