Discretized Gradient Flow for Manifold Learning in the Space of Embeddings
This work addresses a theoretical bottleneck in manifold learning for researchers, but it is incremental as it builds on existing gradient flow methods.
The paper tackles the problem of implementing gradient descent in the infinite-dimensional space of smooth embeddings for manifold learning, by providing an explicit lower bound for the step length to ensure embeddings remain smooth, specifically when the gradient is normal to the embedded manifold under diffeomorphism-invariant penalty functions.
Gradient descent, or negative gradient flow, is a standard technique in optimization to find minima of functions. Many implementations of gradient descent rely on discretized versions, i.e., moving in the gradient direction for a set step size, recomputing the gradient, and continuing. In this paper, we present an approach to manifold learning where gradient descent takes place in the infinite dimensional space $\mathcal{E} = {\rm Emb}(M,\mathbb{R}^N)$ of smooth embeddings $φ$ of a manifold $M$ into $\mathbb{R}^N$. Implementing a discretized version of gradient descent for $P:\mathcal{E}\to {\mathbb R}$, a penalty function that scores an embedding $φ\in \mathcal{E}$, requires estimating how far we can move in a fixed direction -- the direction of one gradient step -- before leaving the space of smooth embeddings. Our main result is to give an explicit lower bound for this step length in terms of the Riemannian geometry of $φ(M)$. In particular, we consider the case when the gradient of $P$ is pointwise normal to the embedded manifold $φ(M)$. We prove this case arises when $P$ is invariant under diffeomorphisms of $M$, a natural condition in manifold learning.