Learning Operators with Stochastic Gradient Descent in General Hilbert Spaces
This work addresses the problem of operator learning for researchers in machine learning and functional analysis, providing theoretical foundations but is incremental as it builds on existing SGD frameworks.
The study tackles learning operators between general Hilbert spaces using stochastic gradient descent (SGD), establishing convergence rate bounds and minimax lower bounds under regularity conditions, with results extending to nonlinear operators and refining existing literature in kernel-based settings.
This study investigates leveraging stochastic gradient descent (SGD) to learn operators between general Hilbert spaces. We propose weak and strong regularity conditions for the target operator to depict its intrinsic structure and complexity. Under these conditions, we establish upper bounds for convergence rates of the SGD algorithm and conduct a minimax lower bound analysis, further illustrating that our convergence analysis and regularity conditions quantitatively characterize the tractability of solving operator learning problems using the SGD algorithm. It is crucial to highlight that our convergence analysis is still valid for nonlinear operator learning. We show that the SGD estimator will converge to the best linear approximation of the nonlinear target operator. Moreover, applying our analysis to operator learning problems based on vector-valued and real-valued reproducing kernel Hilbert spaces yields new convergence results, thereby refining the conclusions of existing literature.