Can LLMs predict the convergence of Stochastic Gradient Descent?
This could enable zero-shot randomized trials for deep learning models, though it appears incremental as it builds on known links between SGD and Markov chains.
The paper tackled the problem of predicting where stochastic gradient descent (SGD) converges in optimization by leveraging LLMs' understanding of Markovian dynamics, showing zero-shot performance in identifying local minima for unseen starting points.
Large-language models are notoriously famous for their impressive performance across a wide range of tasks. One surprising example of such impressive performance is a recently identified capacity of LLMs to understand the governing principles of dynamical systems satisfying the Markovian property. In this paper, we seek to explore this direction further by studying the dynamics of stochastic gradient descent in convex and non-convex optimization. By leveraging the theoretical link between the SGD and Markov chains, we show a remarkable zero-shot performance of LLMs in predicting the local minima to which SGD converges for previously unseen starting points. On a more general level, we inquire about the possibility of using LLMs to perform zero-shot randomized trials for larger deep learning models used in practice.