From Text to Trajectories: GPT-2 as an ODE Solver via In-Context
It provides insights into the mechanisms of in-context learning for NLP and its potential for solving nonlinear numerical problems, though it is incremental in applying existing methods to a new domain.
This paper investigates whether large language models (LLMs) can solve ordinary differential equations (ODEs) using in-context learning, finding that GPT-2 achieves convergence comparable to or better than the Euler method and shows exponential accuracy gains with more demonstrations.
In-Context Learning (ICL) has emerged as a new paradigm in large language models (LLMs), enabling them to perform novel tasks by conditioning on a few examples embedded in the prompt. Yet, the highly nonlinear behavior of ICL for NLP tasks remains poorly understood. To shed light on its underlying mechanisms, this paper investigates whether LLMs can solve ordinary differential equations (ODEs) under the ICL setting. We formulate standard ODE problems and their solutions as sequential prompts and evaluate GPT-2 models on these tasks. Experiments on two types of ODEs show that GPT-2 can effectively learn a meta-ODE algorithm, with convergence behavior comparable to, or better than, the Euler method, and achieve exponential accuracy gains with increasing numbers of demonstrations. Moreover, the model generalizes to out-of-distribution (OOD) problems, demonstrating robust extrapolation capabilities. These empirical findings provide new insights into the mechanisms of ICL in NLP and its potential for solving nonlinear numerical problems.