Dynamic Bayesian Optimization Framework for Instruction Tuning in Partial Differential Equation Discovery
This addresses the issue of suboptimal performance in equation discovery for researchers, though it is incremental as it builds on existing optimization methods.
The paper tackles the problem of instruction brittleness in LLMs for PDE discovery by proposing NeuroSymBO, which adaptively selects instructions using Bayesian Optimization, resulting in higher recovery rates and more parsimonious solutions compared to fixed prompts.
Large Language Models (LLMs) show promise for equation discovery, yet their outputs are highly sensitive to prompt phrasing, a phenomenon we term instruction brittleness. Static prompts cannot adapt to the evolving state of a multi-step generation process, causing models to plateau at suboptimal solutions. To address this, we propose NeuroSymBO, which reframes prompt engineering as a sequential decision problem. Our method maintains a discrete library of reasoning strategies and uses Bayesian Optimization to select the optimal instruction at each step based on numerical feedback. Experiments on PDE discovery benchmarks show that adaptive instruction selection significantly outperforms fixed prompts, achieving higher recovery rates with more parsimonious solutions.