Low-Rank Curvature for Zeroth-Order Optimization in LLM Fine-Tuning
This addresses the challenge of efficient fine-tuning for large language models, particularly in memory-constrained settings, though it is incremental as it builds on existing zeroth-order optimization techniques.
The paper tackled the problem of high variance and suboptimal search directions in zeroth-order optimization for fine-tuning large language models by introducing LOREN, a curvature-aware method that outperformed state-of-the-art ZO methods with higher accuracy, faster convergence, and up to 27.3% lower peak memory usage compared to MeZO-Adam.
We introduce LOREN, a curvature-aware zeroth-order (ZO) optimization method for fine-tuning large language models (LLMs). Existing ZO methods, which estimate gradients via finite differences using random perturbations, often suffer from high variance and suboptimal search directions. Our approach addresses these challenges by: (i) reformulating the problem of gradient preconditioning as that of adaptively estimating an anisotropic perturbation distribution for gradient estimation, (ii) capturing curvature through a low-rank block diagonal preconditioner using the framework of natural evolution strategies, and (iii) applying a REINFORCE leave-one-out (RLOO) gradient estimator to reduce variance. Experiments on standard LLM benchmarks show that our method outperforms state-of-the-art ZO methods by achieving higher accuracy and faster convergence, while cutting peak memory usage by up to 27.3% compared with MeZO-Adam.