LiFT: Local Search via Linear Programming for Overfitting-Controlled Transformers
For practitioners fine-tuning large language models, LiFT offers a principled method to reduce overfitting without extensive hyperparameter tuning, though it is an incremental improvement over existing regularization techniques.
LiFT introduces a linear programming-based local search framework for fine-tuning transformers that jointly updates parameters and regularization hyperparameters to control overfitting. On GPT-2 Small fine-tuned on WikiText-2, it achieves consistent test perplexity improvements, especially in overfitting-prone scenarios.
This paper proposes a Linear Programming (LP)-based local search framework for fine-tuning pretrained transformer models with explicit control against overfitting. The approach formulates transformer fine-tuning as a bilevel optimization-based regularization problem, in which model parameters and regularization hyperparameters are jointly updated. Information collected during initial warm-up iterations, including validation gradients and training Hessian information, is used to construct a local descent direction by solving an LP that minimizes a scaled directional derivative while preserving training optimality. This validation-aware descent direction enables focused local updates of both parameters and regularization hyperparameters, reducing overfitting without requiring repeated full retraining cycles. The resulting method, termed Linear Programming-based Fine-Tuning (LiFT) for transformers, differs from conventional fine-tuning by systematically identifying task-specific updates rather than relying on heuristic or grid-based hyperparameter selection. Experiments on GPT-2 Small fine-tuned on WikiText-2 demonstrate that LiFT enables effective adaptation through selective tuning of transformer blocks and regularization parameters, yielding consistent improvements in test perplexity across multiple layer configurations and regularization settings, with particularly pronounced gains in overfitting-prone scenarios. Beyond empirical performance, LiFT establishes a principled connection between transformer fine-tuning, bilevel optimization, local search, and regularization theory.