Sharpness-Aware Black-Box Optimization
This addresses generalization issues in black-box optimization for machine learning applications like reinforcement learning and prompt fine-tuning, representing an incremental improvement.
The paper tackles the problem of suboptimal model quality and generalization in black-box optimization by proposing a Sharpness-Aware Black-Box Optimization (SABO) algorithm, which improves generalization through sharpness-aware minimization and demonstrates effectiveness in black-box prompt fine-tuning tasks.
Black-box optimization algorithms have been widely used in various machine learning problems, including reinforcement learning and prompt fine-tuning. However, directly optimizing the training loss value, as commonly done in existing black-box optimization methods, could lead to suboptimal model quality and generalization performance. To address those problems in black-box optimization, we propose a novel Sharpness-Aware Black-box Optimization (SABO) algorithm, which applies a sharpness-aware minimization strategy to improve the model generalization. Specifically, the proposed SABO method first reparameterizes the objective function by its expectation over a Gaussian distribution. Then it iteratively updates the parameterized distribution by approximated stochastic gradients of the maximum objective value within a small neighborhood around the current solution in the Gaussian distribution space. Theoretically, we prove the convergence rate and generalization bound of the proposed SABO algorithm. Empirically, extensive experiments on the black-box prompt fine-tuning tasks demonstrate the effectiveness of the proposed SABO method in improving model generalization performance.