Sequential Policy Gradient for Adaptive Hyperparameter Optimization
This addresses the problem of prohibitive resource requirements for hyperparameter optimization in machine learning, offering a more efficient solution for practitioners, though it appears incremental as an extension of policy gradient methods.
The paper tackles the high computational cost of reinforcement learning for hyperparameter optimization by proposing Sequential Policy Gradient (SPG), a lightweight trajectory generation method that improves model performance by +0.2–7% across five diverse datasets while reducing computational costs.
Reinforcement learning is essential for neural architecture search and hyperparameter optimization, but the conventional approaches impede widespread use due to prohibitive time and computational costs. Inspired by DeepSeek-V3 multi-token prediction architecture, we propose Sequential Policy Gradient modeling (SPG), a novel trajectory generation paradigm for lightweight online hyperparameter optimization. In contrast to conventional policy gradient methods, SPG extends the base model with temporary modules, enabling it to generate state-action (padded) trajectories in a single forward pass. Our experiments demonstrate that models gain performance when retrained with SPG on their original datasets and also outperform standard transfer fine-tuning. We evaluate on five datasets spanning computer vision (ImageNet, COCO), natural language processing (GLUE, SQuAD), and audio (SUPERB) to assess the industrial applicability of SPG. The proposed method demonstrates consistent improvements across widely adopted models, achieving performance gains of $+0.2\sim7\%$, with significantly low computational costs. Fully reproducible code and pre-trained models: https://huggingface.co/UniversalAlgorithmic/SPG.