Policy Gradient with Adaptive Entropy Annealing for Continual Fine-Tuning
This addresses the problem of catastrophic forgetting for practitioners adapting large pretrained vision models to new tasks, offering an incremental improvement over existing parameter-efficient fine-tuning approaches.
The paper tackles catastrophic forgetting in continual fine-tuning of vision models by proposing adaptive entropy annealing (aEPG), a method that transitions from exploratory to exploitative learning to directly minimize misclassification error, outperforming cross-entropy-based methods across diverse benchmarks.
Despite their success, large pretrained vision models remain vulnerable to catastrophic forgetting when adapted to new tasks in class-incremental settings. Parameter-efficient fine-tuning (PEFT) alleviates this by restricting trainable parameters, yet most approaches still rely on cross-entropy (CE) loss, a surrogate for the 0-1 loss, to learn from new data. We revisit this choice and revive the true objective (0-1 loss) through a reinforcement learning perspective. By formulating classification as a one-step Markov Decision Process, we derive an Expected Policy Gradient (EPG) method that directly minimizes misclassification error with a low-variance gradient estimation. Our analysis shows that CE can be interpreted as EPG with an additional sample-weighting mechanism: CE encourages exploration by emphasizing low-confidence samples, while EPG prioritizes high-confidence ones. Building on this insight, we propose adaptive entropy annealing (aEPG), a training strategy that transitions from exploratory (CE-like) to exploitative (EPG-like) learning. aEPG-based methods outperform CE-based methods across diverse benchmarks and with various PEFT modules. More broadly, we evaluate various entropy regularization methods and demonstrate that lower entropy of the output prediction distribution enhances adaptation in pretrained vision models.