Frozen Layers: Memory-efficient Many-fidelity Hyperparameter Optimization
This work addresses the challenge of hyperparameter optimization under compute and memory constraints for deep learning practitioners, offering an incremental improvement by introducing a new fidelity source.
The paper tackles the problem of memory-efficient hyperparameter optimization for large models by proposing frozen layers as a novel fidelity source, demonstrating significant compute and memory savings while preserving rank correlations in evaluations on ResNets and Transformers.
As model sizes grow, finding efficient and cost-effective hyperparameter optimization (HPO) methods becomes increasingly crucial for deep learning pipelines. While multi-fidelity HPO (MF-HPO) trades off computational resources required for DL training with lower fidelity estimations, existing fidelity sources often fail under lower compute and memory constraints. We propose a novel fidelity source: the number of layers that are trained or frozen during training. For deep networks, this approach offers significant compute and memory savings while preserving rank correlations between hyperparameters at low fidelities compared to full model training. We demonstrate this in our empirical evaluation across ResNets and Transformers and additionally analyze the utility of frozen layers as a fidelity in using GPU resources as a fidelity in HPO, and for a combined MF-HPO with other fidelity sources. This contribution opens new applications for MF-HPO with hardware resources as a fidelity and creates opportunities for improved algorithms navigating joint fidelity spaces.