AUTOMATA: Gradient Based Data Subset Selection for Compute-Efficient Hyper-parameter Tuning
This addresses the computational inefficiency of hyper-parameter tuning for practitioners, though it is incremental as it builds on existing subset selection methods.
The paper tackles the problem of time-consuming hyper-parameter optimization in deep neural networks by using informative data subsets for training runs, achieving speedups of 3x to 30x while maintaining comparable performance to using the full dataset.
Deep neural networks have seen great success in recent years; however, training a deep model is often challenging as its performance heavily depends on the hyper-parameters used. In addition, finding the optimal hyper-parameter configuration, even with state-of-the-art (SOTA) hyper-parameter optimization (HPO) algorithms, can be time-consuming, requiring multiple training runs over the entire dataset for different possible sets of hyper-parameters. Our central insight is that using an informative subset of the dataset for model training runs involved in hyper-parameter optimization, allows us to find the optimal hyper-parameter configuration significantly faster. In this work, we propose AUTOMATA, a gradient-based subset selection framework for hyper-parameter tuning. We empirically evaluate the effectiveness of AUTOMATA in hyper-parameter tuning through several experiments on real-world datasets in the text, vision, and tabular domains. Our experiments show that using gradient-based data subsets for hyper-parameter tuning achieves significantly faster turnaround times and speedups of 3$\times$-30$\times$ while achieving comparable performance to the hyper-parameters found using the entire dataset.