Architecture-Aware Learning Curve Extrapolation via Graph Ordinary Differential Equation
This work addresses the need for more accurate learning curve extrapolation in AutoML, particularly for hyperparameter tuning and neural architecture search, though it is incremental by building on existing dynamical system views.
The paper tackled the problem of predicting neural network performance from early training epochs by incorporating neural network architecture information into learning curve modeling, resulting in a model that outperforms state-of-the-art methods for MLP and CNN-based curves.
Learning curve extrapolation predicts neural network performance from early training epochs and has been applied to accelerate AutoML, facilitating hyperparameter tuning and neural architecture search. However, existing methods typically model the evolution of learning curves in isolation, neglecting the impact of neural network (NN) architectures, which influence the loss landscape and learning trajectories. In this work, we explore whether incorporating neural network architecture improves learning curve modeling and how to effectively integrate this architectural information. Motivated by the dynamical system view of optimization, we propose a novel architecture-aware neural differential equation model to forecast learning curves continuously. We empirically demonstrate its ability to capture the general trend of fluctuating learning curves while quantifying uncertainty through variational parameters. Our model outperforms current state-of-the-art learning curve extrapolation methods and pure time-series modeling approaches for both MLP and CNN-based learning curves. Additionally, we explore the applicability of our method in Neural Architecture Search scenarios, such as training configuration ranking.