Behavior of Hyper-Parameters for Selected Machine Learning Algorithms: An Empirical Investigation
This work addresses hyper-parameter tuning challenges for practitioners using common ML algorithms, but it is incremental as it builds on existing empirical studies without introducing new methods.
The paper empirically investigates how hyper-parameters affect performance for XGBoost, Random Forest, and Feedforward Neural Networks on structured data, quantifying HP importance and stability near optimal regions, and proposes guidelines to reduce search space for efficient tuning.
Hyper-parameters (HPs) are an important part of machine learning (ML) model development and can greatly influence performance. This paper studies their behavior for three algorithms: Extreme Gradient Boosting (XGB), Random Forest (RF), and Feedforward Neural Network (FFNN) with structured data. Our empirical investigation examines the qualitative behavior of model performance as the HPs vary, quantifies the importance of each HP for different ML algorithms, and stability of the performance near the optimal region. Based on the findings, we propose a set of guidelines for efficient HP tuning by reducing the search space.