Hierarchical Proxy Modeling for Improved HPO in Time Series Forecasting
This work solves hyperparameter tuning challenges for time series forecasting practitioners, offering a generalizable method that is incremental but effective across multiple datasets.
The paper tackles the problem of hyperparameter optimization (HPO) in time series forecasting by addressing test-validation mismatch, proposing H-Pro to use hierarchical data proxies for improved HPO, resulting in outperforming state-of-the-art methods on datasets like Tourism, Wiki, and Traffic, and beating the M5 competition winner.
Selecting the right set of hyperparameters is crucial in time series forecasting. The classical temporal cross-validation framework for hyperparameter optimization (HPO) often leads to poor test performance because of a possible mismatch between validation and test periods. To address this test-validation mismatch, we propose a novel technique, H-Pro to drive HPO via test proxies by exploiting data hierarchies often associated with time series datasets. Since higher-level aggregated time series often show less irregularity and better predictability as compared to the lowest-level time series which can be sparse and intermittent, we optimize the hyperparameters of the lowest-level base-forecaster by leveraging the proxy forecasts for the test period generated from the forecasters at higher levels. H-Pro can be applied on any off-the-shelf machine learning model to perform HPO. We validate the efficacy of our technique with extensive empirical evaluation on five publicly available hierarchical forecasting datasets. Our approach outperforms existing state-of-the-art methods in Tourism, Wiki, and Traffic datasets, and achieves competitive result in Tourism-L dataset, without any model-specific enhancements. Moreover, our method outperforms the winning method of the M5 forecast accuracy competition.