The Morgan-Pitman Test of Equality of Variances and its Application to Machine Learning Model Evaluation and Selection
This provides a reliable tool for model evaluation and selection in diverse contexts, addressing a specific statistical limitation in machine learning.
The authors tackled the problem of model selection in non-linear models by proposing a statistical test for equality of variances in forecasting errors, demonstrating its effectiveness through simulations and real-world applications.
Model selection in non-linear models often prioritizes performance metrics over statistical tests, limiting the ability to account for sampling variability. We propose the use of a statistical test to assess the equality of variances in forecasting errors. The test builds upon the classic Morgan-Pitman approach, incorporating enhancements to ensure robustness against data with heavy-tailed distributions or outliers with high variance, plus a strategy to make residuals from machine learning models statistically independent. Through a series of simulations and real-world data applications, we demonstrate the test's effectiveness and practical utility, offering a reliable tool for model evaluation and selection in diverse contexts.