The Impact of Feature Scaling In Machine Learning: Effects on Regression and Classification Tasks
It provides model-specific guidance for practitioners on selecting optimal feature scaling techniques, addressing a critical gap in ML workflows, though it is incremental as it systematically evaluates existing methods.
This research tackled the lack of comprehensive studies on feature scaling by evaluating 12 scaling techniques across 14 ML algorithms and 16 datasets, finding that ensemble methods like Random Forest and XGBoost are robust to scaling, while models such as Logistic Regression and SVMs show significant performance variations dependent on the scaler.
This research addresses the critical lack of comprehensive studies on feature scaling by systematically evaluating 12 scaling techniques - including several less common transformations - across 14 different Machine Learning algorithms and 16 datasets for classification and regression tasks. We meticulously analyzed impacts on predictive performance (using metrics such as accuracy, MAE, MSE, and $R^2$) and computational costs (training time, inference time, and memory usage). Key findings reveal that while ensemble methods (such as Random Forest and gradient boosting models like XGBoost, CatBoost and LightGBM) demonstrate robust performance largely independent of scaling, other widely used models such as Logistic Regression, SVMs, TabNet, and MLPs show significant performance variations highly dependent on the chosen scaler. This extensive empirical analysis, with all source code, experimental results, and model parameters made publicly available to ensure complete transparency and reproducibility, offers model-specific crucial guidance to practitioners on the need for an optimal selection of feature scaling techniques.