Neural Networks Perform Sufficient Dimension Reduction
It addresses the problem of dimension reduction in regression for statisticians and machine learning practitioners, providing a theoretical justification for neural networks in this context, but it appears incremental as it builds on existing SDR concepts.
This paper demonstrates that neural networks inherently perform sufficient dimension reduction (SDR) in regression tasks, showing that first-layer weights span the central mean subspace and establishing statistical consistency for this estimator.
This paper investigates the connection between neural networks and sufficient dimension reduction (SDR), demonstrating that neural networks inherently perform SDR in regression tasks under appropriate rank regularizations. Specifically, the weights in the first layer span the central mean subspace. We establish the statistical consistency of the neural network-based estimator for the central mean subspace, underscoring the suitability of neural networks in addressing SDR-related challenges. Numerical experiments further validate our theoretical findings, and highlight the underlying capability of neural networks to facilitate SDR compared to the existing methods. Additionally, we discuss an extension to unravel the central subspace, broadening the scope of our investigation.