Deep Regression Ensembles
This addresses the computational bottleneck for researchers and practitioners in machine learning, offering a more efficient alternative to traditional DNNs, though it appears incremental as it builds on existing random feature and regression methods.
The paper tackles the problem of high computational cost in deep neural networks by introducing Deep Regression Ensembles (DRE), which uses random input weights and myopically trained output weights via linear ridge regression, achieving performance comparable to or better than state-of-the-art DNNs on many datasets while reducing computational cost by orders of magnitude.
We introduce a methodology for designing and training deep neural networks (DNN) that we call "Deep Regression Ensembles" (DRE). It bridges the gap between DNN and two-layer neural networks trained with random feature regression. Each layer of DRE has two components, randomly drawn input weights and output weights trained myopically (as if the final output layer) using linear ridge regression. Within a layer, each neuron uses a different subset of inputs and a different ridge penalty, constituting an ensemble of random feature ridge regressions. Our experiments show that a single DRE architecture is at par with or exceeds state-of-the-art DNN in many data sets. Yet, because DRE neural weights are either known in closed-form or randomly drawn, its computational cost is orders of magnitude smaller than DNN.