HARFE: Hard-Ridge Random Feature Expansion
This work addresses the challenge of efficient function approximation in high-dimensional settings, which is incremental as it builds on existing random feature and sparse regression methods.
The authors tackled the problem of approximating high-dimensional sparse additive functions by proposing HARFE, a random feature model that combines hard-thresholding pursuit with sparse ridge regression, and demonstrated that it achieves lower or comparable error than state-of-the-art algorithms on synthetic and real datasets.
We propose a random feature model for approximating high-dimensional sparse additive functions called the hard-ridge random feature expansion method (HARFE). This method utilizes a hard-thresholding pursuit-based algorithm applied to the sparse ridge regression (SRR) problem to approximate the coefficients with respect to the random feature matrix. The SRR formulation balances between obtaining sparse models that use fewer terms in their representation and ridge-based smoothing that tend to be robust to noise and outliers. In addition, we use a random sparse connectivity pattern in the random feature matrix to match the additive function assumption. We prove that the HARFE method is guaranteed to converge with a given error bound depending on the noise and the parameters of the sparse ridge regression model. Based on numerical results on synthetic data as well as on real datasets, the HARFE approach obtains lower (or comparable) error than other state-of-the-art algorithms.