Locally Near Optimal Piecewise Linear Regression in High Dimensions via Difference of Max-Affine Functions
For high-dimensional regression problems requiring piecewise linear models, this work provides a theoretically grounded algorithm with optimal sample complexity, though it is an incremental improvement over existing max-affine methods.
This paper introduces the Adaptive Block Gradient Descent (ABGD) algorithm for piecewise linear regression, parameterizing functions as difference of max-affine (DoMA) functions. The algorithm achieves linear convergence with $ ilde{\mathcal{O}}(d\max(\sigma_z/\epsilon,1)^2)$ samples, which is minimax optimal up to log factors, and exact recovery with $ ilde{\mathcal{O}}(d)$ samples in the noiseless case.
This paper presents a parametric solution to piecewise linear regression through the Adaptive Block Gradient Descent (ABGD) algorithm. The heart of the method is the parametrization of piecewise linear functions as the difference of max-affine (DoMA) functions. A non-asymptotic local convergence analysis for ABGD is provided under sub-Gaussian covariate and noise distributions. To initialize ABGD, we adapt a prior algorithm originally developed for the simpler setting of max-affine functions. When suitably initialized, ABGD converges linearly to an $ε$-accurate estimate given $\tilde{\mathcal{O}}(d\max(σ_z/ε,1)^2)$ observations where $σ_z^2$ denotes the noise variance. This implies exact recovery given $\tilde{\mathcal{O}}(d)$ samples in the noiseless case. Also, such a rate is shown to be minimax optimal up to logarithmic factors. Synthetic numerical results corroborate the theoretical guarantees for ABGD. We also observe competitive performance compared to the state-of-the-art methods on real-world datasets.