Locally Adaptive and Differentiable Regression
This addresses a methodological gap in modern machine learning for researchers and practitioners using locally-adaptive models, though it appears incremental as it builds on existing local modeling techniques.
The authors tackled the problem of maintaining continuity and differentiability in over-parameterized, locally-adaptive regression models by proposing a framework that combines locally learned models into a global continuous and differentiable model. They demonstrated that mixing kernel ridge and polynomial regression terms in local models leads to faster statistical convergence and improved performance in practical settings.
Over-parameterized models like deep nets and random forests have become very popular in machine learning. However, the natural goals of continuity and differentiability, common in regression models, are now often ignored in modern overparametrized, locally-adaptive models. We propose a general framework to construct a global continuous and differentiable model based on a weighted average of locally learned models in corresponding local regions. This model is competitive in dealing with data with different densities or scales of function values in different local regions. We demonstrate that when we mix kernel ridge and polynomial regression terms in the local models, and stitch them together continuously, we achieve faster statistical convergence in theory and improved performance in various practical settings.