Consistent Interpolating Ensembles via the Manifold-Hilbert Kernel
This work addresses a theoretical gap in overparametrized learning for ensemble methods, which is incremental as it extends existing results to a new class of methods.
The authors tackled the lack of generalization guarantees for ensemble methods in the overparametrized interpolating regime by devising an ensemble classification method that interpolates training data and is consistent for broad data distributions, proving weak consistency for kernel smoothing regression using a newly defined manifold-Hilbert kernel.
Recent research in the theory of overparametrized learning has sought to establish generalization guarantees in the interpolating regime. Such results have been established for a few common classes of methods, but so far not for ensemble methods. We devise an ensemble classification method that simultaneously interpolates the training data, and is consistent for a broad class of data distributions. To this end, we define the manifold-Hilbert kernel for data distributed on a Riemannian manifold. We prove that kernel smoothing regression using the manifold-Hilbert kernel is weakly consistent in the setting of Devroye et al. 1998. For the sphere, we show that the manifold-Hilbert kernel can be realized as a weighted random partition kernel, which arises as an infinite ensemble of partition-based classifiers.