Boosting with List-Decodable Codes
This work provides a theoretical breakthrough for boosting by showing that the round complexity can be reduced for concept classes with a mild closure property, which is relevant to learning theory and algorithm design.
The paper presents a new boosting algorithm that achieves strong learning using only O(log(1/ε)) calls to a weak learner for concept classes closed under O(log(1/γ))-XOR, circumventing the known lower bound of Ω(log(1/ε)/γ^2) for generic boosters. The algorithm leverages a novel connection between boosting and list-decodable codes.
Boosting is a fundamental technique for generically improving the accuracy of learning algorithms (Schapire 1989). Existing boosting algorithms construct a strong learner using $O(\log(\frac{1}ε)/γ^2)$ calls to a $γ$-advantage weak learner, and this round complexity is known to be optimal for generic boosters that succeed on all concept classes (Freund 1995). We show that this lower bound can be circumvented for concept classes that satisfy a mild closure property. Specifically, we present a new boosting algorithm that, for any class $\mathcal{F}$ closed under $O(\log \frac{1}γ)$-XOR, strong learns $\mathcal{F}$ using $O(\log \frac{1}ε)$ calls to a $γ$-advantage weak learner and a single batch of $\tilde{O}(\log(\frac{1}ε)/γ^2)$ additional samples. Our algorithm arises from a new and simple connection between boosting and list-decodable codes. Viewing the target function as a message, we run the weak learner on its encoding and view the resulting weak hypothesis as a corrupted codeword. Feeding this corrupted codeword to a list decoder, we obtain a small list of candidate hypotheses, at least one of which is a strong hypothesis for the original function. Using additional samples, we identify and output this strong hypothesis.