Input and Weight Space Smoothing for Semi-supervised Learning
This addresses semi-supervised learning for machine learning practitioners, offering a simpler alternative to data-heavy methods, though it appears incremental as it builds on known smoothing techniques.
The paper tackles the problem of semi-supervised learning by proposing a method that regularizes both input and weight spaces, showing they are complementary for representation minimality and insensitivity to nuisance variability, achieving comparable performance to state-of-the-art without heavy data augmentation.
We propose regularizing the empirical loss for semi-supervised learning by acting on both the input (data) space, and the weight (parameter) space. We show that the two are not equivalent, and in fact are complementary, one affecting the minimality of the resulting representation, the other insensitivity to nuisance variability. We propose a method to perform such smoothing, which combines known input-space smoothing with a novel weight-space smoothing, based on a min-max (adversarial) optimization. The resulting Adversarial Block Coordinate Descent (ABCD) algorithm performs gradient ascent with a small learning rate for a random subset of the weights, and standard gradient descent on the remaining weights in the same mini-batch. It achieves comparable performance to the state-of-the-art without resorting to heavy data augmentation, using a relatively simple architecture.