Proximal Mapping for Deep Regularization
This addresses the challenge of effective regularization in deep learning for applications like adversarial robustness and multimodal correlations, offering a direct optimization approach.
The paper tackles the problem of regularizing hidden layer outputs in deep learning by proposing a proximal mapping layer that directly produces well-regularized outputs, resulting in novel algorithms for robust temporal learning and multiview modeling that outperform state-of-the-art methods.
Underpinning the success of deep learning is effective regularizations that allow a variety of priors in data to be modeled. For example, robustness to adversarial perturbations, and correlations between multiple modalities. However, most regularizers are specified in terms of hidden layer outputs, which are not themselves optimization variables. In contrast to prevalent methods that optimize them indirectly through model weights, we propose inserting proximal mapping as a new layer to the deep network, which directly and explicitly produces well regularized hidden layer outputs. The resulting technique is shown well connected to kernel warping and dropout, and novel algorithms were developed for robust temporal learning and multiview modeling, both outperforming state-of-the-art methods.