Contrastive Learning for Lifted Networks
This work addresses the problem of improving training efficiency for lifted networks, which are useful for parallel hardware and energy modeling, but it appears incremental as it builds on prior methods.
The paper tackles the limitations of existing training methods for lifted networks by introducing a contrastive loss, showing that this approach approximates back-propagation and outperforms the standard training objective for lifted networks.
In this work we address supervised learning of neural networks via lifted network formulations. Lifted networks are interesting because they allow training on massively parallel hardware and assign energy models to discriminatively trained neural networks. We demonstrate that the training methods for lifted networks proposed in the literature have significant limitations and show how to use a contrastive loss to address those limitations. We demonstrate that this contrastive training approximates back-propagation in theory and in practice and that it is superior to the training objective regularly used for lifted networks.