Efficient Convolutional Auto-Encoding via Random Convexification and Frequency-Domain Minimization
This addresses the limitation of reliance on labeled data and specialized hardware for new applications, though it is incremental as it builds on existing auto-encoding methods.
The paper tackled the problem of inefficient unsupervised training of deep convolutional neural networks on conventional hardware by proposing a strategy that randomly convexifies the reconstruction contractive auto-encoding objective and solves it in the frequency domain via coordinate descent, resulting in linear scaling with image size, filters, and filter size in worst-case scenarios.
The omnipresence of deep learning architectures such as deep convolutional neural networks (CNN)s is fueled by the synergistic combination of ever-increasing labeled datasets and specialized hardware. Despite the indisputable success, the reliance on huge amounts of labeled data and specialized hardware can be a limiting factor when approaching new applications. To help alleviating these limitations, we propose an efficient learning strategy for layer-wise unsupervised training of deep CNNs on conventional hardware in acceptable time. Our proposed strategy consists of randomly convexifying the reconstruction contractive auto-encoding (RCAE) learning objective and solving the resulting large-scale convex minimization problem in the frequency domain via coordinate descent (CD). The main advantages of our proposed learning strategy are: (1) single tunable optimization parameter; (2) fast and guaranteed convergence; (3) possibilities for full parallelization. Numerical experiments show that our proposed learning strategy scales (in the worst case) linearly with image size, number of filters and filter size.