Convex Dual Theory Analysis of Two-Layer Convolutional Neural Networks with Soft-Thresholding
This work addresses the problem of achieving globally optimal solutions in soft-thresholding neural networks for researchers and practitioners, though it appears incremental as it builds on existing convexification methods.
The paper tackled the nonconvex training difficulty in two-layer convolutional neural networks with soft-thresholding by designing a convex dual network, proving strong duality holds theoretically and numerically, and validating it in linear fitting and denoising experiments.
Soft-thresholding has been widely used in neural networks. Its basic network structure is a two-layer convolution neural network with soft-thresholding. Due to the network's nature of nonlinearity and nonconvexity, the training process heavily depends on an appropriate initialization of network parameters, resulting in the difficulty of obtaining a globally optimal solution. To address this issue, a convex dual network is designed here. We theoretically analyze the network convexity and numerically confirm that the strong duality holds. This conclusion is further verified in the linear fitting and denoising experiments. This work provides a new way to convexify soft-thresholding neural networks.