Frank-Wolfe Network: An Interpretable Deep Structure for Non-Sparse Coding
This addresses a difficulty in signal processing and machine learning for non-sparse coding, offering an interpretable deep learning method with potential applications in image tasks, though it appears incremental as it builds on existing optimization algorithms.
The paper tackles the problem of non-sparse coding with Lp-norm constraints for p>1, proposing the Frank-Wolfe Network (F-W Net) by unrolling the Frank-Wolfe algorithm, which introduces a novel closed-form nonlinear unit and allows learnable p. It demonstrates strong performance in simulations, handwritten digit recognition, and tasks like image denoising and super-resolution, showing effectiveness and flexibility.
The problem of $L_p$-norm constrained coding is to convert signal into code that lies inside an $L_p$-ball and most faithfully reconstructs the signal. Previous works under the name of sparse coding considered the cases of $L_0$ and $L_1$ norms. The cases with $p>1$ values, i.e. non-sparse coding studied in this paper, remain a difficulty. We propose an interpretable deep structure namely Frank-Wolfe Network (F-W Net), whose architecture is inspired by unrolling and truncating the Frank-Wolfe algorithm for solving an $L_p$-norm constrained problem with $p\geq 1$. We show that the Frank-Wolfe solver for the $L_p$-norm constraint leads to a novel closed-form nonlinear unit, which is parameterized by $p$ and termed $pool_p$. The $pool_p$ unit links the conventional pooling, activation, and normalization operations, making F-W Net distinct from existing deep networks either heuristically designed or converted from projected gradient descent algorithms. We further show that the hyper-parameter $p$ can be made learnable instead of pre-chosen in F-W Net, which gracefully solves the non-sparse coding problem even with unknown $p$. We evaluate the performance of F-W Net on an extensive range of simulations as well as the task of handwritten digit recognition, where F-W Net exhibits strong learning capability. We then propose a convolutional version of F-W Net, and apply the convolutional F-W Net into image denoising and super-resolution tasks, where F-W Net all demonstrates impressive effectiveness, flexibility, and robustness.