DFWLayer: Differentiable Frank-Wolfe Optimization Layer
This work addresses the need for efficient, projection-free optimization layers in machine learning, offering a domain-specific improvement for handling large-scale convex problems with norm constraints.
The paper tackles the problem of integrating constrained optimization into neural networks by proposing DFWLayer, a differentiable layer based on the Frank-Wolfe method, which achieves competitive accuracy in solutions and gradients while maintaining constraint adherence.
Differentiable optimization has received a significant amount of attention due to its foundational role in the domain of machine learning based on neural networks. This paper proposes a differentiable layer, named Differentiable Frank-Wolfe Layer (DFWLayer), by rolling out the Frank-Wolfe method, a well-known optimization algorithm which can solve constrained optimization problems without projections and Hessian matrix computations, thus leading to an efficient way of dealing with large-scale convex optimization problems with norm constraints. Experimental results demonstrate that the DFWLayer not only attains competitive accuracy in solutions and gradients but also consistently adheres to constraints.