Position: Adopt Constraints Over Penalties in Deep Learning
This addresses a key problem in developing trustworthy AI systems by providing a more reliable method for incorporating constraints, though it is incremental as it builds on existing constrained optimization techniques.
The paper argues that using penalties to enforce constraints in deep learning is fundamentally flawed because it cannot guarantee constraint satisfaction and optimal performance simultaneously, and it requires costly tuning. It advocates for adopting constrained optimization methods like the Lagrangian approach, which solves the constrained problem accountably, eliminates tuning, and integrates with deep learning pipelines.
Recent efforts to develop trustworthy AI systems with accountability guarantees have led to widespread use of machine learning formulations incorporating external requirements, or constraints. These requirements are often enforced via penalization--adding fixed-weight terms to the task loss. We argue this approach is fundamentally ill-suited since there may be no penalty coefficient that simultaneously ensures constraint satisfaction and optimal constrained performance, i.e., that truly solves the constrained problem. Moreover, tuning these coefficients requires costly trial-and-error, incurring significant time and computational overhead. We, therefore, advocate for broader adoption of tailored constrained optimization methods--such as the Lagrangian approach, which jointly optimizes the penalization "coefficients" (the Lagrange multipliers) and the model parameters. Such methods (i) truly solve the constrained problem and do so accountably, by clearly defining feasibility and verifying when it is achieved, (ii) eliminate the need for extensive penalty tuning, and (iii) integrate seamlessly with modern deep learning pipelines.