POLICE: Provably Optimal Linear Constraint Enforcement for Deep Neural Networks
This addresses a critical problem for researchers and practitioners in machine learning who need to incorporate prior knowledge or physical properties into DNNs without compromising training efficiency.
The paper tackles the challenge of strictly enforcing affine constraints on deep neural networks (DNNs) during training and testing, proposing POLICE, a method that provably ensures constraint fulfillment with minimal changes to the forward-pass and allows standard gradient-based optimization.
Deep Neural Networks (DNNs) outshine alternative function approximators in many settings thanks to their modularity in composing any desired differentiable operator. The formed parametrized functional is then tuned to solve a task at hand from simple gradient descent. This modularity comes at the cost of making strict enforcement of constraints on DNNs, e.g. from a priori knowledge of the task, or from desired physical properties, an open challenge. In this paper we propose the first provable affine constraint enforcement method for DNNs that only requires minimal changes into a given DNN's forward-pass, that is computationally friendly, and that leaves the optimization of the DNN's parameter to be unconstrained, i.e. standard gradient-based method can be employed. Our method does not require any sampling and provably ensures that the DNN fulfills the affine constraint on a given input space's region at any point during training, and testing. We coin this method POLICE, standing for Provably Optimal LInear Constraint Enforcement. Github: https://github.com/RandallBalestriero/POLICE