Imposing Hard Constraints on Deep Networks: Promises and Limitations
This work addresses a theoretical challenge in deep learning for researchers, but it is incremental as it shows limitations without practical gains.
The paper tackled the problem of imposing hard constraints on deep neural networks to improve prediction quality with less labeled data, finding that while computationally feasible, the approach did not outperform existing soft constraint methods.
Imposing constraints on the output of a Deep Neural Net is one way to improve the quality of its predictions while loosening the requirements for labeled training data. Such constraints are usually imposed as soft constraints by adding new terms to the loss function that is minimized during training. An alternative is to impose them as hard constraints, which has a number of theoretical benefits but has not been explored so far due to the perceived intractability of the problem. In this paper, we show that imposing hard constraints can in fact be done in a computationally feasible way and delivers reasonable results. However, the theoretical benefits do not materialize and the resulting technique is no better than existing ones relying on soft constraints. We analyze the reasons for this and hope to spur other researchers into proposing better solutions.