Semantic Objective Functions: A distribution-aware method for adding logical constraints in deep learning
This work addresses the need for safer and more explainable AI systems by integrating logical constraints into neural networks, though it is incremental as it builds on prior symbolic and distillation techniques.
The authors tackled the problem of embedding logical constraints into deep learning models for safety and explainability by proposing a loss-based method that combines original loss functions with distribution distances like Fisher-Rao or KL divergence. They demonstrated its effectiveness on tasks such as constrained classification and knowledge distillation, achieving competitive results with existing methods.
Issues of safety, explainability, and efficiency are of increasing concern in learning systems deployed with hard and soft constraints. Symbolic Constrained Learning and Knowledge Distillation techniques have shown promising results in this area, by embedding and extracting knowledge, as well as providing logical constraints during neural network training. Although many frameworks exist to date, through an integration of logic and information geometry, we provide a construction and theoretical framework for these tasks that generalize many approaches. We propose a loss-based method that embeds knowledge-enforces logical constraints-into a machine learning model that outputs probability distributions. This is done by constructing a distribution from the external knowledge/logic formula and constructing a loss function as a linear combination of the original loss function with the Fisher-Rao distance or Kullback-Leibler divergence to the constraint distribution. This construction includes logical constraints in the form of propositional formulas (Boolean variables), formulas of a first-order language with finite variables over a model with compact domain (categorical and continuous variables), and in general, likely applicable to any statistical model that was pretrained with semantic information. We evaluate our method on a variety of learning tasks, including classification tasks with logic constraints, transferring knowledge from logic formulas, and knowledge distillation from general distributions.