On Incorporating Semantic Prior Knowledge in Deep Learning Through Embedding-Space Constraints
This addresses the challenge of efficiently integrating human knowledge beyond supervised data in AI systems, particularly for tasks requiring causal relations or domain-specific invariances, though it is incremental as it builds on existing constraint-based approaches.
The paper tackles the problem of incorporating semantic prior knowledge, such as equivalence and logical entailment relations, into deep learning models for tasks like visual question answering, by enforcing these relations as strict constraints on the embedding space, leading to significant improvements in accuracy and robustness compared to soft regularization methods.
The knowledge that humans hold about a problem often extends far beyond a set of training data and output labels. While the success of deep learning mostly relies on supervised training, important properties cannot be inferred efficiently from end-to-end annotations alone, for example causal relations or domain-specific invariances. We present a general technique to supplement supervised training with prior knowledge expressed as relations between training instances. We illustrate the method on the task of visual question answering to exploit various auxiliary annotations, including relations of equivalence and of logical entailment between questions. Existing methods to use these annotations, including auxiliary losses and data augmentation, cannot guarantee the strict inclusion of these relations into the model since they require a careful balancing against the end-to-end objective. Our method uses these relations to shape the embedding space of the model, and treats them as strict constraints on its learned representations. In the context of VQA, this approach brings significant improvements in accuracy and robustness, in particular over the common practice of incorporating the constraints as a soft regularizer. We also show that incorporating this type of prior knowledge with our method brings consistent improvements, independently from the amount of supervised data used. It demonstrates the value of an additional training signal that is otherwise difficult to extract from end-to-end annotations alone.