ExpBERT: Representation Engineering with Natural Language Explanations
This addresses the challenge of reducing labeled data requirements for relation extraction in NLP, though it is incremental as it builds on existing BERT methods.
The paper tackles the problem of incorporating inductive biases into relation extraction models by allowing developers to specify them as natural language explanations, using BERT fine-tuned on MultiNLI to interpret these explanations. The result is that ExpBERT matches a BERT baseline with 3–20x less labeled data and improves F1 scores by 3–10 points with the same data across three tasks.
Suppose we want to specify the inductive bias that married couples typically go on honeymoons for the task of extracting pairs of spouses from text. In this paper, we allow model developers to specify these types of inductive biases as natural language explanations. We use BERT fine-tuned on MultiNLI to ``interpret'' these explanations with respect to the input sentence, producing explanation-guided representations of the input. Across three relation extraction tasks, our method, ExpBERT, matches a BERT baseline but with 3--20x less labeled data and improves on the baseline by 3--10 F1 points with the same amount of labeled data.