Beyond Distributional Hypothesis: Let Language Models Learn Meaning-Text Correspondence
This addresses a trustworthiness issue in language models for NLP applications, but it is incremental as it builds on existing probing and training methods.
The paper tackles the problem that pre-trained language models (PLMs) frequently violate the logical negation property (LNP), which is crucial for trustworthiness, by proposing a meaning-matching intermediate training task to learn meaning-text correspondence. The result shows that this approach improves LNP understanding, enables learning of lexical semantics, and outperforms previous methods while being time- and resource-efficient, with similar or better performance on 7 GLUE tasks.
The logical negation property (LNP), which implies generating different predictions for semantically opposite inputs, is an important property that a trustworthy language model must satisfy. However, much recent evidence shows that large-size pre-trained language models (PLMs) do not satisfy this property. In this paper, we perform experiments using probing tasks to assess PLM's LNP understanding. Unlike previous studies that only examined negation expressions, we expand the boundary of the investigation to lexical semantics. Through experiments, we observe that PLMs violate the LNP frequently. To alleviate the issue, we propose a novel intermediate training task, names meaning-matching, designed to directly learn a meaning-text correspondence, instead of relying on the distributional hypothesis. Through multiple experiments, we find that the task enables PLMs to learn lexical semantic information. Also, through fine-tuning experiments on 7 GLUE tasks, we confirm that it is a safe intermediate task that guarantees a similar or better performance of downstream tasks. Finally, we observe that our proposed approach outperforms our previous counterparts despite its time and resource efficiency.