Revisiting the Systematicity in Negation in the Era of In-Context Learning
For NLP researchers, this work provides an incremental analysis of LLMs' limitations in understanding negation, a known challenge.
The paper investigates how large language models understand negation through in-context learning, finding that while they can recognize negation expressions and scope to some degree, performance is imperfect and varies with output format; function vectors for scope recognition are harder to construct than for cue extraction.
Understanding the meaning of negated sentences remains one of the challenges for language models, even in the era of large language models (LLMs). We analyze systematicity regarding LLM understanding of negation from two perspectives: behavioral systematicity and representational systematicity. For behavioral systematicity, we confirm that through demonstrations and in-context learning, LLMs can recognize negation expressions and scope within sentences to some extent, but they fail to achieve perfect performance. In particular, the difficulty of the negation scope recognition for models varies depending on the output format. For representational systematicity, we analyze the extent to which function vectors can be robustly constructed from in-context examples for tasks that are essential to understanding negation. The experiments suggest that while function vectors can be composed for negation cue extraction tasks, extracting function vectors for recognizing scope is more challenging.