Word Order's Impacts: Insights from Reordering and Generation Analysis
This work addresses the role of word order in language models for NLP researchers, but it is incremental as it builds on prior studies with new tasks and datasets.
The paper revisits hypotheses about word order's impact in natural language processing by analyzing ChatGPT's performance on order reconstruction and generation tasks across four datasets, finding that ChatGPT relies on word order for inference but cannot confirm or deny redundancy with lexical semantics.
Existing works have studied the impacts of the order of words within natural text. They usually analyze it by destroying the original order of words to create a scrambled sequence, and then comparing the models' performance between the original and scrambled sequences. The experimental results demonstrate marginal drops. Considering this findings, different hypothesis about word order is proposed, including ``the order of words is redundant with lexical semantics'', and ``models do not rely on word order''. In this paper, we revisit the aforementioned hypotheses by adding a order reconstruction perspective, and selecting datasets of different spectrum. Specifically, we first select four different datasets, and then design order reconstruction and continuing generation tasks. Empirical findings support that ChatGPT relies on word order to infer, but cannot support or negate the redundancy relations between word order lexical semantics.