Word Order's Impacts: Insights from Reordering and Generation Analysis
Qinghua Zhao Jiaang Li Lei Li Zenghui Zhou Junfeng Liu
Existing works have studied the impacts of the order of words within natural
text. They usually analyze it by destroying the original order of words to
create a scrambled sequence, and then comparing the models' performance between
the original and scrambled sequences. The experimental results demonstrate
marginal drops. Considering this findings, different hypothesis about word
order is proposed, including ``the order of words is redundant with lexical
semantics'', and ``models do not rely on word order''. In this paper, we
revisit the aforementioned hypotheses by adding a order reconstruction
perspective, and selecting datasets of different spectrum. Specifically, we
first select four different datasets, and then design order reconstruction and
continuing generation tasks. Empirical findings support that ChatGPT relies on
word order to infer, but cannot support or negate the redundancy relations
between word order lexical semantics.