Revising with a Backward Glance: Regressions and Skips during Reading as Cognitive Signals for Revision Policies in Incremental Processing
This work addresses the efficiency of incremental processing in NLP by proposing a novel signal for training revision policies, though it is incremental as it builds on existing models and datasets.
The paper tackled the problem of identifying when incremental NLP models should revise their output by investigating human reading behaviors (regressions and skips) as training signals, finding that these behaviors can predict revisions in models like BiLSTMs and Transformers across multiple languages.
In NLP, incremental processors produce output in instalments, based on incoming prefixes of the linguistic input. Some tokens trigger revisions, causing edits to the output hypothesis, but little is known about why models revise when they revise. A policy that detects the time steps where revisions should happen can improve efficiency. Still, retrieving a suitable signal to train a revision policy is an open problem, since it is not naturally available in datasets. In this work, we investigate the appropriateness of regressions and skips in human reading eye-tracking data as signals to inform revision policies in incremental sequence labelling. Using generalised mixed-effects models, we find that the probability of regressions and skips by humans can potentially serve as useful predictors for revisions in BiLSTMs and Transformer models, with consistent results for various languages.