The Road to Quality is Paved with Good Revisions: A Detailed Evaluation Methodology for Revision Policies in Incremental Sequence Labelling
This work addresses the problem of improving revision policies for incremental dialogue models, which is incremental as it builds on existing methods for sequence labeling.
The authors formalized and characterized edits and revisions in incremental sequence labeling, proposing metrics to evaluate revision policies, and applied this methodology to profile three Transformer-based encoders across various tasks.
Incremental dialogue model components produce a sequence of output prefixes based on incoming input. Mistakes can occur due to local ambiguities or to wrong hypotheses, making the ability to revise past outputs a desirable property that can be governed by a policy. In this work, we formalise and characterise edits and revisions in incremental sequence labelling and propose metrics to evaluate revision policies. We then apply our methodology to profile the incremental behaviour of three Transformer-based encoders in various tasks, paving the road for better revision policies.