Artificial Disfluency Detection, Uh No, Disfluency Generation for the Masses
This addresses the data scarcity issue in disfluency detection for natural language processing applications, though it is an incremental improvement over existing generation methods.
The paper tackles the problem of limited annotated datasets for disfluency detection by proposing LARD, a method to automatically generate artificial disfluencies from fluent text, which increased the accuracy of existing detectors in evaluations.
Existing approaches for disfluency detection typically require the existence of large annotated datasets. However, current datasets for this task are limited, suffer from class imbalance, and lack some types of disfluencies that can be encountered in real-world scenarios. This work proposes LARD, a method for automatically generating artificial disfluencies from fluent text. LARD can simulate all the different types of disfluencies (repetitions, replacements and restarts) based on the reparandum/interregnum annotation scheme. In addition, it incorporates contextual embeddings into the disfluency generation to produce realistic context-aware artificial disfluencies. Since the proposed method requires only fluent text, it can be used directly for training, bypassing the requirement of annotated disfluent data. Our empirical evaluation demonstrates that LARD can indeed be effectively used when no or only a few data are available. Furthermore, our detailed analysis suggests that the proposed method generates realistic disfluencies and increases the accuracy of existing disfluency detectors.