DUAL: Diversity and Uncertainty Active Learning for Text Summarization
This addresses the challenge of efficiently collecting high-quality annotated data for text summarization, which is crucial for training and evaluating models, especially when annotation resources are limited, representing an incremental improvement over prior active learning methods.
The paper tackled the problem of active learning for text summarization, where existing methods often underperform random sampling, by proposing DUAL, a novel algorithm that combines uncertainty and diversity to select samples, and demonstrated that it consistently matches or outperforms the best strategies in experiments with various models and datasets.
With the rise of large language models, neural text summarization has advanced significantly in recent years. However, even state-of-the-art models continue to rely heavily on high-quality human-annotated data for training and evaluation. Active learning is frequently used as an effective way to collect such datasets, especially when annotation resources are scarce. Active learning methods typically prioritize either uncertainty or diversity but have shown limited effectiveness in summarization, often being outperformed by random sampling. We present Diversity and Uncertainty Active Learning (DUAL), a novel algorithm that combines uncertainty and diversity to iteratively select and annotate samples that are both representative of the data distribution and challenging for the current model. DUAL addresses the selection of noisy samples in uncertainty-based methods and the limited exploration scope of diversity-based methods. Through extensive experiments with different summarization models and benchmark datasets, we demonstrate that DUAL consistently matches or outperforms the best performing strategies. Using visualizations and quantitative metrics, we provide valuable insights into the effectiveness and robustness of different active learning strategies, in an attempt to understand why these strategies haven't performed consistently in text summarization. Finally, we show that DUAL strikes a good balance between diversity and robustness.