Boosting Summarization with Normalizing Flows and Aggressive Training
This work addresses challenges in variational summarization for NLP researchers, offering incremental improvements with a novel training strategy and analysis of posterior collapse.
The paper tackled insufficient semantic information and posterior collapse in variational summarization by introducing FlowSUM, a normalizing flows-based framework with aggressive training, resulting in significantly enhanced summary quality and potential for knowledge distillation with minimal inference time impact.
This paper presents FlowSUM, a normalizing flows-based variational encoder-decoder framework for Transformer-based summarization. Our approach tackles two primary challenges in variational summarization: insufficient semantic information in latent representations and posterior collapse during training. To address these challenges, we employ normalizing flows to enable flexible latent posterior modeling, and we propose a controlled alternate aggressive training (CAAT) strategy with an improved gate mechanism. Experimental results show that FlowSUM significantly enhances the quality of generated summaries and unleashes the potential for knowledge distillation with minimal impact on inference time. Furthermore, we investigate the issue of posterior collapse in normalizing flows and analyze how the summary quality is affected by the training strategy, gate initialization, and the type and number of normalizing flows used, offering valuable insights for future research.