HydraSum: Disentangling Stylistic Features in Text Summarization using Multi-Decoder Models
This work addresses the challenge of controlling summary styles for users in text summarization, offering a novel method to disentangle and enforce stylistic features, though it is incremental as it builds on existing single-decoder frameworks.
The paper tackles the problem of implicitly encoded stylistic features in text summarization by introducing HydraSum, a multi-decoder architecture that automatically learns contrasting summary styles without extra supervision, and demonstrates its ability to generate stylistically-diverse summaries that outperform baseline models on three datasets (CNN, Newsroom, and XSum).
Summarization systems make numerous "decisions" about summary properties during inference, e.g. degree of copying, specificity and length of outputs, etc. However, these are implicitly encoded within model parameters and specific styles cannot be enforced. To address this, we introduce HydraSum, a new summarization architecture that extends the single decoder framework of current models to a mixture-of-experts version with multiple decoders. We show that HydraSum's multiple decoders automatically learn contrasting summary styles when trained under the standard training objective without any extra supervision. Through experiments on three summarization datasets (CNN, Newsroom and XSum), we show that HydraSum provides a simple mechanism to obtain stylistically-diverse summaries by sampling from either individual decoders or their mixtures, outperforming baseline models. Finally, we demonstrate that a small modification to the gating strategy during training can enforce an even stricter style partitioning, e.g. high- vs low-abstractiveness or high- vs low-specificity, allowing users to sample from a larger area in the generation space and vary summary styles along multiple dimensions.