Positional Encoding to Control Output Sequence Length
This addresses a practical constraint in real-world summarization applications where summaries must not exceed desired lengths, though it is incremental as it builds on existing positional encoding methods.
The paper tackles the problem of controlling output sequence length in neural encoder-decoder models for abstractive summarization, proposing a sinusoidal positional encoding extension that allows generation of any length, even unseen in training, and improves ROUGE scores.
Neural encoder-decoder models have been successful in natural language generation tasks. However, real applications of abstractive summarization must consider additional constraint that a generated summary should not exceed a desired length. In this paper, we propose a simple but effective extension of a sinusoidal positional encoding (Vaswani et al., 2017) to enable neural encoder-decoder model to preserves the length constraint. Unlike in previous studies where that learn embeddings representing each length, the proposed method can generate a text of any length even if the target length is not present in training data. The experimental results show that the proposed method can not only control the generation length but also improve the ROUGE scores.