Logit Reweighting for Topic-Focused Summarization
This addresses the problem of maintaining topical focus in summarization for users needing resource-efficient alternatives to fine-tuning, though it is incremental as it builds on existing logit manipulation methods.
The paper tackled the challenge of generating abstractive summaries that adhere to a specific topic by proposing a lightweight method that reweights logits of topic-relevant tokens during generation, showing that techniques like Threshold Selection effectively increase topical vocabulary use without compromising quality on the NEWTS dataset with Gemma-2B and Llama-3-8B models.
Generating abstractive summaries that adhere to a specific topic remains a significant challenge for language models. While standard approaches, such as fine-tuning, are resource-intensive, simpler methods like prompt engineering often struggle to maintain topical focus, particularly with smaller models. To address this, we propose a lightweight method that enhances topical relevance by directly reweighting the logits of topic-relevant tokens during generation. We evaluate three such reweighting techniques: Constant Shift, which adds a constant value to logits; Factor Scaling, which multiplies them by a factor; and Threshold Selection, which selectively boosts logits that exceed a probability threshold. Experiments on the NEWTS topical summarization dataset, using both Gemma-2B and Llama-3-8B models, show that these techniques effectively increase the use of topic-relevant vocabulary. Notably, the Threshold Selection method successfully improves topical focus without compromising summary quality-a trade-off often seen in other approaches. Our findings demonstrate that directly reweighting logits is a practical and resource-efficient alternative to fine-tuning, offering a promising pathway for precisely controlling the thematic content of generated text.