Attention-Seeker: Dynamic Self-Attention Scoring for Unsupervised Keyphrase Extraction
This addresses the problem of extracting keyphrases from text, especially long documents, for applications like information retrieval, but it is incremental as it builds on existing self-attention methods.
The paper tackled unsupervised keyphrase extraction by proposing Attention-Seeker, a method that uses self-attention maps from a Large Language Model to score candidate phrases dynamically without manual tuning, achieving state-of-the-art performance on three out of four datasets.
This paper proposes Attention-Seeker, an unsupervised keyphrase extraction method that leverages self-attention maps from a Large Language Model to estimate the importance of candidate phrases. Our approach identifies specific components - such as layers, heads, and attention vectors - where the model pays significant attention to the key topics of the text. The attention weights provided by these components are then used to score the candidate phrases. Unlike previous models that require manual tuning of parameters (e.g., selection of heads, prompts, hyperparameters), Attention-Seeker dynamically adapts to the input text without any manual adjustments, enhancing its practical applicability. We evaluate Attention-Seeker on four publicly available datasets: Inspec, SemEval2010, SemEval2017, and Krapivin. Our results demonstrate that, even without parameter tuning, Attention-Seeker outperforms most baseline models, achieving state-of-the-art performance on three out of four datasets, particularly excelling in extracting keyphrases from long documents.