EntropyRank: Unsupervised Keyphrase Extraction via Side-Information Optimization for Language Model-based Text Compression
This addresses the problem of extracting keyphrases without supervision for text analysis, but it is incremental as it builds on existing language models and benchmarks.
The paper tackles unsupervised keyphrase extraction by proposing EntropyRank, which selects phrases with highest conditional entropy under a pre-trained language model to optimize text compression. Empirically, it achieves results comparable to commonly used methods on various benchmark challenges.
We propose an unsupervised method to extract keywords and keyphrases from texts based on a pre-trained language model (LM) and Shannon's information maximization. Specifically, our method extracts phrases having the highest conditional entropy under the LM. The resulting set of keyphrases turns out to solve a relevant information-theoretic problem: if provided as side information, it leads to the expected minimal binary code length in compressing the text using the LM and an entropy encoder. Alternately, the resulting set is an approximation via a causal LM to the set of phrases that minimize the entropy of the text when conditioned upon it. Empirically, the method provides results comparable to the most commonly used methods in various keyphrase extraction benchmark challenges.