Markov-Enhanced Clustering for Long Document Summarization: Tackling the 'Lost in the Middle' Challenge with Large Language Models
This addresses the challenge of retaining key information in lengthy documents for users needing efficient summarization, representing an incremental improvement over existing hybrid methods.
The paper tackles the 'lost in the middle' problem in long document summarization with large language models by proposing a hybrid approach that splits documents into chunks, clusters embeddings, and uses a Markov chain to order ideas, achieving a ROUGE-L score of 0.45 on a benchmark dataset.
The rapid expansion of information from diverse sources has heightened the need for effective automatic text summarization, which condenses documents into shorter, coherent texts. Summarization methods generally fall into two categories: extractive, which selects key segments from the original text, and abstractive, which generates summaries by rephrasing the content coherently. Large language models have advanced the field of abstractive summarization, but they are resourceintensive and face significant challenges in retaining key information across lengthy documents, which we call being "lost in the middle". To address these issues, we propose a hybrid summarization approach that combines extractive and abstractive techniques. Our method splits the document into smaller text chunks, clusters their vector embeddings, generates a summary for each cluster that represents a key idea in the document, and constructs the final summary by relying on a Markov chain graph when selecting the semantic order of ideas.