Multi-Query Focused Disaster Summarization via Instruction-Based Prompting
This work addresses disaster management by improving summarization of emergency events from social media and news, though it is incremental as it builds on existing retrieval and prompting techniques.
The paper tackled the challenge of multi-query disaster summarization from web sources by using a retrieval and instruction-based prompting method with LLaMA-13b, achieving strong results in automatic metrics and human evaluation but noting a performance gap compared to proprietary systems.
Automatic summarization of mass-emergency events plays a critical role in disaster management. The second edition of CrisisFACTS aims to advance disaster summarization based on multi-stream fact-finding with a focus on web sources such as Twitter, Reddit, Facebook, and Webnews. Here, participants are asked to develop systems that can extract key facts from several disaster-related events, which ultimately serve as a summary. This paper describes our method to tackle this challenging task. We follow previous work and propose to use a combination of retrieval, reranking, and an embarrassingly simple instruction-following summarization. The two-stage retrieval pipeline relies on BM25 and MonoT5, while the summarizer module is based on the open-source Large Language Model (LLM) LLaMA-13b. For summarization, we explore a Question Answering (QA)-motivated prompting approach and find the evidence useful for extracting query-relevant facts. The automatic metrics and human evaluation show strong results but also highlight the gap between open-source and proprietary systems.