LaMSUM: Amplifying Voices Against Harassment through LLM Guided Extractive Summarization of User Incident Reports
This work addresses the problem of efficiently processing user incident reports for stakeholders in public safety platforms, though it is incremental as it adapts existing LLM capabilities to a specific domain.
The authors tackled the challenge of summarizing large volumes of code-mixed language reports on sexual harassment from platforms like Safe City in India by introducing LaMSUM, a multi-level framework using LLMs for extractive summarization, which outperformed state-of-the-art methods in evaluations with models like Llama, Mistral, and GPT-4o.
Citizen reporting platforms like Safe City in India help the public and authorities stay informed about sexual harassment incidents. However, the high volume of data shared on these platforms makes reviewing each individual case challenging. Therefore, a summarization algorithm capable of processing and understanding various Indian code-mixed languages is essential. In recent years, Large Language Models (LLMs) have shown exceptional performance in NLP tasks, including summarization. LLMs inherently produce abstractive summaries by paraphrasing the original text, while the generation of extractive summaries - selecting specific subsets from the original text - through LLMs remains largely unexplored. Moreover, LLMs have a limited context window size, restricting the amount of data that can be processed at once. We tackle these challenge by introducing LaMSUM, a novel multi-level framework designed to generate extractive summaries for large collections of Safe City posts using LLMs. LaMSUM integrates summarization with different voting methods to achieve robust summaries. Extensive evaluation using three popular LLMs (Llama, Mistral and GPT-4o) demonstrates that LaMSUM outperforms state-of-the-art extractive summarization methods for Safe City posts. Overall, this work represents one of the first attempts to achieve extractive summarization through LLMs, and is likely to support stakeholders by offering a comprehensive overview and enabling them to develop effective policies to minimize incidents of unwarranted harassment.