Efficacy of Utilizing Large Language Models to Detect Public Threat Posted Online
This work addresses the problem of automated threat detection for online content moderators, but it is incremental as it applies existing LLMs to a new, small dataset.
This paper tackled the problem of detecting public threats posted online by evaluating large language models (LLMs) on a dataset of Korean online posts, finding that GPT-4 achieved 97.9% accuracy for non-threats and 100% for threats, indicating LLMs can effectively augment content moderation.
This paper examines the efficacy of utilizing large language models (LLMs) to detect public threats posted online. Amid rising concerns over the spread of threatening rhetoric and advance notices of violence, automated content analysis techniques may aid in early identification and moderation. Custom data collection tools were developed to amass post titles from a popular Korean online community, comprising 500 non-threat examples and 20 threats. Various LLMs (GPT-3.5, GPT-4, PaLM) were prompted to classify individual posts as either "threat" or "safe." Statistical analysis found all models demonstrated strong accuracy, passing chi-square goodness of fit tests for both threat and non-threat identification. GPT-4 performed best overall with 97.9% non-threat and 100% threat accuracy. Affordability analysis also showed PaLM API pricing as highly cost-efficient. The findings indicate LLMs can effectively augment human content moderation at scale to help mitigate emerging online risks. However, biases, transparency, and ethical oversight remain vital considerations before real-world implementation.