LLM-Powered Automatic Translation and Urgency in Crisis Scenarios
This work addresses risks in deploying language technologies for crisis communication, which is critical for multilingual crisis response, but it is incremental as it builds on existing evaluation methods.
The study evaluated the performance of state-of-the-art LLMs and machine translation systems in crisis-domain translation, focusing on preserving urgency, and found substantial degradation and instability, with translations distorting perceived urgency and classifications varying by language.
Large language models (LLMs) are increasingly proposed for crisis preparedness and response, particularly for multilingual communication. However, their suitability for high-stakes crisis contexts remains insufficiently evaluated. This work examines the performance of state-of-the-art LLMs and machine translation systems in crisis-domain translation, with a focus on preserving urgency, which is a critical property for effective crisis communication and triaging. Using multilingual crisis data and a newly introduced urgency-annotated dataset covering over 32 languages, we show that both dedicated translation models and LLMs exhibit substantial performance degradation and instability. Crucially, even linguistically adequate translations can distort perceived urgency, and LLM-based urgency classifications vary widely depending on the language of the prompt and input. These findings highlight significant risks in deploying general-purpose language technologies for crisis communication and underscore the need for crisis-aware evaluation frameworks.