LLMs in the Real World: Evaluating "AI" in Emergency Contexts
For researchers and stakeholders in AI deployment, this paper identifies overlooked 'easy' problems that are critical for safe real-world use, though it is primarily a call to action rather than presenting novel results.
The paper calls for researchers to engage more with public communication of AI findings, using a case study of an LLM-based text-2-911 system to highlight misconceptions and provide recommendations for deployment in emergency contexts.
This paper offers a call to action. We urge our colleagues in the research community to play a greater role in the articulation of our findings to the public. To illustrate the stakes we present a case study on the initial stages of an LLM-based machine translation application's deployment in a real-world context: a text-2-911 system advertising capabilities in 55 languages for use in emergencies in which it may be difficult to call operators directly. We identify a number of common misconceptions about technologies such as these, concluding with a set of concrete recommendations and best practices for stakeholders at every stage of the development and deployment pipeline. While the advancement of scientific research often lies in solving the "hard" problems, we argue it is often the "easy" ones -- problems for which the latest technology is often unnecessary -- that are most overlooked.