ChEmREF: Evaluating Language Model Readiness for Chemical Emergency Response
This addresses the need for rapid decision-making in chemical emergencies for responders, but it is incremental as it evaluates existing models on a new domain-specific benchmark.
The paper tackled the problem of whether language models can assist emergency responders in hazardous material incidents by introducing ChEmREF, a benchmark with tasks like chemical translation and response generation, finding that models achieved up to 68.0% exact match in translation but require oversight due to limitations.
Emergency responders managing hazardous material HAZMAT incidents face critical, time-sensitive decisions, manually navigating extensive chemical guidelines. We investigate whether today's language models can assist responders by rapidly and reliably understanding critical information, identifying hazards, and providing recommendations. We introduce the Chemical Emergency Response Evaluation Framework (ChEmREF), a new benchmark comprising questions on 1,035 HAZMAT chemicals from the Emergency Response Guidebook and the PubChem Database. ChEmREF is organized into three tasks: (1) translation of chemical representation between structured and unstructured forms (e.g., converting C2H6O to ethanol), (2) emergency response generation (e.g., recommending appropriate evacuation distances) and (3) domain knowledge question answering from chemical safety and certification exams. Our best evaluated models received an exact match of 68.0% on unstructured HAZMAT chemical representation translation, a LLM Judge score of 52.7% on incident response recommendations, and a multiple-choice accuracy of 63.9% on HAMZAT examinations. These findings suggest that while language models show potential to assist emergency responders in various tasks, they require careful human oversight due to their current limitations.