CrisisBench: Benchmarking Crisis-related Social Media Datasets for Humanitarian Information Processing
This work addresses the need for standardized datasets to enable better model comparison and training in crisis informatics for humanitarian organizations, but it is incremental as it combines existing data.
The authors tackled the problem of comparing and measuring progress in crisis informatics by consolidating eight existing datasets into a benchmark with 166.1k tweets for informativeness and 141.5k for humanitarian classification tasks, providing benchmarks using deep learning architectures.
Time-critical analysis of social media streams is important for humanitarian organizations for planing rapid response during disasters. The \textit{crisis informatics} research community has developed several techniques and systems for processing and classifying big crisis-related data posted on social media. However, due to the dispersed nature of the datasets used in the literature (e.g., for training models), it is not possible to compare the results and measure the progress made towards building better models for crisis informatics tasks. In this work, we attempt to bridge this gap by combining various existing crisis-related datasets. We consolidate eight human-annotated datasets and provide 166.1k and 141.5k tweets for \textit{informativeness} and \textit{humanitarian} classification tasks, respectively. We believe that the consolidated dataset will help train more sophisticated models. Moreover, we provide benchmarks for both binary and multiclass classification tasks using several deep learning architecrures including, CNN, fastText, and transformers. We make the dataset and scripts available at: https://crisisnlp.qcri.org/crisis_datasets_benchmarks.html