Why is it Difficult to Detect Sudden and Unexpected Epidemic Outbreaks in Twitter?
This work addresses the problem of improving early detection of unpredictable outbreaks for public health officials, but it appears incremental as it builds on existing Twitter-based monitoring with specific enhancements.
The paper tackled the problem of detecting sudden and unexpected epidemic outbreaks in Twitter data, which is challenging due to noise and limited prior focus on common events, by developing a Twitter-based Epidemic Intelligence system that addresses dynamic classification, alert generation, and ranking; the result was an empirically evaluated approach validated with domain experts, though no concrete numbers are provided.
Social media services such as Twitter are a valuable source of information for decision support systems. Many studies have shown that this also holds for the medical domain, where Twitter is considered a viable tool for public health officials to sift through relevant information for the early detection, management, and control of epidemic outbreaks. This is possible due to the inherent capability of social media services to transmit information faster than traditional channels. However, the majority of current studies have limited their scope to the detection of common and seasonal health recurring events (e.g., Influenza-like Illness), partially due to the noisy nature of Twitter data, which makes outbreak detection and management very challenging. Within the European project M-Eco, we developed a Twitter-based Epidemic Intelligence (EI) system, which is designed to also handle a more general class of unexpected and aperiodic outbreaks. In particular, we faced three main research challenges in this endeavor: 1) dynamic classification to manage terminology evolution of Twitter messages, 2) alert generation to produce reliable outbreak alerts analyzing the (noisy) tweet time series, and 3) ranking and recommendation to support domain experts for better assessment of the generated alerts. In this paper, we empirically evaluate our proposed approach to these challenges using real-world outbreak datasets and a large collection of tweets. We validate our solution with domain experts, describe our experiences, and give a more realistic view on the benefits and issues of analyzing social media for public health.