Detection of Anomalies in a Time Series Data using InfluxDB and Python
This work is significant for water supply companies and environmental monitoring systems, as it aims to enable early recognition of undesirable changes in drinking water quality by detecting anomalies in sensor data, allowing for timely rectification of issues.
This paper addresses the problem of detecting anomalous data points in time-series water and environmental data, which can arise from sensor malfunctions or system breakdowns. The authors propose cost-sensitive machine learning algorithms, specifically modified Logistic Regression, Random Forest, and Support Vector Machines, to minimize misclassification costs. Their results indicate that Random Forest performed best at identifying anomalies.
Analysis of water and environmental data is an important aspect of many intelligent water and environmental system applications where inference from such analysis plays a significant role in decision making. Quite often these data that are collected through sensible sensors can be anomalous due to different reasons such as systems breakdown, malfunctioning of sensor detectors, and more. Regardless of their root causes, such data severely affect the results of the subsequent analysis. This paper demonstrates data cleaning and preparation for time-series data and further proposes cost-sensitive machine learning algorithms as a solution to detect anomalous data points in time-series data. The following models: Logistic Regression, Random Forest, Support Vector Machines have been modified to support the cost-sensitive learning which penalizes misclassified samples thereby minimizing the total misclassification cost. Our results showed that Random Forest outperformed the rest of the models at predicting the positive class (i.e anomalies). Applying predictive model improvement techniques like data oversampling seems to provide little or no improvement to the Random Forest model. Interestingly, with recursive feature elimination, we achieved a better model performance thereby reducing the dimensions in the data. Finally, with Influxdb and Kapacitor the data was ingested and streamed to generate new data points to further evaluate the model performance on unseen data, this will allow for early recognition of undesirable changes in the drinking water quality and will enable the water supply companies to rectify on a timely basis whatever undesirable changes abound.