Statistical Anomaly Detection via Composite Hypothesis Testing for Markov Models
This work addresses anomaly detection for enhancing cyber security in communication networks and building smarter transportation systems, representing an incremental improvement in threshold estimation.
The paper tackled the problem of statistical anomaly detection under Markovian assumptions by deriving a new threshold estimator for the Hoeffding test, which improved false alarm control while maintaining detection probabilities, as validated through numerical experiments.
Under Markovian assumptions, we leverage a Central Limit Theorem (CLT) for the empirical measure in the test statistic of the composite hypothesis Hoeffding test so as to establish weak convergence results for the test statistic, and, thereby, derive a new estimator for the threshold needed by the test. We first show the advantages of our estimator over an existing estimator by conducting extensive numerical experiments. We find that our estimator controls better for false alarms while maintaining satisfactory detection probabilities. We then apply the Hoeffding test with our threshold estimator to detecting anomalies in two distinct applications domains: one in communication networks and the other in transportation networks. The former application seeks to enhance cyber security and the latter aims at building smarter transportation systems in cities.