RDD: Pareto Analysis of the Rate-Distortion-Distinguishability Trade-off
This work addresses the challenge of balancing data compression with detection accuracy for network monitoring systems, though it is incremental as it extends an existing framework.
The paper tackles the trade-off between compression efficiency, distortion, and anomaly detection performance in monitoring systems, showing that a Pareto analysis can better manage this three-way trade-off than optimizing only for rate-distortion.
Extensive monitoring systems generate data that is usually compressed for network transmission. This compressed data might then be processed in the cloud for tasks such as anomaly detection. However, compression can potentially impair the detector's ability to distinguish between regular and irregular patterns due to information loss. Here we extend the information-theoretic framework introduced in [1] to simultaneously address the trade-off between the three features on which the effectiveness of the system depends: the effectiveness of compression, the amount of distortion it introduces, and the distinguishability between compressed normal signals and compressed anomalous signals. We leverage a Gaussian assumption to draw curves showing how moving on a Pareto surface helps administer such a trade-off better than simply relying on optimal rate-distortion compression and hoping that compressed signals can be distinguished from each other.