JANUS: Benchmarking Commercial and Open-Source Cloud and Edge Platforms for Object and Anomaly Detection Workloads
This work provides a comparative analysis of performance and cost for IoT analytics workloads across various platforms, which is useful for developers and businesses deploying IoT solutions to make informed decisions about platform selection.
This paper benchmarks commercial and open-source cloud and edge platforms for object and anomaly detection workloads, analyzing performance and cost implications. For compute-light outlier detection, AWS IoT Greengrass achieved at least 2X lower latency and 1.25X lower cost than other cloud platforms. For compute-intensive streaming video analytics, open-source solutions on cloud VMs were cheaper but up to 6X slower than proprietary solutions, while running on low-powered edge devices reduced latency by up to 49X.
With diverse IoT workloads, placing compute and analytics close to where data is collected is becoming increasingly important. We seek to understand what is the performance and the cost implication of running analytics on IoT data at the various available platforms. These workloads can be compute-light, such as outlier detection on sensor data, or compute-intensive, such as object detection from video feeds obtained from drones. In our paper, JANUS, we profile the performance/$ and the compute versus communication cost for a compute-light IoT workload and a compute-intensive IoT workload. In addition, we also look at the pros and cons of some of the proprietary deep-learning object detection packages, such as Amazon Rekognition, Google Vision, and Azure Cognitive Services, to contrast with open-source and tunable solutions, such as Faster R-CNN (FRCNN). We find that AWS IoT Greengrass delivers at least 2X lower latency and 1.25X lower cost compared to all other cloud platforms for the compute-light outlier detection workload. For the compute-intensive streaming video analytics task, an opensource solution to object detection running on cloud VMs saves on dollar costs compared to proprietary solutions provided by Amazon, Microsoft, and Google, but loses out on latency (up to 6X). If it runs on a low-powered edge device, the latency is up to 49X lower.