Benchmarking Web API Quality -- Revisited
This addresses the gap in understanding runtime web API quality, which impacts applications and user experience, by providing updated empirical evidence, though it is incremental as it builds on prior work.
The paper revisits a 3-month, geo-distributed benchmark of popular web APIs from 2015 by repeating it in 2018, comparing availability and latency, and adding new results on provider security preferences and outreach attempts. It finds that web API quality varies based on geo-distribution, time, and between experiments, with concrete data from the benchmarks.
Modern applications increasingly interact with web APIs -- reusable components, deployed and operated outside the application, and accessed over the network. Their existence, arguably, spurs application innovations, making it easy to integrate data or functionalities. While previous work has analyzed the ecosystem of web APIs and their design, little is known about web API quality at runtime. This gap is critical, as qualities including availability, latency, or provider security preferences can severely impact applications and user experience. In this paper, we revisit a 3-month, geo-distributed benchmark of popular web APIs, originally performed in 2015. We repeat this benchmark in 2018 and compare results from these two benchmarks regarding availability and latency. We furthermore introduce new results from assessing provider security preferences, collected both in 2015 and 2018, and results from our attempts to reach out to API providers with the results from our 2015 experiments. Our extensive experiments show that web API qualities vary 1.) based on the geo-distribution of clients, 2.) during our individual experiments, and 3.) between the two experiments. Our findings provide evidence to foster the discussion around web API quality, and can act as a basis for the creation of tools and approaches to mitigate quality issues.