Demystifying the Performance of Data Transfers in High-Performance Research Networks
This addresses the lack of monitoring mechanisms for data transfers in high-performance research networks, which is critical for improving utilization of expensive infrastructures, but is incremental as it builds on existing monitoring concepts.
The paper tackles the problem of data transfers in high-performance research networks failing to achieve promised rates due to various issues, presenting a scalable monitoring framework that can monitor up to 400 transfers per host and over 40,000 transfers total, with a heuristic method identifying root causes of anomalies with an F-score of 87-98%.
High-speed research networks are built to meet the ever-increasing needs of data-intensive distributed workflows. However, data transfers in these networks often fail to attain the promised transfer rates for several reasons, including I/O and network interference, server misconfigurations, and network anomalies. Although understanding the root causes of performance issues is critical to mitigating them and increasing the utilization of expensive network infrastructures, there is currently no available mechanism to monitor data transfers in these networks. In this paper, we present a scalable, end-to-end monitoring framework to gather and store key performance metrics for file transfers to shed light on the performance of transfers. The evaluation results show that the proposed framework can monitor up to 400 transfers per host and more than 40, 000 transfers in total while collecting performance statistics at one-second precision. We also introduce a heuristic method to automatically process the gathered performance metrics and identify the root causes of performance anomalies with an F-score of 87 - 98%.