Magpie: Automatically Tuning Static Parameters for Distributed File Systems using Deep Reinforcement Learning
This addresses the difficulty of optimizing performance in distributed file systems for users and administrators, representing a novel method for a known bottleneck.
The paper tackles the problem of tuning static parameters in distributed file systems, which is challenging and time-consuming, and proposes Magpie, a deep reinforcement learning approach that improves Lustre's throughput by an average of 91.8% compared to default configurations and 39.7% over a baseline.
Distributed file systems are widely used nowadays, yet using their default configurations is often not optimal. At the same time, tuning configuration parameters is typically challenging and time-consuming. It demands expertise and tuning operations can also be expensive. This is especially the case for static parameters, where changes take effect only after a restart of the system or workloads. We propose a novel approach, Magpie, which utilizes deep reinforcement learning to tune static parameters by strategically exploring and exploiting configuration parameter spaces. To boost the tuning of the static parameters, our method employs both server and client metrics of distributed file systems to understand the relationship between static parameters and performance. Our empirical evaluation results show that Magpie can noticeably improve the performance of the distributed file system Lustre, where our approach on average achieves 91.8% throughput gains against default configuration after tuning towards single performance indicator optimization, while it reaches 39.7% more throughput gains against the baseline.