Rethinking Burst Buffer Optimization: Enabling Layout Heterogeneity via Hybrid Analysis and LLM Guidance
For HPC system administrators and users, Proteus addresses the fundamental mismatch between fixed data layouts and diverse application I/O patterns, eliminating the need for prior training or exhaustive profiling.
Proteus optimizes burst buffer performance by enabling layout heterogeneity based on application I/O semantics, achieving 91.30% decision accuracy and up to 3.24× speedup for write-intensive workloads and 2.9× for metadata-intensive workloads.
Burst buffers (BBs) are essential for mitigating I/O bottlenecks in modern HPC systems. However, existing BB file systems often suffer from structural performance degradation due to fixed data layouts that fail to align with diverse application behaviors. While current machine-learning-based optimizations focus primarily on tuning storage stack parameters for a given layout, they offer diminishing returns when a fundamental mismatch exists between I/O patterns and the underlying data organization. Furthermore, these approaches typically incur prohibitive costs due to extensive training or intrusive profiling. To bridge this gap, we present Proteus, a semantic-aware BB system that treats data layout as a first-class optimization dimension. The core insight of Proteus is that application I/O intent can be reconstructed by synergetically combining static code structures with lightweight runtime signals. Through a hybrid pipeline and a single execution probe, Proteus extracts latent semantic cues to determine the optimal layout prior to production runs-eliminating the need for prior training or exhaustive profiling. Evaluation with representative HPC workloads shows that Proteus achieves 91.30\% decision accuracy, delivering up to 3.24$\times$ and 2.9$\times$ speedups for write-intensive and metadata-intensive workloads, respectively.