Reinforcement Learning-based Adaptive Mitigation of Uncorrected DRAM Errors in the Field
This addresses a critical reliability issue for large-scale computing systems like supercomputers, offering a cost-effective solution to reduce downtime and computational waste, though it is an incremental improvement over existing methods.
The paper tackles the problem of uncorrected DRAM errors causing job failures and wasted computation in large systems by introducing an adaptive mitigation method using reinforcement learning, which reduces lost compute time by 54% compared to no mitigation and performs within 6% of an optimal Oracle method on production supercomputer logs.
Scaling to larger systems, with current levels of reliability, requires cost-effective methods to mitigate hardware failures. One of the main causes of hardware failure is an uncorrected error in memory, which terminates the current job and wastes all computation since the last checkpoint. This paper presents the first adaptive method for triggering uncorrected error mitigation. It uses a prediction approach that considers the likelihood of an uncorrected error and its current potential cost. The method is based on reinforcement learning, and the only user-defined parameters are the mitigation cost and whether the job can be restarted from a mitigation point. We evaluate our method using classical machine learning metrics together with a cost-benefit analysis, which compares the cost of mitigation actions with the benefits from mitigating some of the errors. On two years of production logs from the MareNostrum supercomputer, our method reduces lost compute time by 54% compared with no mitigation and is just 6% below the optimal Oracle method. All source code is open source.