Characterizing Metastable Faults and Failures
For systems engineers, this work offers a foundational causal understanding and practical methodology to address a previously poorly understood class of failures.
The paper provides the first causal characterization of metastable failures, identifying their origin in metastable faults—structural destabilizing cycles among system components. It derives a methodology to predict such failures and build metastable-fault-tolerant systems, demonstrated through three case studies.
Metastable failures are hard to detect, prevent, and mitigate. During a metastable failure, a system exhibits self-sustaining bad behavior even in the absence of adversarial conditions. Prior work focuses on symptoms and has portrayed metastable failures as instances of self-sustaining overload. This characterization leaves the underlying failure causes and dynamics unknown, and does not account for metastable failures that do not manifest as overload. We present the first causal characterization of metastable failures by identifying their origin in metastable faults, i.e., structural destabilizing cycles of interaction among systems components that, in isolation, are stabilizing. Metastable failures arise when scheduling decisions let these destabilizing interactions gain the upper hand over the individual components' stabilizing tendencies. We then derive a methodology to predict metastable failures, and to build metastable-fault-tolerant (MFT) systems. We apply our methodology to three case studies, showcasing the generality of our results.