Conformal Safety Monitoring for Flight Testing: A Case Study in Data-Driven Safety Learning
This addresses safety monitoring for pilots in flight testing with uncertain parameters, but it is incremental as it combines existing methods like conformal prediction with nearest neighbor classification.
The paper tackles the problem of runtime safety monitoring in flight testing by developing a data-driven approach that uses offline stochastic trajectory simulation to learn a calibrated statistical model of short-term safety risk, enabling reliable identification of unsafe scenarios and outperforming baselines in preemptive risk classification.
We develop a data-driven approach for runtime safety monitoring in flight testing, where pilots perform maneuvers on aircraft with uncertain parameters. Because safety violations can arise unexpectedly as a result of these uncertainties, pilots need clear, preemptive criteria to abort the maneuver in advance of safety violation. To solve this problem, we use offline stochastic trajectory simulation to learn a calibrated statistical model of the short-term safety risk facing pilots. We use flight testing as a motivating example for data-driven learning/monitoring of safety due to its inherent safety risk, uncertainty, and human-interaction. However, our approach consists of three broadly-applicable components: a model to predict future state from recent observations, a nearest neighbor model to classify the safety of the predicted state, and classifier calibration via conformal prediction. We evaluate our method on a flight dynamics model with uncertain parameters, demonstrating its ability to reliably identify unsafe scenarios, match theoretical guarantees, and outperform baseline approaches in preemptive classification of risk.