Conformal Signal Temporal Logic for Robust Reinforcement Learning Control: A Case Study
This work addresses safety and reliability issues in autonomous flight control for aerospace systems, representing an incremental improvement by integrating formal specifications with data-driven methods.
The paper tackled the problem of enhancing safety and robustness in reinforcement learning control for aerospace applications by introducing a conformal Signal Temporal Logic shield, which preserved STL satisfaction and provided stronger robustness guarantees compared to baselines in simulations under severe stress scenarios.
We investigate how formal temporal logic specifications can enhance the safety and robustness of reinforcement learning (RL) control in aerospace applications. Using the open source AeroBench F-16 simulation benchmark, we train a Proximal Policy Optimization (PPO) agent to regulate engine throttle and track commanded airspeed. The control objective is encoded as a Signal Temporal Logic (STL) requirement to maintain airspeed within a prescribed band during the final seconds of each maneuver. To enforce this specification at run time, we introduce a conformal STL shield that filters the RL agent's actions using online conformal prediction. We compare three settings: (i) PPO baseline, (ii) PPO with a classical rule-based STL shield, and (iii) PPO with the proposed conformal shield, under both nominal conditions and a severe stress scenario involving aerodynamic model mismatch, actuator rate limits, measurement noise, and mid-episode setpoint jumps. Experiments show that the conformal shield preserves STL satisfaction while maintaining near baseline performance and providing stronger robustness guarantees than the classical shield. These results demonstrate that combining formal specification monitoring with data driven RL control can substantially improve the reliability of autonomous flight control in challenging environments.