Online Action-Stacking Improves Reinforcement Learning Performance for Air Traffic Control
This addresses a key gap for air traffic control by enabling more efficient and realistic command generation, though it is incremental as it builds on existing reinforcement learning methods.
The paper tackled the challenge of applying reinforcement learning to air traffic control by introducing online action-stacking, which compiles primitive actions into realistic commands, reducing instruction frequency by using only five actions while matching the performance of a 37-dimensional action space.
We introduce online action-stacking, an inference-time wrapper for reinforcement learning policies that produces realistic air traffic control commands while allowing training on a much smaller discrete action space. Policies are trained with simple incremental heading or level adjustments, together with an action-damping penalty that reduces instruction frequency and leads agents to issue commands in short bursts. At inference, online action-stacking compiles these bursts of primitive actions into domain-appropriate compound clearances. Using Proximal Policy Optimisation and the BluebirdDT digital twin platform, we train agents to navigate aircraft along lateral routes, manage climb and descent to target flight levels, and perform two-aircraft collision avoidance under a minimum separation constraint. In our lateral navigation experiments, action stacking greatly reduces the number of issued instructions relative to a damped baseline and achieves comparable performance to a policy trained with a 37-dimensional action space, despite operating with only five actions. These results indicate that online action-stacking helps bridge a key gap between standard reinforcement learning formulations and operational ATC requirements, and provides a simple mechanism for scaling to more complex control scenarios.