A Deep Multi-Agent Reinforcement Learning Approach to Autonomous Separation Assurance
This addresses the bottleneck of human cognitive limitations in air traffic control to potentially increase sector capacity, though it is incremental as it builds on existing methods like PPO.
The paper tackles the problem of autonomous separation assurance for aircraft in high-density air traffic by proposing a deep multi-agent reinforcement learning framework with an attention network, which significantly reduces training time and increases performance in simulated case studies.
A novel deep multi-agent reinforcement learning framework is proposed to identify and resolve conflicts among a variable number of aircraft in a high-density, stochastic, and dynamic sector. Currently the sector capacity is constrained by human air traffic controller's cognitive limitation. We investigate the feasibility of a new concept (autonomous separation assurance) and a new approach to push the sector capacity above human cognitive limitation. We propose the concept of using distributed vehicle autonomy to ensure separation, instead of a centralized sector air traffic controller. Our proposed framework utilizes Proximal Policy Optimization (PPO) that we modify to incorporate an attention network. This allows the agents to have access to variable aircraft information in the sector in a scalable, efficient approach to achieve high traffic throughput under uncertainty. Agents are trained using a centralized learning, decentralized execution scheme where one neural network is learned and shared by all agents. The proposed framework is validated on three challenging case studies in the BlueSky air traffic control environment. Numerical results show the proposed framework significantly reduces offline training time, increases performance, and results in a more efficient policy.