A Hierarchical Deep Reinforcement Learning Framework for 6-DOF UCAV Air-to-Air Combat
This work addresses a domain-specific challenge in autonomous aerial combat for military or robotics applications, representing an incremental improvement by combining existing methods like PPO and fictitious self-play.
The paper tackles the problem of 6-DOF unmanned combat air vehicle air-to-air combat by proposing a hierarchical deep reinforcement learning framework, resulting in an inner loop controller that outperforms a fine-tuned PID controller and an outer loop strategy that achieves higher winning rates through evolution.
Unmanned combat air vehicle (UCAV) combat is a challenging scenario with continuous action space. In this paper, we propose a general hierarchical framework to resolve the within-vision-range (WVR) air-to-air combat problem under 6 dimensions of degree (6-DOF) dynamics. The core idea is to divide the whole decision process into two loops and use reinforcement learning (RL) to solve them separately. The outer loop takes into account the current combat situation and decides the expected macro behavior of the aircraft according to a combat strategy. Then the inner loop tracks the macro behavior with a flight controller by calculating the actual input signals for the aircraft. We design the Markov decision process for both the outer loop strategy and inner loop controller, and train them by proximal policy optimization (PPO) algorithm. For the inner loop controller, we design an effective reward function to accurately track various macro behavior. For the outer loop strategy, we further adopt a fictitious self-play mechanism to improve the combat performance by constantly combating against the historical strategies. Experiment results show that the inner loop controller can achieve better tracking performance than fine-tuned PID controller, and the outer loop strategy can perform complex maneuvers to get higher and higher winning rate, with the generation evolves.