Model-aided Federated Reinforcement Learning for Multi-UAV Trajectory Planning in IoT Networks
This addresses the problem of efficient and low-cost training for UAV coordination in IoT networks, though it is incremental as it builds on existing MARL and federated learning methods.
The paper tackles the challenge of costly real-world training data for multi-agent reinforcement learning (MARL) in UAV trajectory planning for IoT data harvesting by proposing a model-aided federated MARL algorithm, which reduces the need for real-world training experiences by around three magnitudes while achieving similar data collection performance.
Deploying teams of unmanned aerial vehicles (UAVs) to harvest data from distributed Internet of Things (IoT) devices requires efficient trajectory planning and coordination algorithms. Multi-agent reinforcement learning (MARL) has emerged as a solution, but requires extensive and costly real-world training data. To tackle this challenge, we propose a novel model-aided federated MARL algorithm to coordinate multiple UAVs on a data harvesting mission with only limited knowledge about the environment. The proposed algorithm alternates between building an environment simulation model from real-world measurements, specifically learning the radio channel characteristics and estimating unknown IoT device positions, and federated QMIX training in the simulated environment. Each UAV agent trains a local QMIX model in its simulated environment and continuously consolidates it through federated learning with other agents, accelerating the learning process. A performance comparison with standard MARL algorithms demonstrates that our proposed model-aided FedQMIX algorithm reduces the need for real-world training experiences by around three magnitudes while attaining similar data collection performance.