Abstract
Unmanned Aerial Vehicle (UAV) swarms can provide promising solutions for unmanned delivery, search and rescue, tracking, monitoring, and post-disaster communication recovery in terms of safety and cost. The key challenges in collision avoidance for UAV swarms are safety, energy efficiency, cooperation, and reaction time. Conventional solutions suffer from low energy efficiency, long reaction times, and ineffective cooperation. Recent methods based on machine learning, such as Multi-Agent Reinforcement Learning (MARL), can achieve high energy efficiency, effective cooperation, and fast reaction. However, they have high failure rates in collision avoidance. How to address all these challenges in collision avoidance remains an open and critical problem. This thesis proposes three solutions to address the challenges progressively.
The first solution is the E2Coop algorithm that combines Artificial Potential Field (APF) and Particle Swarm Optimization (PSO) to achieve high energy efficiency and effective cooperation while ensuring safety. In E2Coop, APF provides environmental awareness and coordination to UAVs by constructing a potential field to represent the environment. At the same time, PSO searches optimal trajectories for UAVs, considering safety and energy efficiency under the coordination of APF. The second solution is CoDe, which uses Multi-Agent Reinforcement Learning (MARL) to train cooperative policies for UAVs operating in a swarm. The key contribution in CoDe is a novel credit assignment scheme based on difference rewards and counterfactual policy gradients. The credit assignment scheme requires no assumption on value functions, has low computational complexity, and applies to continuous action space. CoDe is over 90% faster than E2Coop in execution while reducing energy consumption by over 20%. The third solution is CoDe+, which combines E2Coop and CoDe to reduce learning variances and improve sample efficiency. It achieves at least 40% higher average score and saves over 50% of energy than E2Coop and nearly 30% than CoDe on average. The three solutions progressively address the key challenges of collision avoidance in UAV swarms, enabling more applications of UAV swarms in large-scale infrastructure-less and contact-less connections.