Each pursuer can be viewed as an agent. The pursuers engage in an interactive process, wherein they gather information on their teammates within the communication range, information on the escapee within the detection range, and information gleaned from their own detection. Based on this information, the pursuers generate observations of their environment and output corresponding actions according to their policy. The environment provides feedback in the form of rewards for the actions taken by the pursuers. The policy is dynamically updated throughout the pursuit process to promote optimal actions and ensure coordinated pursuit by the UAV swarm.

Due to the inherent limitations of their sensing capabilities, each UAV is only able to observe a limited portion of the real environment. As shown in Fig. 1, the observation of UAV i is oi={oii,oij,oie}, where i,j∈{1,2,…,N},i≠j, N is the number of pursuers,

e is the escapee,

ϕi, ϕj, and ϕe are the heading angles of pursuer i, pursuer j and escapee e, respectively,

oii={xi,yi,vi,ϕi} is the information of UAV i itself,

oij={dij,δij} is the information of other pursuers in the communication range, in which dij is the Euclidean distance between pursuer i with other pursuer j, and δij is the heading error, defined as the angle between the heading of pursuer i and the vector from pursuer i to pursuer j,

oie={die,δie} is the information of the escapee, in which die is the Euclidean distance between pursuer i with escapee e, and δie is the heading error, defined as the angle between the heading of pursuer i and the vector from pursuer i to escapee e.

Consequently, the state representation of each pursuer comprises 2N+4 variables, with the number of variables being linearly proportional to the number of pursuers N. To standardize the value range, eliminate scale-related errors, and enhance the effectiveness of neural network training, each state variable is normalized.

The action space is the set of all actions that the pursuers can perform. During the multi-UAV confrontation process, each pursuer selects its actions based on its own observations. In a pursuit-evasion scenario where UAVs are flying at a constant speed, the control can be simplified to angular velocity control, ignoring the effect of wind speed on the airspace. As the angular velocity varies, the states of UAVs change accordingly. The range of angular velocity falls within the interval of [−3 rad/s,3 rad/s]. To facilitate the learning process, we discretize the continuous angular velocity range into eight distinct actions for each pursuer to learn.

It is important to design an effective reward function to guide the learning process of the multi-UAV system. In the pursuit task, the reward function plays a crucial role in controlling the pursuers to chase the escapee while avoiding collisions and staying within the boundaries. We propose two types of rewards: cooperative pursuit reward and punishment reward.

The formation reward received by the pursuers at each step t is given by: (10)rformation(i)=(1N∑j=1Ncos⁡(θij))2,where θij denotes the angle between pursuer i and pursuer j relative to the direction of the evader, i,j∈{1,2,…,N},i≠j. A smaller value of this angle indicates that the pursuers are distributed in more diverse directions (i.e., forming larger mutual angles), which implies a better encirclement effect. Conversely, a larger angle suggests that the pursuers are aligned in similar directions, leading to a less effective encirclement.

The distance-based reward function for the pursuers is defined as follows: (11)rdist(i)=11+e−ς(die−dth),where dth denotes the distance threshold for a successful capture, ς is a constant controlling the steepness of the curve, and die represents the Euclidean distance between pursuer i and evader e. A reward function with dynamic decay encourages the pursuers to be more sensitive to distance. The penalty is large when the distance is far but decreases rapidly as it narrows, which helps avoid overly aggressive close-range pursuit.

Therefore, the cooperative reward function for the pursuers is defined as: (12)rcoo(i)={rwin,if die≤dthrhelper,if dje≤dth,∃j≠i−ωfrformation(i)−ωdrdist(i),  otherwisewhere ωf and ωd are the weights of formation reward and distance, respectively. Each pursuer receives a negative reward at every stage of the unfinished mission, which is a weighted linear combination of rformation(i) and rdist(i). The pursuers that capture the escapee will receive the reward rwin, and the others will receive rhelper, and rwin>rhelper.

We design a cooperative reward function that encourages pursuers to approach the evader from different directions, forming an encirclement pattern that yields a formation reward. However, if the agents only aim to spread around the evader without moving closer, the pursuit task cannot be completed. To address this, we introduce a dynamically decaying distance reward. When a pursuer is far from the evader, the distance reward dominates the cooperative reward function; as the pursuer approaches the evader, the formation reward becomes more prominent. This new reward formulation encourages pursuers to spread out around the evader while penalizing similar approach angles, thereby promoting the learning of complex cooperative pursuit strategies. It also increases the probability of successful capture in multi-UAV systems by incentivizing agents to quickly acquire optimal policies.

Punishment reward. UAVs need to pursue the evader while avoiding collisions with others and flying out of the mission area to ensure flight safety. Therefore, it is essential to create a justifiable reward function to direct UAVs toward a safe flight. The penalty reward includes the collision penalty and the boundary penalty.

The collision penalty reward is given when two pursuers are closer together than the safe distance dsafe. It is (13)rsafe(i)={−10,if dij≤dsafe,∃j≠i0,otherwisewhere dij is the Euclidean distance between two chasers.

There are security risks when UAVs fly outside of the mission area. By flying too close to boundaries, UAVs have a portion of their perception range fall into the non-mission area, which is useless. We define dxmax, dxmin, dymax, dymin are the xi and yi maximum and minimum distance to the boundary, respectively. The boundary penalty reward is (14)rbound(i)={0,if dxmin<xi<dxmax and dymin<yi<dymax−10.otherwise

For the pursuer i, the penalty reward for each step is (15)rcol(i)=0.5rsafe(i)+0.5rbound(i).

The final reward function can be formulated as (16)r(i)=rcoo(i)+rcol(i).

The goal is to train the optimal policy for the decentralized execution of each UAV through centralized learning that obtains the states and actions of all UAVs. Each UAV learns to take the optimal flight actions by itself through local observation to complete the pursuit mission. The UAV i uses the policy π(i) to select the action at(i) based on the current observation ot(i). The issue can be stated as (17)maxΩ={π(1),…,π(N)}η(π),where π is the joint policy; Ω is the collection of policies of pursuers; η(π) is the expected discount return, and η(π)=Es0∼ζ0(s0)[Vπ(s0)], and ζ0(s0) is the distribution of the initial state s0. The optimal joint policy π∗ of the pursuers in a fully cooperative multi-agent task is expressed as (18)π∗(at|ot)=∏i=1Nπ(i)(at(i)|ot(i)).

We evaluate the proposed approach and develop a simulation platform that simulates the multi-UAV pursuit-evasion game.

We assume that all UAVs fly at the same altitude and their airspace is limited to an area of 400 m * 400 m. UAVs can begin at any location within the permitted space. The initial locations of all UAVs are provided in relatively stable positions to prevent collisions and facilitate the training process. The aircraft we modeled is a four-rotor UAV. The velocity of a chasing UAV is vp=9 m/s for the situation of a fixed linear velocity, while the velocity of an escaping UAV is ve=11 m/s. Both types of UAVs can fly at a range of angular velocities ω∈[−3 rad/s,3 rad/s]. The first-order differential of the UAVs’ velocity and position coordinate values is determined using the fourth-order Runge-Kutta [43] differential equation numerical solution method. The new state can be calculated for each UAV during the simulation time unit t, as it determines its angular velocity through decision-making. This approach is more precise than the Euler method of direct numerical computation.

The simulation time unit is set to t=0.15 s, during which all UAVs concurrently execute their selected maneuvers. The resulting motion states collectively define the system’s next state.

It is assumed that the maximum communication distance of each UAV is 50 m, the maximum detection distance is 20 m, and the capture range of the pursuer is 15 m. When the distance between the evader and at least one pursuer is less than 15 m, the pursuit task is deemed accomplished due to the air combat simulation platform does not model and simulate the close-range combat missile. The safe distance between UAVs to prevent collisions is 30 m. UAVs modify flight directions to avoid collisions by sharing information with nearby companions.

We train the pursuers using the proposed algorithm, while an escapee can use the DRL model to learn or just follow the rules. The proposed approach is used to control the angular velocity of the pursuers. We adopt the clear and effective rules for a rule-based escapee: within the permissible range of angular velocity, the escapee can choose with greed the maximum angular velocity of a speedier escape. The weights of rewards ωf and ωd are set to 0.1 and 0.002, respectively. The environmental parameters are shown in Table 1.

All experiments in the training process are performed on a workstation with an Intel i9-10850K CPU and a Nvidia GeForce GTX 3060 GPU. We use PyTorch for network implementation.

The algorithm is trained at 5,000,000 steps with a constant set of 200 steps per episode. The environment and UAVs are reset after the confrontation procedure is finished or when the maximum steps allowed per episode are achieved.

All actor networks in MA2PPO share the same structure, as do all critic networks. A fully connected neural network with two hidden layers makes up the actor networks. We parameterize the actors using the MLP with 128 units per layer. Rectified linear unit (ReLu) functions serve as the activation function in both hidden layers. The network layers of the critics have 256 neurons. As the activation function in the buried layer, we employ Leaky ReLU. There is no activation function in the output layer, which is merely a linear layer. The Adam optimizer is adopted by both the actor network and the critic network, with a learning rate of 0.0005. Other hyperparameters are set to the discount factor γ is 0.99 and the GAE lambda is 0.95. Table 2 depicts the hyperparameter configurations of our algorithm.

We choose MAAC, COMA, and MAPPO as the benchmark algorithms to compare the performance of the proposed algorithm in dealing with the issue of multi-UAV pursuit game. The implementation details of the benchmark algorithm are shown below.

MAAC: The MAAC algorithm is a traditional MARL algorithm with an attention mechanism built on the soft actor-critic (SAC) and CTDE. The actor networks have a hidden layer unit number of 128, and ReLu is their activation function. Leaky ReLu is an activation function for the output layer of the network. The hidden layer unit number of the critic networks is 256. Its activation function is Leaky ReLu, and the output layer of the network is activated using the ReLu function. The actor and critic networks are optimized using the Adam optimizer, and the learning rate is 0.001. Networks for the target actors and the target critics both employ soft updates. Other hyperparameters include: the buffer size is 1000 K, the batch size is 1024, and the discount factor is 0.9.

COMA: The multi-agent credit assignment problem is the focus of COMA. It utilizes the single, central network to anticipate the Q value of each agent with distinct forwarding. An input layer, two hidden recurrent neural network (RNN) layers and an output layer make up the policy network. There are 64 hidden layer units in the RNN hidden layers, which uses the GRU unit. The critic network has 128 hidden layer units and is a four-layer fully connected linear unit. Both the activation mechanisms of the policy and the critic are ReLu activation functions. The RMS optimizer is used by COMA. The actor network and the critic network have learning rates of 0.0001 and 0.001, respectively. The discount factor is 0.99.

MAPPO: It is a widely used MARL algorithm that has demonstrated strong performance in various applications. Its hyperparameters are similar to those of MA2PPO, and the parameters for various implementation details are set to the same as those from [44].

We employ three evaluation metrics to demonstrate the effectiveness of the proposed approach.

Average episode reward (AER): The average episode reward is a significant evaluation criterion in MARL. It measures the effectiveness of the proposed algorithm’s training by tracking its growth and convergence. To demonstrate the optimization process of the model, it is crucial to consider the cumulative reward value of the confrontation and maximize it.

Average pursuit success rate (APSR): The average pursuit success rate is the ratio of the sum of successful steps to all steps in an episode. It is used to evaluate the quality of the pursuit. The algorithm performs better with a higher APSR. We define it as (26)APSR=1|H|∑t=1HIt,

(27)It={1,if κt>00,if κt=0where |H| represents the number of steps in the episodes; κt is the number of UAVs that are successfully pursued in the time step t.

We train MA2PPO, MAPPO, MAAC, and COMA in the same environment to evaluate and analyze their performance of learning strategies in the multi-UAV pursuit environment.

Fig. 4 shows that MA2PPO outperforms other algorithms in terms of the AER, the APSR demonstrating its superior and steady performance in the multi-UAV pursuit game. The AER of the four methods is displayed in Fig. 4a. As evidenced by the fact that the AER values of MA2PPO and MAPPO dramatically increase at the beginning of the training period, and both obtain respectably high scores from negative starting values. The AER curve illustrates that both algorithms have learned the pursuit strategy, but MA2PPO has learned it better. For MA2PPO, the AER value can be roughly converged to 50,000 steps. MAPPO takes twice as many convergence steps as MA2PPO, and its AER has a certain gap with MA2PPO. MAAC and COMA both performed poorly overall, and MA2PPO has significantly surpassed them in the AER.

We can see that the trend of the APSR curve shown in Fig. 4b resembles the AER curve. The pursuers, in the beginning, could not complete the task well. The APSR curve increased notably when the pursuers use the MA2PPO algorithm and begin chasing the escapee effectively as training progresses. At roughly 50,000 steps, the success rate of the pursuers is nearly 98%. MAPPO can also learn successful pursuit strategies, but its average success rate is lower than MA2PPO. There was no significant increase in the APSR values of COMA and MAAC.

Two ablative versions of MA2PPO, namely MA2PPO without dynamic decoupling and MA2PPO without formation reward, are also contrasted and examined as baselines to show the efficacy of various components. MA2PPO without dynamic decoupling highlights the importance of formation reward. It replaces the simple distance reward with the novel reward function we presented, based on PPO and our suggested framework. The purpose of MA2PPO without formation reward is to confirm how the dynamic decoupling affects the resolution of credit assignment issues. Only the dynamic decoupling portion of the algorithm is kept, and the reward function is just the straightforward distance reward. Fig. 5 displays the MA2PPO and its ablative versions’ training results. The comparisons of two evaluation criteria demonstrate that MA2PPO performs better than the other two algorithms.

Ablation Study of Dynamic Decoupling: We employ MHA to decouple the multi-UAV cooperative pursuit problem, as discussed in Section 4.2. Dynamic decoupling is effective at resolving the credit assignment problem and promoting active collaboration between nearby UAVs. By using the attention mechanism, UAVs are incentivized to approach one another when others are helpful for their task. Conversely, collisions between UAVs are penalized in the reward function when they are too close to one another. In situations where they need to avoid each other, UAVs dynamically decouple from one another. These incentives are what propel each UAV forward. Ensuring that UAVs coordinate their motivations to keep an eye on each other while avoiding collisions is critical. Effective tracking of an escapee by UAVs is dependent on ensuring that the incentives of the individual UAVs are aligned. The MA2PPO algorithm, which we propose, facilitates this by enabling UAVs to exhibit a wide range of cooperative behaviors that can be easily adapted to different situations.

In Figs. 6 and 7, we illustrate the process of dynamic decoupling by visualizing the trajectories of all UAVs and the corresponding attention weights. The trajectories of all UAVs at the 13th step after initialization is shown in Fig. 6a. The escapee flies to the upper left as the pursuers begin their hunt in accordance with the MA2PPO method they have learned. The flight directions of UAVs 1, 2, and 4 make it easier to complete the chase. As seen in Fig. 7a, UAV 1 has a higher attention weight value for UAVs 2 and 4 than UAV 3. UAVs 2 and 3 are more practical than UAV 1 to work with to accomplish the interception of escapees when the distance between them and their respective flight directions is considered. Therefore, UAV 2 pays more attention to UAV 3 than to UAV 1 while UAV 2 is also focused on UAV 1. Similarly, the attention weights of UAVs 3 and 4 in Fig. 7a support our analysis of UAVs 1 and 2.

Fig. 6b shows the trajectories of the multi-UAV at the 51st step. UAVs 1, 2, and 4 may be seen flying in the direction of the escapee as the hunt goes on. UAV 3 is currently near the direction of the escapee. Fig. 7b demonstrates that UAVs 1 and 2 both recognize that working with UAV 3 is more favorable to completing the hunt. Hence, the attention weight of UAV 3 is increased by UAV 1 while the attention weight of UAV 4 is decreased. UAV 2 also learned this, which has increased attention to UAVs 1 and 3, reducing attention to UAV 4. UAVs 1, 2, and 3 consequently dynamically disconnect UAV 4 and concentrate more on each other.

The trajectories of pursuers in Fig. 6c serves as a confirmation of the analysis in Fig. 7b. UAVs 1, 2, and 3 fly straight to the escapee, as shown in Fig. 6c, as they prepare to form a round-up state to complete the pursuit of the evader. In Fig. 7c, UAVs 1, 2, and 3 add attention weights to each other, whereas UAV 4 receives nearly no attention. The results demonstrate that the dynamic decoupling enables pursuers to determine in real-time which companions are more beneficial to the pursuit process to pay them more attention and work more closely together. This approach decomposes the large-scale cooperative problem into smaller, decoupled subproblems and eliminates lazy agents, motivating the pursuers to collaborate more effectively with one another.

Ablation Study of Formation Reward: We analyze the outcome of giving each pursuer the formation reward, which is offered to promote a great pursuit manner at each time step. The AER of MA2PPO is higher than MA2PPO without formation reward, as seen in Fig. 5. Additionally, the average pursuit success rate will rise when utilizing the formation reward. The MA2PPO algorithm enables UAVs to engage in effective multi-UAV pursuit cooperation by reshaping their original rewards through the formation reward received from their neighboring UAVs. It also helps to address the drawbacks of the distance reward, hence strengthening system performance and boosting individual UAV rewards. This further emphasizes the value of UAV cluster cooperation.

UAVs use the MA2PPO method to work together to track down the escapee and intuitively determine whether they have learned to create a formation by displaying the trajectory visualization of various tracking processes. As seen in Fig. 8a, the pursuers use the cooperative chasing policies in response to the escapee’s unexpected turn and successfully catch up with it. UAV 1 keeps close track of the evader, even if it makes a sudden turn. UAVs 2, 3, and 4 are also flying in the direction of the evader, establishing a hunting state, and effectively intercepting the evader. Fig. 8b and c exhibits the trajectories of pursuers in a variety of additional scenarios. When the escapee flees quickly in one direction in Fig. 8b and 8c, the pursuers are coordinating their activities and pursuing the escapee from various angles. Then, when the besieged state of the escapee is not met, two pursuers closely follow it while others disperse around them. The pursuers then construct an encirclement around the evader to prevent the evader from escaping. While they approach the evader for arrest after the evader is in the encirclement and the encirclement is fulfilled. They coordinate the actions to move from encirclement to capture. Finally, they go straight after the escapee untill it is apprehended. The results demonstrate that the pursuers successfully handle the chase process in various states. The formation reward incentivizes the pursuers to encircle the escapee to meet the siege, then shorten the siege until the escapee is apprehended, enabling closer multi-UAV cooperation.

Ablation analysis of reward function weights. To systematically evaluate the impact of the formation reward weight ωf and the distance reward weight ωd on learning performance, we conduct an ablation study using the APSR as the performance metric while keeping all other settings unchanged. Five different combinations of weights are tested, including the parameter settings: ωf=0.025, ωd=0.0005; ωf=0.05, ωd=0.001; ωf=0.1, ωd=0.002; ωf=0.15, ωd=0.003; ωf=0.2, ωd=0.004.

As shown in Fig. 9, the algorithm achieves the best performance in terms of convergence speed and final success rate when ωf=0.1 and ωd=0.002. We observe that a moderate ωf encourages agents to collaboratively encircle the evader without causing premature clustering, while the ωd ensures that pursuing the evader remains the primary incentive. This balance promotes rapid learning in the early stages and sustains a high success rate later in training. These results indicate that the chosen weights effectively balance cooperative encirclement and target approach. Therefore, we set ωf and ωd to 0.1 and 0.002, respectively.

We test the capacity of strategies that are learned by the MA2PPO algorithm to scale in various situations. We trained all six of the algorithms we previously compared in various settings with various numbers of UAVs used for chase and escape. We still evaluated them using the two criteria listed in Section 5.1. Table 3 displays the statistics of the test findings.

From an overall perspective, MA2PPO consistently outperforms the other five baseline methods across all tested scenarios, demonstrating strong stability and generalization capabilities. Notably, in the highly complex 8 vs. 2 task, MA2PPO maintains an APSR of 0.96 and an AER of 403.99, significantly exceeding the performance of the comparative algorithms. These results highlight the superior ability of MA2PPO in multi-target task allocation and strategic coordination under challenging conditions.

The ablation studies on MA2PPO variants without key modules reveal that removing the dynamic decoupling mechanism leads to reduced coordination accuracy, while excluding the formation reward significantly decreases group encirclement efficiency. For instance, in the 6 vs. 2 scenario, the complete MA2PPO achieves an AER of 311.22, which drops to 240.18 when the formation reward is removed, validating the critical role of our proposed modules in enhancing system-level cooperation. In contrast, MAPPO, MAAC, and COMA exhibit a noticeable decline in performance when the number of evaders increases or the task scale expands. Although MAAC also incorporates an attention mechanism, its overall success rate across different scenarios is substantially lower than that of MA2PPO. As the task scale increases, most baseline methods show significant performance fluctuations, whereas MA2PPO maintains relatively stable performance, indicating that MA2PPO-based learning demonstrates superior generalization and scalability across diverse scenarios.

An in-depth analysis is conducted to uncover the reasons behind the advantages of the MA2PPO algorithm. The use of dynamic decoupling may be able to identify those highly correlated connections that are essential and eliminate those weakly correlated interactions. Additionally, the formation reward encourages the pursuers to consider nearby companions that can create a particular form when determining the reward for each UAV. The pursuers can learn better cooperative manners to execute the mission from these nearby companions, which dramatically increases pursuit efficiency.

In summary, the proposed MA2PPO algorithm enhances coordination efficiency and adaptability across different task configurations and can be extended to a variety of cooperative scenarios involving more pursuers and escapees.

We analyze the algorithm from the perspective of computational complexity. There are N homogeneous UAVs, each with a state dimensionality of M, and a continuous action dimensionality of 1. During centralized training, the critic network for each UAV must incorporate the states and actions of the other N−1 UAVs, resulting in an input dimensionality of 𝒪(N2×M). However, with the use of a MHA mechanism, the input dimensionality is reduced to 𝒪(N×M). For small-scale UAV teams, the computational overhead introduced by the attention mechanism is negligible compared to approaches that do not employ it. Nevertheless, in larger-scale UAV swarms (e.g., N>10), the computational burden increases dramatically without the attention mechanism, while with attention, the growth remains linear with respect to N. Therefore, our algorithm is more suitable for large-scale swarm tasks.

Moreover, in the extreme case where each UAV communicates with all other N−1 UAVs, interactions are treated as uniformly contributing, which causes the variance of the advantage function estimate to grow quadratically with N. However, not all neighboring UAV messages are relevant to the task. High-variance advantage estimates lead to unstable gradient directions during policy updates, significantly slowing down convergence. By leveraging dynamic decoupling, our method enables each pursuer to identify and retain only strongly correlated interactions with teammates, effectively removing irrelevant advantage terms that do not contribute to policy gradients. This reduces the overall variance and facilitates more efficient policy learning.