In this paper, UAVs are assumed to fly at a fixed altitude H, with control focused on horizontal trajectories. The horizontal position of UAV u at time t is denoted as vu(t)=[xu(t),yu(t)]. UAV motion in each time slot t∈T is governed by heading angle ϕu(t)∈[0,2π] and flight speed δu(t), which jointly determine its trajectory. The horizontal position at t+1 is updated as follows: (1){xu(t+1)=xu(t)+δu(t)⋅τ⋅cos⁡(ϕu(t))yu(t+1)=yu(t)+δu(t)⋅τ⋅sin⁡(ϕu(t)),∀u∈𝒰,t∈T

To prevent collisions, a minimum safety distance Dsafe is enforced between UAVs, leading to the following collision avoidance constraints: (2)‖vu(t)−vu′(t)‖=(xu(t)−xu′(t))2+(yu(t)−yu′(t))2≥Dsafe,∀t∈T,u≠u′

Each UAV is also constrained to operate within a fixed rectangular area, so its position vu(t)=[xu(t),yu(t)] must satisfy the constraints in Eq. (3): (3){0≤xu(t)≤Xmax0≤yu(t)≤Ymax,∀u∈𝒰,t∈T

Limited computational resources at terminals and UAVs require implementing FIFO task queues to buffer tasks orderly and ensure sequential execution under high concurrency.

(1) UT queue model

Each user terminal m generates a set of indivisible computation-intensive tasks 𝒜m(t) at time slot t, following a Poisson arrival process. A task queue Qm(t) is maintained with initial state Qm(0)=∅ to store pending tasks. The offloading decision is represented by amu(t)∈{0,1}, where amu(t)=1 indicates the task is offloaded to UAV u, and otherwise it is processed locally. Each terminal operates in a single-threaded mode, processing one task at a time and connecting to at most one UAV.

Let Tm,locwait(t) denote the waiting time of task lm(t) in the queue, representing the number of time slots the terminal must wait before processing begins, with Tm,locwait(1)=0. The calculation is as follow: (4)Tm,locwait(t)={⌈maxt′∈{1,2,⋯,t−1}Tmloc(t′)−t+1⌉}+where Tmloc denote the completion time of the task being processed. The start time of a new task is determined by the maximum completion time among all its preceding tasks in the queue. The operator {e}+ is an indicator function ensuring the waiting time remains non-negative, as detailed in Eq. (5): (5){e}+={e,if e>00,if e≤0

(2) UAV queue model

When a terminal’s computational capacity is insufficient, tasks are offloaded to UAVs. Each UAV u maintains a separate queue Qm,uuav for each associated terminal m, receiving offloading requests within its coverage at time slot t. The waiting time of task lm(t) in the UAV queue, Tm,uwait(t), counts the time slots before processing starts, with Tm,uwait(1)=0. It is calculated as follows: (6)Tm,uwait(t)={⌈maxt′∈{1,2,⋯,t−1}Tm,uuav(t′)−t+1⌉}+where Tm,uuav is the processing completion time of the task in the UAV u.

(1) G2A communication model

The Ground-to-Air (G2A) link may be blocked by buildings. Therefore, the channel model accounts for both Line-of-Sight (LoS) and Non-Line-of-Sight (NLoS) conditions to better reflect real-world propagation. The path loss between terminal and UAV is thus given by: (7)Lossm,u(t)=20log10⁡(4πfcdm,u2+H2c)+pm,uLoS(t)ηm,uLoS(t)+(1−pm,uLoS(t))ηm,uNLoS(t)where dm,u is the horizontal distance between the terminal and the UAV, fc is the carrier frequency, and c denotes the speed of light. ηm,uLoS and ηm,uNLoS represent the additional losses for LoS and NLoS links, respectively. The probability of a LoS link, pm,uLoS, is determined using an empirical statistical model, as follows: (8)pm,uLoS=11+aexp⁡(−b[180πarctan⁡(Hdmu)−a])where the value of (a,b) is set according to the specific communication environment.

In summary, the data transmission rate between the terminal m and the UAV u can be expressed as: (9)rm,u(t)=Bm,ulog2⁡(1+Pmσ210Lossm,u(t)/10)where Pm is the transmission power of the terminal device and σ2 is the noise power. Bm,u=BuNu(t) denotes the channel bandwidth allocated to the terminal m, where Bu represents the UAV bandwidth, and Nu(t) denotes the number of terminals covered by the UAV u at the time of t.

(2) A2B communication model

The Air-to-Base Station (A2B) link may be obstructed by obstacles like high-rise buildings; thus, both LoS and NLoS conditions are considered to accurately model the channel. The path loss between UAV u and base station bs is expressed as: (10)Lossu,bs(t)=20log10⁡(4πfcdu,bs2+H2c)+pu,bsLoS(t)ηu,bsLoS(t)+(1−pu,bsLoS(t))ηu,bsNLoS(t)

The transmission rate is denoted as: (11)ru,bs(t)=Bu,bslog2⁡(1+Puσ210Lossu,bs(t)/10)where Pu is the transmission power of the UAV and σ2 is the noise power. Bu,bs=BbsNbs(t) denotes the channel bandwidth allocated to the UAV u, Bbs represents the base station bandwidth, and Nbs(t) denotes the number of UAVs served by the base station at the time of t.

(1) Local computing model

When terminal m processes a task locally, the total response delay consists of the local waiting time and processing time, as calculated by Eq. (12): (12)Tmloc(t)=Tm,locwait(t)+Tm,locproc(t)where Tm,locproc(t) indicates the processing time of task lm(t), determined by the terminal’s computing power and required computation, calculated as: (13)Tm,locproc(t)=⌈dm(t)⋅ζm(t)fm⋅τ⌉

f_m is the computing resources of the user terminal.

Local task processing consumes energy dependent on the allocated CPU frequency, calculated as: (14)Emloc(t)=kloc⋅Tm,locproc(t)⋅(fm)3where kloc is the energy consumption factor of the terminal device.

(2) Offloading to the UAV computing model

When a terminal cannot process a task locally, it offloads it to its associated UAV. At time slot t, tthe set of tasks offloaded from terminal m to UAV u is denoted by Am,u(t). The total UAV processing latency includes transmission, queuing, and computation delays. Since UAVs fly at low altitudes near terminals, propagation delay is neglected. The response delay for UAV-offloaded tasks is calculated as follows: (15)Tm,uuav(t)=Tm,utrans(t)+Tm,uwait(t)+Tm,uproc(t)where Tm,utrans(t)=⌈dm(t)rm,u(t)⋅τ⌉ denotes the number of time slots needed to transmit the task data from the terminal to the UAV. The UAV’s task processing time is calculated as: (16)Tm,uproc(t)=⌈dm(t)⋅ζm(t)fm,uuav(t)⋅τ⌉

fm,uuav(t) denotes the computational resources allocated by the UAV u to task lm(t).

When a task is offloaded to the UAV, the system energy consumption comprises the user terminal’s transmission energy Em,utrans(t) and the UAV’s computation energy Em,uuav(t), which can be calculated as follows: (17)Em,utrans(t)=Pm⋅dm(t)rm,u(t) (18)Em,uproc(t)=kuav⋅Tm,uproc(t)⋅(fm,uuav)3where kuav denotes the UAV energy consumption coefficient. Then the total system energy consumption in t time slot can be expressed as: (19)Em,uuav(t)=Em,utrans(t)+Em,uproc(t)

(3) Offloading to the base station

When a task’ computational demand exceeds the UAV’s capacity, it is offloaded to the ground base station. Assuming sufficient base station resources and immediate processing, the total delay includes UAV-to-base station transmission and base station processing times. (20)Tm,u,bsbs(t)=Tm,u,bstrans(t)+Tm,u,bsproc(t)where Tm,u,bstrans(t)=⌈dm(t)ru,bs(t)⋅τ⌉ represents the number of time slots required for transmitting task data from the UAV to the base station is represented accordingly. The processing time at the base station is calculated as follows: (21)Tm,u,bsproc(t)=⌈dm(t)⋅ζm(t)fm,u,bsbs(t)⋅τ⌉

fm,u,bsbs(t) represents the compute resources assigned by the base station to the task lm(t).

When a task is offloaded to the base station, the system energy consumption consists of the UAV’s transmission energy Em,u,bstrans(t) and the base station’s computation energy Em,u,bsbs(t), which are calculated as follows: (22)Em,u,bstrans(t)=Pu⋅dm(t)ru,bs(t) (23)Em,u,bsproc(t)=kbs⋅Tm,u,bsproc(t)⋅(fm,u,bsbs)3where kbs denotes the energy consumption coefficient of the base station. Thus, the total system energy consumption in time slot t can be expressed as: (24)Em,u,bsbs(t)=Em,u,bstrans(t)+Em,u,bsproc(t)

To minimize end-to-end task delay and overall energy consumption, this paper proposes a joint optimization framework integrating UAV trajectory planning, task offloading, and resource allocation. Based on the offloading decision amu(t), tasks are processed locally or offloaded to UAVs, which may further offload them to the base station. The total system delay Ttotal(t) and energy consumption Etotal(t) at time slot t are formulated as follows: (25)Tmtotal(t)=∑m∈ℳ∑u∈𝒰[(1−amu(t))⋅Tmloc(t)+amu(t)⋅min(Tm,uuav(t),Tm,u,bsbs(t))] (26)Emtotal(t)=∑m∈ℳ∑u∈𝒰[(1−amu(t))⋅Emloc(t)+amu(t)⋅min(Em,uuav(t),Em,u,bsbs(t))]

The optimization objective minimizes a weighted sum of delay and energy consumption: (27)P1:minx,a,f∑t∈T(WTTtotal(t)+WEEtotal(t))s.t.C1:0≤ϕu(t)≤2π,∀t∈T,u∈𝒰C2:0≤xu(t)≤Xmax,0≤yu(t)≤ΥmaxC3:amu(t)∈{0,1},aubs(t)∈{0,1},∀m∈ℳC4:Tmtotal(t)≤ζm(t)C5:fm,uuav(t)≤fu, fm,u,bsbs(t)≤fbsC6:‖vu(t)−vu′(t)‖≥Dsafe,u≠u′where x denotes the UAV trajectory control decision, a represents the task offloading decision, and f is the resource allocation decision. C1 and C2 define the UAVs’ flight direction and range; C3 governs task offloading decisions at each time slot; C4 ensures task execution time does not exceed its maximum tolerable delay; C5 restricts allocated resources to not exceed the total available at the UAV and base station, and C6 enforces a minimum safe distance between UAVs to prevent collisions.

P1 is non-convex due to the coupling of hybrid discrete-continuous variables and nonlinear relationships among task offloading, resource allocation, and UAV trajectory, making it intractable for conventional optimization methods. To address this challenge, this paper introduces the PER-MATD3 algorithm, which leverages deep reinforcement learning to efficiently learn near-optimal solutions in dynamic environments.

The optimization problem (P1) is modeled as a Markov Decision Process (MDP) ⟨S,A,P,r,γ⟩ where S and A denote the system state and action spaces. P the state transition probabilities, r the immediate reward, and γ the discount factor.

(1) State space

User terminals observe local task information and the position of the UAV they are currently associated with, represented as sM(t)={vu(t),Qm(t),Am(t)}, where vu(t) is the accessed UAV’s position, Qm(t) the terminal’s task queue, and Am(t) newly generated tasks. UAVs observe their own and other UAVs’ positions, task queues, and new tasks: sM+U(t)={vu(t),vu′(t),Qm,uuav,Am,u(t)}. The joint state space of the system in time slot can be denoted as s(t)=sM(t)∪sM+U(t).

(2) Action space

Terminals decide task offloading aM(t)={amu(t)}. UAV actions include trajectory control, resource allocation, and offloading: aM+U(t)={ϕu(t),δu(t),aubs(t),fm,uuav(t),fm,u,bsbs(t)}. Therefore, the joint action of the system in the time slot can be denoted as a(t)=aM(t)∪aM+U(t).

(3) Reward function

The reward guides agents to minimize total system cost under constraints. If all constraints are met, the reward equals the negative cost; otherwise, a penalty is applied. Formally: (28)r(t)={−(WTTtotal(t)+WEEtotal(t)),if all conditions are met−η1−η2,otherwise

When the flight trajectory of the UAV violates C1∼C3, it will get the penalty η1; if the task is discarded or exceeds the maximum tolerance time, it will get the penalty η2.

In multi-agent UAV-assisted MEC networks, the high-dimensional continuous action space challenges traditional RL methods like Q-learning, DQN, and PG. To address this, we propose PER-MATD3, a prioritized experience replay extension of MATD3. Building on TD3’s twin critics, target policy smoothing, and delayed updates to reduce overestimation and improve stability, PER-MATD3 prioritizes samples with high TD errors to accelerate convergence and enhance performance in complex multi-agent settings.

Leveraging centralized training with decentralized execution (CTDE), critics use global information to overcome partial observability, while actors act on local observations for scalability. Dual-delay updates and prioritized replay further stabilize training and improve adaptability in high-dimensional, continuous, and collaborative environments.

The PER-MATD3 framework, illustrated in Fig. 2. Each agent i, where i∈ℳ∪𝒰, equips with an Actor network πi(si,ai;θi), Qi1(si,ai;ϕi1) and two Critic networks Qi1′(si,ai;ϕi1′), Qi2′(si,ai;ϕi2′), with corresponding target networks πi′, Qi1′, and Qi2′, where θ and ϕ are the parameters of the Actor and Critic networks, respectively. During centralized training, the Critic networks utilize global states S(t) and actions A(t) from all agents to compute rewards, while decentralized execution relies on local observations. The Actor network for each user terminal receives inputs sm(t)={vm(t),Qm(t),Am(t)},∀m∈M, representing the accessed UAV location, local task queue, and new task information. For UAV agents, the input is su(t)={vu(t),vu′(t),Quuav,Au(t)},∀u∈U, including its own and neighboring UAV locations, task queue, and offloaded task information. Each agent outputs an action ai(t) based on its observation. The Critic evaluates the action ai(t) using local observations, providing Q-values for policy updates.

Actor networks are optimized via policy gradient methods during centralized training, with the policy gradient computed as follows: (29)∇θJ(θ)=1L∑l=1L∇θπi(sil)∇ailQi1(sl,al)|ail=πi(sil))where L denotes the mini-batch size sampled via prioritized experience replay. Each agent’s two Critic networks independently estimate Q-values, and the minimum is used to enhance stability and accuracy. Critic training follows the Temporal-Difference (TD) principle, leveraging the same prioritized replay mechanism as the Actor. The TD target for each sample is computed as: (30)yl=γmink∈{1,2}{Qik′(sil,al),Qik′(sil,a¯l}+rlwhere rl denotes the immediate reward, γ is the discount factor balancing immediate and long-term rewards, and Qik′(si′l,a¯l) the target Q-value, where the minimum of the two Critic outputs is used to reduce overestimation. Prioritized experience replay assigns sampling probabilities based on the TD error magnitude, favoring samples with larger errors to improve learning efficiency and convergence. Each experience is assigned a priority pl related to its TD error, and importance sampling (IS) is incorporated. The priority pl is defined as: (31)pl=|yl|+εwhere ε prevents zero priority. Sampling probability is given by: (32)Pl=(pl)∂∑dD(pd)∂where D is the replay buffer size, ∂∈[0,1] controlling sampling randomness and bias. The importance sampling (IS) weight is: (33)cl=(1D⋅1Pl)ðwhere ð∈[0,1] adjusts IS correction strength. Finally, weights are normalized as: (34)c^l=clmaxl∈{1,2,…,L}cl

To mitigate value overestimation caused by sharp fluctuations in the Actor’s output for state sil, random noise is added to the action to produce a smoother and more stable target, calculated as: (35)a¯l=πi′⁡(si′l)+εwhere πi′(si′l) is the action output by the target Actor network in the next state. ε ∼clip(N(0,δ2),−1,1) is the added smoothing noise obeying Gaussian distribution.

Critic network parameters are updated using Mean Squared Error (MSE) loss, and the loss function is defined as follows: (36)L(ϕik)=1L∑l=1Lc^i⋅[yl−Qik(sil,ail)]2,k∈{1,2}

Therefore, the parameters of the Actor and Critic main network of each agenti are updated in the following way:(37){θi←θi+λ∇θJ(θi)ϕik←ϕik−λ∇ϕikL(ϕik),k∈{1,2}where λ denotes the learning rate, controlling the step size of gradient updates to enhance training stability.

PER-MATD3 uses delayed policy updates: Critics update every step for rapid adaptation, while Actors update every d steps to stabilize policy changes from value bias. Target networks are softly updated to enhance convergence and robustness, as follows:(38){θi′←ρθi+(1−ρ)θi′ϕik′←ρϕik+(1−ρ)ϕik′,k∈{1,2}where ρ denotes the soft update factor of the target network. The algorithm pseudo-code of PER-MATD3 is shown in Algorithm 1.

The computational complexity of PER-MATD3 includes two phases: interaction with the environment and training with prioritized experience replay. The interaction phase with M + N agents over T time slots costs 𝒪((M+N)⋅T). Training involves prioritized sampling with complexity 𝒪(L⋅log⁡|𝒟|) and actor-critic updates across all agents at cost 𝒪((M+N)⋅L⋅NP), where NP is the neural network size. Compared to standard MATD3, which uses uniform replay with 𝒪(L), PER-MATD3 introduces an additional 𝒪(L⋅log⁡|𝒟|) overhead due to priority management. Therefore, the total complexity is 𝒪((M+N)⋅T+L⋅log⁡|𝒟|+(M+N)⋅L⋅NP).

In order to verify the effectiveness of the algorithm proposed in this paper, we consider a UAV-assisted MEC scenario consisting of three UAVs and multiple ground user terminals and an edge base station. Among them, the UAVs fly in an area of 400 × 400 m2 to provide computational offloading services for the user terminals in the coverage area, and each user terminal randomly generates delay-sensitive computational tasks at the beginning of each time slot. Each task is associated with a maximum tolerable delay. If a task cannot be scheduled for execution before this deadline due to resource constraints, it is deemed infeasible and discarded. The experimental parameter settings refer to [9,18], and the main experimental parameters are shown in Table 1.

In this paper, three classical reinforcement learning algorithms—MATD3 [19], MADDPG [20], and PPO [21]—are used as baseline benchmarks. MATD3 extends TD3 to multi-agent settings, improving stability and reducing overestimation in continuous action spaces, making it suitable for coordinated UAV task offloading. MADDPG enables centralized training with decentralized execution, allowing multiple UAVs to learn cooperative strategies. PPO is a single-agent policy gradient method that provides stable and efficient updates, serving as a baseline for independent UAV scenarios. We compare these algorithms under the same simulation settings in terms of convergence speed, training stability, and average system task latency and energy consumption, providing a comprehensive evaluation of the proposed method.

Fig. 3 presents the training convergence curves of the compared algorithms in terms of average episode reward. PER-MATD3 achieves rapid improvement in the early training phase and stabilizes after approximately 1000 episodes, demonstrating fast convergence and high policy stability. In contrast, MATD3 converges more slowly, while MADDPG and PPO continue to exhibit significant oscillations beyond 1000 episodes, indicating poorer learning stability. This improved convergence is attributed to the integration of prioritized experience replay and the dual-Q network architecture, which together enhance learning efficiency and mitigate value overestimation.

Fig. 4 compares the performance of different algorithms under varying numbers of user terminals. Fig. 4a depicts the average task completion delay of each algorithm as the number of user terminals increases from 20 to 100. With more terminals generating delay-sensitive tasks, task completion delay rises for all methods due to limited computational and communication resources. Nevertheless, the proposed PER-MATD3 consistently achieves the lowest delay, demonstrating superior decision efficiency. Specifically, when the number of terminals reaches 100, PER-MATD3 reduces the average delay by 22.7% compared to MATD3. Notably, beyond 60 terminals, PER-MATD3 maintains low and stable latency, while other algorithms exhibit significant delay spikes, indicating better robustness and scalability under heavy load. Fig. 4b shows the system’s total energy consumption with increasing user terminals. As the number of user terminals increases from 20 to 100, the system’s total energy consumption gradually rises due to higher task loads and intensified resource contention. It can be observed from the results that PER-MATD3 consistently achieves the lowest energy consumption among all methods, demonstrating its superior resource scheduling and energy efficiency. Specifically, at 100 terminals, PER-MATD3 reduces energy consumption by 20.8% compared to MATD3. This improvement is attributed to the enhanced learning efficiency of prioritized experience replay, which assigns higher sampling priority to more informative transitions, thereby accelerating convergence to energy-efficient offloading and UAV trajectory policies.

Fig. 5a depicts the deployment and trajectory planning of three UAVs in a planar coordinate system. All UAVs are launched from the center of the experimental area, and they fly in different directions to avoid inter-UAV collisions. Each UAV determines its destination by optimizing its trajectory to cover the maximum number of user terminals (UTs) within its communication range. Consequently, the UAVs converge to the centroids of three high-density UT clusters and hover there to provide stable computation offloading services. This strategy achieves efficient spatial coverage, reduces transmission delay, and improves overall system performance. Fig. 5b compares the average task delay under varying UAV computing capacities with 50 UTs. As the UAV computation capacity increases from 1 to 4 GHz, tasks are processed more faster, leading to shorter queuing delays and less offloading to the base station, which significantly reduces overall latency. PER-MATD3 consistently achieves the lowest latency. Experimental results show that PER-MATD3 achieves the best performance overall.