As an emerging technology, DL has been used in several research studies to improve RA for terrestrial networks. For instance, the authors in [12] investigated the deep reinforcement learning (DRL)-based time division duplex configuration to allocate radio resources dynamically in an online manner and with high mobility. In [1], Lee et al. proposed deep power control based on a convolutional neural network to maximize spectral efficiency (SE) and energy efficiency (EE). In this study, a comparison between the DL model and a conventional weighted minimum mean square error was realized. In the same context, [13] performed a max-min and max-prod power allocation in downlink massive MIMO. To maximize EE, a deep artificial neural network scheme was applied in [14], where interference and system propagation channels were considered. Deep Q-learning (DQL)-based RA has also attracted much attention in recent literature. In [15], the authors studied the RA problem to enhance EE. The proposed method formulated a combined optimization problem, considering EE and QoS. More recently, a supervised DL approach in 5G multitier networks was adopted in [16] to solve the joint RA and remote–radio–head association. For this model, efficient subchannel and power allocation were used to generate training data. According to the decentralized RA mechanism, the authors in [17] developed a novel decentralized DL for vehicle-to-vehicle communications. The main objective was to determine the optimal sub-band and power level for transmission without requiring or waiting for global information. The authors used a DRL-based power control to investigate the problem of spectrum sharing in a cognitive radio system. The aim of this framework is that the secondary user shares the common spectrum with the primary user. Instead of unsupervised learning, the authors in [18] introduced supervised learning to maximize the throughput of device-to-device with maximum power constraint. The authors in [19] presented a comprehensive approach and considered DRL to maximize the total network throughput. However, this work did not include EE for optimization. Majority of the learning algorithms introduced above do not incorporate constraints directly into the training cost functions. Nowadays, literatures focus on RA in UAV-assisted cellular networks based on artificial intelligence. In reference [20], authors proposed a multiagent reinforcement learning framework to study the dynamic RA of multiple UAVs. The objective of this investigation was to maximize long-term rewards. However, the work did not consider UAV height. In [21], the authors used deep Q-network to solve the RA for UAV-assisted ultradense networks. To maximize system EE, a link selection strategy was proposed to allow users to select the optimal communication links. The authors did not consider the influence of SE on EE. In addition, the authors in [22] studied the RA problem of UAV-assisted wireless-powered Internet of Things systems, aiming to allocate optimal energy resources for wireless power transfer. In [23], the authors thoroughly investigated deep Q-network (DQN), invoking the difference of convex-based optimization method for multicooperative UAV-assisted wireless networks. This work assumed beamforming technique to serve users simultaneously in the same spectrum and maximize the sum user achievable rate. However, the work was not focused on EE. Another DQN work in [24] was presented to study the low utilization rate of resources. A novel DQN-based method was introduced to address the complex problem. The authors in [25] analyzed RA for bandwidth, throughput, and power consumption in different scenarios for multi-UAV-assisted IoT networks. On the basis of machine learning, authors considered DRL to address the joint RA problem. Although the proposed approach remained efficient, it did not consider ground network, EE and total throughput. In the present study, we aim to optimize EE and total throughput in UAV-assisted terrestrial networks subject to the constraints on transmission power and UAV height. Our main efforts are to apply a double deep Q-network (DDQN) that obtains optimal rewards better than DQN [26].

Existing research on RA in UAV-assisted 5G networks focuses on single objective optimization and considers the DQN algorithm to generate data. Following the previous analysis, we investigate the RA problem in UAV-assisted cellular networks that maximize EE and total network throughput. Especially, DDQN is proposed to address intelligent RA. The main contributions of this study are listed below.(1)

We formulate EE and total throughput in mmWave scenario while ensuring the minimum QoS requirements for all users according to the environment. However, the optimization problem is formulated as a mixed integer nonlinear program. Multiple constraints, such as the path loss model, number of users, channel gains, beamforming, and signal-to-interference-plus-noise ratio (SINR) issues, are used to describe the environment.

The remainder of this paper is organized as follows: An overview for DRL is presented in Section 2, and the system model is introduced in Section 3. Then, the DDQN algorithm is discussed in Section 4, followed by simulation and results in Section 5. Lastly, conclusions and perspectives are drawn in Section 6.

As a popular branch of machine learning, Q-learning is based on the main concept of the action value function qπ(s, a) for policy  π . It uses the Bellman equation to learn and calculate the optimum values of the Q-function in an iterative way [30], which is expressed as(1) Q(st, at)←Q(st, at)+αt[rt+1+γ αt+1maxQ(st+1, αt+1)−Q(st, at)]

where αt is the step-size parameter that defines the extent to which the new data contribute to the existing Q value, γ is the MDP discounter factor, rt+1 is the numerical reward for the agent after the execution of the action, and st+1 indicates that the environment changes to a new state, with transition probability p(s′, r|s,  a) , as illustrated in Fig. 2. However, the Q-learning algorithm could be applied only to RA problems with low dimensionality in state and action, resulting in an evolutionary limitation [31]. Moreover, this application is only used when the state and action spaces are discrete (e.g., channel access) [32].

As stated above, the Q-learning algorithm faces difficulties in obtaining the optimal policy when the action and state spaces become exceptionally large [33]. This constraint is often observed in the RA approaches of cellular networks. To solve this problem, the DQN algorithm, which connects the traditional Q-learning algorithm to a convolutional neural network, was proposed [34]. The main difference with the Q-learning algorithm is the replacement of the table with the function approximator called DNN; this process attempts to approximate the Q values. Approximators have two types: a linear function and a nonlinear function [35]. However, in a nonlinear DNN, the new Q function is defined as  Q(st, at|ω)≈Q∗(s, a) , where ω represents the weights of the neural network. At each time  t , action at is taken in accordance with the ϵ -greedy policy, and the transition tuple ( st, at, rt, st+1) is stored mainly in a replay memory denoted by D. During the training process, a minibatch is sampled randomly from experience D to optimize the mean squared error. Thus, the target Q-network is used to improve the stability of DQN, whose ω is regularly adjusted to follow those of the principal Q-network. On the basis of the Bellman equation, the optimal state-action function is given by [35](2) Q∗(st, at)=Est′[r+γat′maxQ∗(st′, at′)|st, at].

To train the DQN, iterative updating of the weight ω is used, thus minimizing the mean squared error of the Bellman equation. Mathematically, the loss function at each iteration is given by(3) L(ωt)=Est,at,rt,st+1∈D[rt(st, at)+γat+1maxQ(st+1, at)|ω′−Q(st, at)|ω]2

The channel between the base station and UE can be fixed or time varying. Fading is defined as the fluctuation in received signal strength with respect to time, and it occurs due to several factors, including transmitter and receiver movement, propagation environment, and atmospheric condition. Similar to [38], we model the channel in a way that it can capture small-scale and large-scale fading. At each time slot  t , the small-scale fading between UAVs, SBSs, and UEs is considered frequency-selective fading, whose objective is to obtain a delay spread greater than the symbol period. By contrast, the channel in every subcarrier is supposed to be flat fading. This combination means that the channel gains can remain unchanged. All UEs periodically transmit their channel quality information to the related BS. In addition, let hBS,k,n designate the channel gain from BS to user k on different subcarriers  n . A binary variable φ is introduced to define the association mode. If UE is associated with the UAV/SBS according to LoS link, then  φ=0 ; otherwise  φ=1 . We apply the following assumption in formulation. The mm-wave signal is affected by various factors, such as buildings in urban areas, making the link susceptible to effect blockage. Thus, the downlink achievable throughput (data rate) of user k on the nth subcarrier can be given by the following equation as(4) RBS,k,n=Plink,UAVRUAV,k,n+Plink,SBSRSBS,k,n (5) RBS,k,n=|(φ−PLoS,UAV)|BUAVlog2(1+SINRUAV,k,n)+|(φ−PLoS, SBS)|BSBSlog2(1+SINRSBS,k,n) 

where  PLoS,UAV , PLoS, SBS are the blockage probabilities when the link between the UAV/SBS and UE is LoS; they are expressed as [39,40](6) PLoS,UAV(z)=11+b+exp(−c(180πtan−1(Hz)−b))

where b and c are constants that depend on the network environment, and z=R2−H2 is the Euclidean distance between the typical UE and UAV (see Fig. 4).(7) PLoS,SBS=1−eβd where β is the blockage parameter that defines the average size of obstacles. Here, d corresponds to the distance between SBS and UE.

Adding additional gain to the system remains necessary due to the propagation losses that occur at mm-wave frequencies. One of the main solutions proposed by several research for future wireless networks is beamforming [41]. The fundamental principle of beamforming is to control the direction of a wavefront toward the UE. According to [42], UAV and SBS serve UE through beamforming technology. In this manner, the SINR of UE from UAV at time slot t can be written as(8) γUAVt=PUAVthUAV,k,ntGUAVtPLUAVtISBS+IUAV′+σ2 

where  GUAVt  represents the directional beamforming gain for the desired link, and σ2  refers to additive white Gaussian noise. ISBS and IUAV′ represent the interference from the adjacent SBS and UAV, respectively, and are expressed as(9) ISBS=PSBSthSBS,k,ntGSBSt (10) IUAV′=PUAV′thUAV′,k,ntGUAV′t

Without loss of generality, different properties are displayed in terms of propagation. For air-to-ground communication, the path loss of LoS and NLoS links at time slot t can be experienced depending on additional path losses in LoS and NLoS links φLoSt,  φNLoSt  and path loss exponents αLoSt , αNLoSt as(11) PLUAVt={φLoSt(R2+H2)αLoSt/2  for LoS linkφNLoSt(R2+H2)αNLoSt/2  for NLoS link 

Similarly, we define the SINR when UE is associated with SBS. In this case, we adopt the standard power-law path loss model with the mean δLoSt , δNLoSt for LoS and NLoS, respectively. Hence, the path loss model can be given as(12) PLSBSt={δLoStd−aLoSt for LoS linkδNLoStd−aNLoSt for NLoS link

The SINR additional loss at the typical UE when it is connected to SBS is given by (13), where IUAV=PUAVthUAV,k,ntGSBSt and ISBS′=PSBS′thSBS′,k,ntGSBS′t are the interferences from UAV and SBS, respectively.(13) γSBSt=PSBSthSBS,k,ntGSBStPLSBStIUAV+ISBS′+σ2 

SE and EE are the key metrics to evaluate any wireless communication system. SE is defined as the efficiency capability of a given channel bandwidth. In other words, it represents the transmission rate per unit of bandwidth and is measured in bits per second per hertz. The EE metric is used to evaluate the total energy consumption for a network. It is defined as a ratio of the total transferred bits to the total power consumption. Nevertheless, EE and SE have a fundamental relationship. Let PC  be the power consumed in the circuit of the transmitter; then, EE can be given by(14) EEi∈{UAV,SBS}=SEi∈{UAV,SBS}Pi+PC where  Pi is the transmit power i∈{UAV, SBS} , which ranges  0<Pi≤Pmax ; the achievable SEi∈{UAV,SBS} of transmitter can be computed as

For UAV(15) SEUAV=log2(1+PUAVthUAV,k,ntGUAVtPLUAVtISBS+IUAV′+BUAV σ2)

For SBS(16) SESBS=log2(1+PSBSthSBS,k,ntGSBStPLSBStIUAV+ISBS′+BSBS σ2)

The proper performance of EE approaches is of paramount importance in UAV-assisted terrestrial networks because it is directly related to the choice of objectives and constraints for relevant optimization problems. In this work, we aim to optimize two specific objectives for RA, namely, the maximization of EE and throughput. From the SE perspective, the EE maximization problem can be formulated as(17) max∑k∈N⁡EEt

s.t. C1 :  0<Pi≤Pmax ∀ i∈{UAV, SBS};  ∀ UAV∈U;  ∀ SBS∈S 

   C2:Hmin≤H≤Hmax ∀ UAV∈U 

   C3:EEUAVt≥EESBSt ∀ UAV∈U;  ∀ SBS∈S 

   C4 : RBS,k,n≥RBS,k,nQos 

   C5 : γUAVt≥γUAVth ∀ UAV∈U 

   C6 : γSBSt≥γSBSth ∀ SBS∈S 

   C7 : ξ∈{0, 1} ∀ UAV∈U;  ∀ SBS∈S 

Constraint C1 means that the transmit power PUAV and PSBS must be in the interval [0, Pmax] . It specifies the upper limit of the power transmission. Constraint  C2 indicates that the UAV should be positioned between a minimum and maximum height. At higher heights, the distance between the UAV and UE increases, resulting in considerable path loss. By contrast, when the UAV is located at a certain minimum height, the NLoS conditions are recorded and may affect EE; hence, this constraint must be studied. The constraint in  C3  guarantees that the EE of UAV must be greater than that of SBS. In  C4 ,  RBS,k,n  defines the maximum downlink achievable data rate, whereas  RBS,k,nQos  accounts for the data rate requirement. Constraints  C5  and  C6  specify that the SINR of the UE should be higher than a certain threshold; the SINR threshold differs from each tier (UAV, SBS). Lastly, the last constraint ensures that the UE is connected with a single BS. In a subsequent section, we will present our second objective, which is to maximize the total network throughput. The overall throughput  RBS,k,n  is defined as the sum of the data rates that are provided perfectly to all UE. Mathematically, the maximization problem can be computed as(18) max∑k∈N⁡RBS,k,nt 

s.t. C1 :  0<Pi≤Pmax ∀ i∈{UAV, SBS};  ∀ UAV∈U;  ∀ SBS∈S

   C2 : Hmin≤H≤Hmax ∀ UAV∈U

   C3 : γUAVt≥γUAVth ∀ UAV∈U 

   C4: RBS,k,n≥RBS,k,nmin ∀ UAV∈U;  ∀ SBS∈S 

   C5 : PLoS,SBS<PLoS,UAV ∀ UAV∈U;  ∀ SBS∈S 

The constraint in C4 indicates the minimum required data rate for QoS. Here, constraint  C5  means that the LoS probability of the SBS must be less than that of the UAV.

The state describes a specific configuration of the environment. At time slot  t , UAVs and SBSs act as agents and define the observation space  Ojt . The observation of each BS=UAV∪⁡SBS includes the SINR measurement from the UAV and SBS to UE, the height of UAVs  H , and spectral efficiency. We define the global state as(20) stj=(Ot1, Ot2, …OtBS) where OtBS  represents the set of observation and can be expressed as(21) Otj=(SINRtj, Htj, SEtj)

In our problem, each agent must choose an appropriate base station (i.e., UAV or SBS), power transmission, UAV height, and LoS/NLoS link probability. At time step  t , the action of UAV/SBS can be expressed as(22) atj=(btjPtj, Htj, Ptj)

where btj∈{0, 1, …β} is the selected BS;  Ptj={0, 1, …Pmax} is the power transmission requirement, which indicates how much power should be assigned to UE; Htj={0, 1, …Hmax} is the UAV elevation.

Reinforcement learning is based on the reward function, stating that the agent (UAV and SBS) is guided toward an optimal policy. As mentioned above, we model this problem as a fully observable MDP to maximize EE and throughput. Therefore, the reward of j can be computed as(23) rtj=w1rEE+w2rThroughput where rEE=∑i=1M+u⁡EEt and rThroughput=∑i=1M+u⁡RBS,k,n . w1  and  w2  are the weights for each objective, respectively. The pseudo code for DDQN is outlined in Algorithm 1.

In this subsection, we show some results of EE, which are obtained using DDNQ. For improved performance validation, we compare our proposed algorithm with the DQN and QL architectures. Moreover, the effect of UE demand, number of UAVs, and beamforming on maximum power Pmax, UAV are discussed. In the simulation evaluation, the parameter values in Table 2 are used, unless otherwise specified. First, we evaluate the effect of UE demand on the EE for different algorithms in Fig. 6. A common observation in Fig. 6 is that increasing UE demand can lead to increased EE; however, from 60 Mbps, EE converges less quickly.

This result is obtained because when UE demand increases considerably ( 60 Mbps>) , all algorithms (DDQN, DQN, and QL) aim to maximize network throughput, which requires high transmission power, causing reduced EE. Another comment from Fig. 6 is that the DDQN algorithm can outperform DQN and QL. This outcome is achieved because the agent selects a more appropriate Q value to estimate the action. This perfection is mainly due to the two separate estimators applied in DDQN. In other terms, the use of the opposite estimator is cost effective for obtaining unbiased Q-values. A proposed solution to the EE problem when UE demand increases is to add base stations. The increase in the number of UAVs has a remarkable effect on EE, as illustrated in Fig. 7. As the number of UAVs increases, EE improves because the number of users covered in UAV LoS increases.

Moreover, Fig. 7 demonstrates that the DDQN algorithm outperforms DQN and QL by 13.3% on EE because traditional RL algorithms use a one-actor network to train multiple agents; thus, conflicts between agents are recorded. Next, EE is plotted as a function of the number of UE for different UAV height ( Hmax constraint), as shown in Fig. 8. Moreover, an increase in the number of UE results in EE degradation because of the increase in energy consumption. Fig. 8 also shows that UAV height can affect EE. Therefore, EE increases as Hmax increases because the increase in UAV height results in additional UEs in the LoS link condition, leading to an increase in the total number of bits transmitted.

As the number of UE increases, the power assigned to UE declines. Therefore, the increase in height compensates this shortcoming. Fig. 9 shows EE vs. the maximum power of UAV Pmax, UAV with and without beamforming. A common observation in Fig. 9 is that EE decreases by extending the maximum transmission power of the UAV due to the increased energy consumption by users. In addition, when the power of UAVs increases, the links between UAVs and UEs are in NLoS condition, thus reducing EE. This analysis is conducted with and without beamforming. As illustrated in Fig. 9, applied beamforming improves EE in each algorithm (DDQN and DQN) because beamforming provides additional gains and can overcome mm-wave blockage constraints.

To validate the accuracy of our approach, we analyze the total throughput (second objective) according to the number of UAVs deployed, UAV height HMax , and beamforming. Considering the first scenario, Fig. 10 depicts the total throughput as a function of the number of UAVs. As the number of UAVs increases, the total throughput is enhanced. Thus, DDQN outperforms DQN and QL. However, this effect is mainly due to the increase in LoS links. The same figure also shows that the total throughput reaches a congestion level at a particular number of UAVs due to the rise in interference between UAVs.

Fig. 11 illustrates the variation of throughput vs. UAV height in different AI algorithms. According to Fig. 11, throughput increases with maximization of altitude  HMax because at low altitude, the propagation condition is in NLoS, and interference between tiers is observed. By contrast, when the UAV height increases, the LoS condition occurs, resulting in reduced loss. Moreover, saturation is experienced from an altitude of 130 m because as UAV height increases, the distance between the UAV and UE increases, leading to signal attenuation.

Fig. 12 shows the variation of the throughput vs. maximum UAV power. As expected, the total throughput increases as PMax, UAV increases. Fig. 12 also reveals that DDQN achieves a maximum throughput of 582.7 Mbps with a maximum power of PMax, UAV=35  dBm. By contrast, DQN achieves a maximum throughput of 269,234 Mbps at the same  PMax, UAV . Again, the proposed DDQN algorithm outperforms DQN. Finally, we plot the throughput as a function of blockage parameter β for SBS when UAVs are assumed to be located at HMax=120 m, as shown in Fig. 13. When β increases, the total throughput of the network decreases. Therefore, with the increase in obstacle density, more UEs are served by NLoS conditions. In addition, Fig. 13 shows that the proposed DDQN scheme converges to highly satisfactory solutions compared with the other approaches because it handles interference perfectly.