In current edge caching networks, the limited storage capacity at edge nodes makes it infeasible to cache all available content locally. Consequently, it is essential to design effective caching strategies to optimize overall system performance. Traditional caching research has primarily relied on fixed planning or statistical models, such as Least Recently Used (LRU), Least Frequently Used (LFU), and others [8–11]. Reference [12] maps the content feature time in the LRU strategy to the classical Coupon Collector’s Problem (CCP) to evaluate caching performance. Reference [13] introduces a joint optimization approach for edge caching and computing resources, employing a low-complexity yet high-performance algorithm to reduce network bandwidth consumption. Building on this, Zhang et al. [14] propose a learning-based collaborative caching strategy and develop a novel dynamic programming algorithm that significantly lowers the average content delivery delay. Moreover, some researchers have accounted for the randomness of network environments and signal uncertainty by modeling caching strategies using stochastic geometry. Gao et al. [15] derive an expression for the average transmission cost based on stochastic geometry and apply standard gradient methods to obtain suboptimal solutions. Compared with strategies that merely maximize the probability of successful offloading, their approach achieves a substantial reduction in transmission costs. Emara et al. [16] utilize stochastic geometry to characterize the hit probability of multi-channel support caches in small cell networks and highlight the need for caching strategies that adapt to varying network parameters and functionalities.

However, these methods predominantly focus on single-pattern content access and static optimization, which are insufficient for addressing the complexity and dynamism of real-world user behaviors and network conditions [17,18]. To overcome these limitations, recent studies have introduced deep learning techniques into edge caching research. Li et al. [19] employ Echo State Networks (ESN) and Long Short-Term Memory (LSTM) networks to predict user mobility and content popularity. Based on these predictions, they propose a deep reinforcement learning framework that dynamically determines which users should establish D2D links for content transmission, thereby enhancing system performance. Pervej et al. [20] propose an LSTM-based sequential model to capture user content preferences, proactively cache popular content, and develop a collaborative edge caching scheme that minimizes content sharing costs using a greedy algorithm. Sun et al. [21] design a DQN-based intelligent caching strategy aimed at improving energy efficiency in mobile edge networks. Chen et al. [22] apply DQN to optimize caching decisions in wireless networks, focusing on addressing the challenge of caching unknown popular content to support large-scale communications.

In summary, existing research on edge caching exhibits three main limitations: (1) Most studies only consider a single-layer edge node architecture, failing to fully exploit the potential of D2D collaborative caching [23]; (2) Traditional static optimization methods struggle to adapt to dynamic network environments and user mobility variations; (3) Current LSTM-based mobility prediction models demonstrate deficiencies in capturing long-term dependency characteristics. To address these issues, this paper proposes a Transformer-DQN collaborative optimization framework that specifically tackles these challenges by: implementing cross-layer cooperative caching through a three-tier heterogeneous network architecture, employing deep reinforcement learning to handle dynamic environmental challenges, and utilizing multi-head attention mechanisms to enhance mobility prediction accuracy.

The main contributions of this paper are as follows:

We derive the content transmission delay and cache hit rate in a D2D-enabled heterogeneous network, where user mobility is modeled using a grid-based representation. An optimization problem is formulated with the objective of minimizing content transmission delay.

As shown in Fig. 1, this paper studies the edge caching system in a three-tier heterogeneous network, which consists of an arbitrary macrocell containing a macro base station (MBS), small base station (SBS), and mobile user (MU). The MBS covers the entire cell’s SBS and MU, and can connect to the cloud server via backhaul links to retrieve requested content. It can also communicate with all SBS through wired connections. The MU can retrieve requested content from the SBS via wireless links, and within a certain region, MU can establish D2D links to access the requested content.

In this system, each user in the user set 𝒰={1,2,…,u,…,U} requests content from the content set ℱ={1,2,…,f,…,F}, with the size of content cf. The set of small base stations is defined as ℐ={1,2,…,i,…,I}. The edge nodes in the system include all users and small base stations. The caching status of all users is defined as X=[xf,u(t)]F×U, where xf,u(t)=1 if content f is cached at user u at time t, and xf,u(t)=0 if it is not cached. Similarly, the caching status of small base stations is defined as Y=[yf,i(t)]F×I, which indicates whether content f is cached at small base station i. Additionally, user locations are modeled using a grid-based representation, which is a simple and general model [24]. In this method, the area around the MBS is divided into an N×N grid, and a location prediction algorithm is used to forecast the user’s position within the network at a given time.

In the communication process, the transmission types include: MBS to SBSi(i∈{1,2,3,…,I}), SBSi to SBSj(j∈{1,2,3,…,I},j≠i), SBSi to MUu(u∈{1,2,3,…,U}), and MUu to MUk(k∈{1,2,3,…,U},k≠u). In this model, SBSi denotes the i-th small base station in the network, with a total of I small base stations, and MUu refers to the u-th mobile user, with a total of U users in the network.

End-to-End Delay consists of content transmission delay and content request delay. In specific scenarios (such as large data transmission or bandwidth-limited environments), content transmission delay dominates and becomes a key factor in optimizing system performance. Similar to the assumption in [25], we ignore content request delay and only evaluate content transmission delay. Additionally, we assume that each SBS contains multiple subchannels with the same bandwidth, and uses Orthogonal Frequency Division Multiple Access (OFDM) for multi-access technology, where interference from D2D communication is negligible. Therefore, the communication rate between SBSi and MUu in time slot t can be expressed as: (1)Ru,iu2s=Bu,iu2slog2⁡(1+piu2sgu,iu2s∑j≠ipju2sgu,iu2s+σ2)where Bi,u denotes the bandwidth allocated by SBSi to MUu, piu2s is the transmission power from SBSi to MUu, gi,uu2s is the channel gain between SBSi to MUu, and σ2 denotes the noise power, which is set to a constant value. The channel gain g can be calculated using a path loss model: (2)gu,i=du,i−αwhere di,u represents the link distance between endpoint i and endpoint u. α denotes the path loss exponent.

Based on Eq. (1), the content transmission delay for SBSi transmitting content f to MUu in time slot t can be expressed as: (3)Tf,u,iu2s(t)=cfRu,iu2s(t)

Similarly, for D2D communication, the communication rate when MUk sends content to MUu can be expressed as: (4)Ru,kd2d=Bu,kd2dlog2⁡(1+pkd2dgu,kd2d∑c≠upcd2dgu,cd2d+σ2)

Therefore, the content transmission delay when MUk sends content f to MUu in time slot t is: (5)Tf,u,kd2d(t)=cfRu,kd2d(t)

Additionally, considering the Quality of Experience, the transmission delay for each user cannot exceed the maximum tolerable threshold τmax.

We assume that the content requested by each user follows a Generalized Zipf Distribution, which is an extension of the classic Zipf distribution. By introducing an additional shift parameter q, the frequency decay pattern can be adjusted more flexibly. This is a commonly used content popularity model [26]. In fact, the accuracy of the Zipf distribution has been validated in experiments [27]. Therefore, the popularity Pf,u of content f for user u can be expressed by the following formula: (6)Pf,u=(kf,u+q)−s∑i=1F(i+q−s)where kf,u represents the popularity ranking of content f for user u, F is the total number of contents, and the shift parameter q controls the frequency decay rate for higher-ranked contents. When q=0, it corresponds to the classic Zipf distribution. The exponent s controls the steepness of the distribution; when s>1, the frequency decreases faster, and the distribution becomes more concentrated on higher-ranked contents, which is a common scenario. The Generalized Zipf Distribution offers greater flexibility, allowing for a more accurate representation of content distribution patterns by adjusting the parameters q and s.

In time slot t, based on the different edge locations of the content requested by user u, the following cases are considered:

Scenario 1: When the content f requested by user u is available on the local terminal, the content f can be directly accessed locally. In this case, the download delay for content f is close to 0 and can be ignored, denoted as Tf,u,u(t)=0.

Scenario 2: When user u does not have the requested content f cached locally, but nearby neighboring users have cached it, user u will establish a D2D link with a neighboring user to retrieve the requested content f. Additionally, when multiple users have cached content f, user u will retrieve the content from the user with the smallest transmission delay. The content download delay in this case is calculated using Eq. (5).

Scenario 3: When neither user u nor the neighboring users have cached the requested content f, but nearby SBSs have cached the content, user u will retrieve the requested content f from the SBS. In this case, the content download delay is calculated using Eq. (1).

Scenario 4: When the requested content f is not cached at the edge nodes, it will be retrieved from the cloud server via the MBS. In this case, the content download delay is mainly concentrated on the Cloud-MBS link, and the transmission rate can be treated as a constant Rcloud. Therefore, the content download delay can be expressed as Tcloud(t)=cf/Rcloud, where Tcloud(t)≫Tu,f(t).

In summary, the download delay of content f by user u in time slot t can be expressed as: (7)Tu,f(t)=min(X(t)Tu,fd2d(t))+(1−X(t))Y(t)Tu,fu2s(t)+(1−Y(t))(1−X(t))Tcloud(t)

The overall CHR of the system is the weighted sum of the hit rates of each layer, with the weights being the probability of a miss in all preceding layers: (8)Htotal=Hlocal+(1−Hlocal)HD2D+(1−Hlocal)(1−HD2D)HU2Swhere Hlocal is the cache hit rate at the user’s local cache, HD2D is the cache hit rate at the D2D layer, and HU2S is the cache hit rate at the small base station.

Based on the content popularity Formula (6), we can obtain: (9)Hlocal=1U∑u=1U∑f=1Fxf,uPf,u (10)HD2D=1U∑u=1U∑k=1,k≠uU∑f=1Fxf,uPf,k (11)HU2S=1U∑u=1U∑k=1U∑f=1Fyf,uPf,k

In summary, the overall cache hit rate of the system can be expressed as: (12)Htotal= 1U∑u=1U⁡∑f=1F⁡xf,uPf,u+1U∑u=1U⁡∑k=1,k≠uU⁡∑f=1F⁡xf,uPf,k(1−1U∑u=1U⁡∑f=1F⁡xf,uPf,u)+1U∑u=1U⁡∑k=1U⁡∑f=1F⁡yf,uPf,k(1U∑u=1U⁡∑f=1F⁡xf,uPf,u)(1−1U∑u=1U⁡∑k=1,k≠uU⁡∑f=1F⁡xf,uPf,k)

We investigate the joint optimization problem in mobile edge caching. First, we predict users’ future geographic locations based on their historical mobility data. Then, by incorporating content popularity distribution and the current cache states of edge nodes, we design a dynamic cache replacement strategy. The objective is to minimize the average content download delay across the entire caching system. According to Eq. (7), the average content download delay at time t can be expressed as: (13)Ttrans(t)= 1U∑u=1U∑i=1I∑f=1F[(minxf,k=1Tf,u,id2d(t))+(1−maxf,kxf,k(t))yf,i(t)Tf,u,iu2s(t)+(1−maxf,kxf,k(t))(1−yf,i(t))Tcloud(t)]

The optimization problem aims to minimize the total content transmission delay with n time slots (n→∞): (14)𝒫:minlimn→∞1n∑t=1nTtrans(t) s.t.C1:xf,k(t)∈{0,1},∀f∈ℱ,∀k∈𝒰 C2:yf,i(t)∈{0,1},∀f∈ℱ,∀i∈I C3:∑i=1Iyf,i(t)=1,∀f∈ℱ,∀i∈I C4:∑f=1Fxf,k(t)≤Vk,∀f∈ℱ,∀k∈𝒰 C5:∑f=1Fyf,i(t)≤Wi,∀f∈ℱ,∀i∈I C6:Tf,u,ktrans(t)≤τmaxu,∀f∈ℱ,∀u,k∈𝒰where C1 defines xf,k(t) as 0 or 1, indicating whether content f is stored at user k, C2 defines yf,i(t) as 0 or 1, indicating whether content f is stored at small base station i, C3 ensures that content f is not stored redundantly at small base stations, C4 restricts the cached content for each user to not exceed the cache capacity Wi, C5 restricts the cached content for each small base station to not exceed the cache capacity Wi, C6 ensures that the request delay for each user does not exceed the maximum tolerable delay τmaxu.

To solve the optimization problem in Eq. (14), it is essential to find the optimal decision variables. However, due to the dynamic nature of the system’s state and actions, it is challenging to collect the massive state information of the caching system in real time and formulate an appropriate caching strategy. Moreover, the problem is a mixed-integer programming problem, which has been proven to be NP-hard [28,29]. Additionally, due to the non-convexity of the feasible set, solving the problem through exhaustive or traditional methods is difficult. Therefore, we propose a DQN-based approach to address the above optimization problem, with the detailed solution presented in Section 5.

RL, as a branch of machine learning, focuses on how agents learn optimal decisions through interaction with the environment. Its core idea is to guide the agent’s behavior through trial-and-error and reward mechanisms to maximize cumulative rewards. In reinforcement learning, there are three key elements: the state space, the action space, and the reward function. We will construct a reinforcement learning model guided by minimizing content transmission delay based on these three elements.

1. State Space (S)

The state space is used to describe the overall state of the system, including user request features and cache states. At time slot t, before the user requests content, the user request features are represented as: (24)Suser(t)={GU×(U−1)D2D(t),GU×IU2S(t),PU×F(t)}where we define a matrix GU×(U−1)D2D(t) of size U×(U−1), where each element gu,k(t) represents the channel gain between user u and user k, with u≠k. Similarly, in the matrix GU×IU2S(t), each element gu,i(t) represents the channel gain between user u and small base station i, where the above channel gains are calculated using Eq. (2). Additionally, we define a matrix PU×F(t) of size U×F, where each element pu,f(t) represents the probability that user u requests content f, calculated using the popularity Formula (6).

The cache state refers to the set of contents currently cached in the edge server, which can be represented as: (25)Scache(t)={XF×U(t),YF×I(t)}where each element xf,u(t) in XU×F(t) represents the cache state of content f at user u, with xf,u(t)∈{0,1}, and each element xf,i(t) in YF×I(t) represents the cache state of content f at small base station i, with yf,i(t)∈{0,1}.

In summary, the state space of the entire system can be represented as: (26)S(t)={GU×(U−1)D2D(t),GU×IU2S(t),PU×F(t),XF×U(t),YF×I(t)}

2. Action Space (A)

The action space defines the actions that the agent can take at each time step. In the edge caching scenario, the actions mainly include the decision to replace cached content. The action space can be represented as: (27)A(t)={𝒜F×UD2D(t),𝒜F×IU2S(t)}where the elements af,u in the F×U matrix 𝒜F×UD2D represent whether content f cached at user u should be replaced, and the elements af,i in the 𝒜F×IU2S represent whether content i cached at small base station i should be replaced by content f. The replacement priority is based on the current popularity of the content among users, giving preference to replacing content with lower popularity.

3. Reward Function (R)

The reward function is used to evaluate the performance of the agent after executing an action. Based on the cache state S(t), when a user requests content, the user and the small base station take the corresponding action A(t) and receive different reward values R(t) based on the outcomes. To minimize content transmission delay and ensure convergence to the ideal result, the reward function is defined as follows: (28)R(t)={1Ttrans(t)−1Trandom,R(t)>00,R(t)≤0where the average transmission delay Ttrans(t) is calculated using Eq. (13), and Trandom is the average transmission delay for random caching.

For the edge caching problem we are studying, traditional RL methods can no longer meet the requirements due to the complexity of the state and action spaces. We adopt a model-free reinforcement learning algorithm, DQN, which combines deep neural networks (DNN) with Q-learning to estimate the action-value function in RL.

In this study, experiments are conducted using the real-world GeoLife dataset, which contains GPS trajectory data of 182 users from April 2007 to August 2012 [33], including latitude, longitude, and altitude information. We first preprocess the dataset as follows: (1) Only low-speed scenarios are considered, and points with speeds exceeding 20 km/h are removed, (2) Static points are filtered by removing data points where the user stayed in the same location for more than 30 min, (3) The trajectory is standardized to a fixed time interval of 30 s, with missing values filled using linear interpolation.

We center the MBS and divide a 200 m × 200 m area into a 20 × 20 grid. The input dimension for the Transformer algorithm is set to 64, with 4 encoder layers, 4 attention heads, and a hidden layer size of 128. To evaluate the effectiveness of the algorithm, we compare the position prediction results with those obtained using the LSTM algorithm.

As shown in Fig. 2, the location prediction results for 50 users using different models are presented. It can be observed that, compared to the LSTM-based predictions, the Transformer-based predicted locations align much more closely with the actual locations. This is due to the self-attention mechanism of the Transformer, which can capture relationships at any position within the trajectory simultaneously. This demonstrates the effectiveness of the self-attention mechanism in location prediction. Furthermore, because the self-attention mechanism allows for parallel computation, the training speed of the Transformer is more than twice as fast as that of LSTM, given the same data volume.

After completing the mobility prediction, the next access location and other features are used as inputs to explore the caching strategy. The proposed caching strategy is then evaluated and tested for its performance in mobile edge networks. We focus on evaluating steady-state worst-case scenarios, while future work could incorporate dedicated burst detection modules to further enhance transient robustness. First, a scenario with a three-layer heterogeneous network is set up. The scenario includes one MBS, three SBSs, and 10 users. The coverage radius of the cellular network is set to 200 m, with the MBS positioned at the center and the remaining SBSs evenly distributed. We assume a total of 100 content items, with each SBS able to cache up to 8 files, each user able to cache up to 3 files, and all files are of equal size (10 MB each). The content popularity distribution follows a Zipf distribution. Based on conservative design principles, we assume that popular content undergoes random variations at every moment. The MBS transmission power is set to 50 W, the transmission bandwidth to 50 MHz, the SBS transmission power to 5 W, and the transmission bandwidth to 10 MHz. The MU transmission power is set to 0.25 W, with a transmission bandwidth of 1 MHz. The path loss factor is set to 3.5, and the noise power is fixed at 10⁻⁹ W. Additionally, the network parameters for the DQN algorithm are set as shown in Table 1.

We evaluate the performance of the DQN-based cache strategy by comparing it with other benchmark cache strategies, as follows:

FIFO (First In, First Out): When the MU (Mobile User) or SBS (Small Base Station) cache is full, the content that was stored the longest is replaced with the current user’s requested popular content.

As shown in Fig. 3, we evaluate the content transmission delay under the proposed DQN-based caching strategy compared with other baseline caching methods. It can be observed that, after a certain number of training steps, the transmission delay of the DQN algorithm gradually decreases and eventually converges to approximately 23 ms. Compared to other caching strategies, the proposed method significantly reduces the system’s transmission delay. Moreover, compared to the scenario without considering user mobility, the transmission delay is reduced by approximately 17.2%. This improvement is attributed to the fact that, without mobility awareness, the caching strategy relies on users’ historical location information. However, users’ positions change over time, leading to a mismatch between the input location and the actual location at the next time step, which in turn causes higher transmission delays.

As shown in Fig. 4, we further evaluate the CHR of the DQN-based caching strategy in comparison with other baseline caching strategies, as well as the CHR of the DQN strategy without location prediction. It can be observed that, after sufficient training, the CHR achieved by the DQN algorithm significantly outperforms that of traditional caching strategies, converging to approximately 55%. In fact, experimental results show that under the same conditions, the optimal caching strategy achieves a CHR of around 61%, indicating that the CHR of the DQN-based method is already close to the optimal performance. Furthermore, considering user mobility improves the CHR by approximately 5%.

Fig. 5 compares the content transmission delay of five caching methods under different numbers of users. With the same number of users, it can be observed that the proposed DQN-based method outperforms the other three caching methods, and the DQN method considering user mobility performs better than the one without mobility consideration. As the number of users increases, the content transmission delay decreases because the user group caches more popular content, and more popular content is transmitted via D2D, reducing reliance on the remote core network, thus significantly lowering the content transmission delay. However, as the number of users continues to increase, a critical point of resource competition is reached, and the content transmission delay begins to rise. This is primarily due to the limited number of available orthogonal subchannels for D2D communications. When excessive concurrent D2D connections occur, co-channel interference emerges, consequently increasing communication latency in D2D links. Moreover, since the DQN-based method dynamically allocates strategies and balances the processing capabilities of edge nodes at different layers, the increase in content transmission delay is slower compared to other caching methods. This indicates that the advantage of the DQN-based method becomes more pronounced when the number of users is large compared to other caching strategies.

Fig. 6 compares the content transmission delay performance of the five methods under different SBS cache capacities. With the same cache capacity, the DQN-based method outperforms the other three methods, and the DQN algorithm considering user mobility performs the best. As the SBS cache capacity increases, the content transmission delay for all algorithms decreases because a larger cache can store more popular content, and user requests are directly served at the edge, without needing to retrieve data from the cloud via the backhaul link. Moreover, when the cache capacity reaches a certain level, the rate of reduction in content transmission delay slows down for all five methods. This is because once the cache covers 90% of the most popular content, further increases in capacity have a minimal impact on content transmission delay.

As shown in Table 2, we evaluated the experimental results of the DQN algorithm under different hyperparameters. The results demonstrate that DQN hyperparameter configurations significantly impact content transmission delay and training efficiency. The initial exploration rate of 0.8 achieved better latency performance compared to the 0.2 scheme, but required 78% more training iterations, revealing a trade-off between exploration capability and computational cost. The exploration rate decay factor of 0.995 reduced performance fluctuations by 17.5% compared to the 0.99 scheme, validating the importance of moderate decay. A learning rate of 0.001 achieved the optimal balance between efficiency and stability, outperforming extreme settings. The target network update interval of 40 steps improved performance by 2.4%, but doubled the training cost. The optimal combination was found to be: initial exploration rate 0.8, decay 0.995, minimum exploration rate 0.01, learning rate 0.001, and update frequency 40, which delivered the best performance despite higher training costs. For scenarios requiring a balance with efficiency, an alternative configuration with an initial exploration rate of 0.2 and update frequency of 10 could be adopted. The study reveals that exploration strategy has the greatest impact on performance, while target network updates and learning rate primarily influence training stability.