Several studies have employed DRL to enhance edge caching efficiency. Cui et al. [8] introduced a real-time caching method that integrates DRL with Q-learning techniques to address cooperative edge caching. Their approach improves energy efficiency, adapts caching policies in real time, and minimizes replacement and interruption costs. Similarly, Sun et al. [9] proposed a decentralized framework combining recommendation systems with edge caching to optimize resource utilization via direct and soft cache hits. Leveraging multi-agent Markov decision processes and the soft actor-critic (SAC) algorithm, their method enables edge servers to independently learn and implement optimal caching policies. However, these methods mainly focus on cache hit rates or coordination efficiency, without explicitly modeling video quality or multi-version delivery trade-offs. These methods, however, largely focus on cache hit rates or cooperation, without explicitly addressing multi-version video placement or quality-aware delivery. Wu et al. [10] proposed a DRL-based content update strategy that adapts in real time to dynamic content popularity. Zhong et al. [11] presented a multi-agent DRL framework to coordinate edge caching decisions in wireless networks, showing performance improvements without prior knowledge of popularity distributions. These methods, however, largely focus on cache hit rates or cooperation, without explicitly addressing multi-version video placement or quality-aware delivery. Kwon and Song [12] proposed a deep reinforcement learning-based approach to enhance the cache hit rate by adapting to dynamic file request patterns. Unlike their work, which considered single-version content in general file caching, our approach targets utility-driven caching in video-on-demand systems, supporting multi-version video placement and quality-aware delivery under dynamic popularity.

Research has also focused on video quality optimization under server capacity constraints. Lee et al. developed an algorithm for optimizing caching and transcoding tasks in multi-access edge computing (MEC) environments, adjusting bitrates based on video popularity and retention rates while considering server capacity limitations [13]. Tran et al. proposed a caching and processing framework for video streaming in MEC systems, addressing bitrate optimization through an online iterative greedy-based adaptive algorithm to tackle the NP-hard nature of the problem [14]. These methods rely on pre-defined rules and are less adaptable to dynamically changing multi-version workloads.

Several studies have explored trade-offs among cache hit ratios, content quality, and latency. Dao et al. [15] proposed a dynamic caching and quality level selection policy in adaptive bitrate streaming, modeled as a multidimensional knapsack problem and solved using a transfer learning-based genetic algorithm. Tran et al. [16] examined caching policies for mobile data traffic management, focusing on trade-offs among hit ratios, latency, and storage efficiency, and categorized algorithms into machine learning, deep learning, and game theory-based approaches.

Several recent studies have focused on similarity caching, where an edge cache stores and delivers similar content, such as lower bitrate versions, instead of the exact requested content. Araldo et al. [17] studied optimal content placement in caching systems by selecting appropriate bitrate versions to improve video delivery performance under storage constraints. Zhou et al. [18] proposed adaptive offline and online caching algorithms that jointly optimize content placement and delivery decisions by considering content similarity and user-perceived quality. Garetto et al. [19] investigated content placement strategies in networks of similarity caches, where content placement decisions are made by leveraging content similarity to improve caching efficiency. Wang et al. [20] developed an online similarity caching algorithm based on an adversarial bandit framework to adaptively handle dynamic user requests in cooperative edge networks.

Several studies have addressed content quality adaptation and multi-version caching, where multiple bitrate versions of the same content are considered to enhance user QoE and caching efficiency. Tran et al. [21] proposed a collaborative caching and processing framework in mobile-edge computing networks to support adaptive bitrate video streaming and improve user experience under storage and network constraints. Bayhan et al. [22] proposed EdgeDASH, a network-assisted adaptive video streaming framework that improves edge caching efficiency by dynamically aligning client requests with cached content through quality adaptation.

To the best of our knowledge, this study is the first to address edge cache management in scenarios with dynamically changing content popularity using DRL. By incorporating a utility function that integrates video quality and cache efficiency, our approach provides a robust and adaptable solution for optimizing edge caching under uncertain and variable content popularity conditions. A comparison with representative related works is summarized in Table 1, highlighting the distinctions in objective, granularity, and learning methodology.

Fig. 1 illustrates the system architecture for caching video content on edge cache. Edge caching reduces network bandwidth consumption by storing popular content closer to end users, thereby decreasing data transmission through core networks, lowering latency, and improving overall network efficiency [23]. From the perspective of the edge cache provider, maximizing the bandwidth utilized by the edge cache is directly linked to the profit generated.

In this context, utility is defined as the profit accrued by the edge service provider, which is proportional to the bandwidth provided by the edge server. The edge caching system operates on the principle that higher bandwidth utilization reflects enhanced service delivery capabilities, leading to increased revenue for the provider [24]. The overarching objective is to maximize the total utility generated for the service provider while satisfying the edge cache capacity constraint.

Assume that the VoD server depicted in Fig. 1 stores Nvideo videos, each transcoded into Nver distinct bitrate versions. Each version, denoted as Vi,j, where i∈{1,…,Nvideo} and j∈{1,…,Nver}, is associated with a time-varying access probability pi,j(k). This probability is measured at regular intervals k∈{1,…,Ninterval}.

The edge server processes user requests based on cached files, with requests handled as follows:

Cache hit: If the requested video is cached, it is directly served from the edge server. This also includes cases where a lower bitrate version of the requested video is cached and served to minimize bandwidth usage [22].

Let Xi,j be a binary variable indicating whether version Vi,j is cached in the edge server, and let Xi→={Xi,j|j∈{1,…,Nver}} represent the set of caching decisions for video i. The bandwidth Bi,j(Xi→) provided for version j of video i under the caching decision Xi→ is defined as: (1)Bi,j(Xi→)={bi,j, if Xi,j=1,bi,k, if  Xi,j=0 and k=max{l∣l<j,Xi,l=1}0, if Xi,j=0 and ∀l<j,Xi,l=0..

Here, bi,j is the bitrate of version Vi,j, representing the bandwidth required for its transmission. This ensures that the requested version Vi,j is transmitted if cached, or the highest available lower version is transmitted if not cached. If no lower-quality versions are available, Bi,j(Xi→)=0, indicating no bandwidth consumption.

We consider three utility models based on the bandwidth provided by the edge cache: proportional (linear growth), convex (accelerating growth), and concave (diminishing growth) relationships. These models enable adaptability to diverse utility requirements and reflect various profit structures [25]. Let Ui,j(Xi→) represent the utility derived from the caching decisions Xi→. The utility function is defined as: (2)Ui,j(Xi→)={αBi,j(Xi→),linear model,βer⋅Bi,j(Xi→),convex model,γln⁡(1+Bi,j(Xi→)),concave model,where α,β,γ are scaling factors, r controls the growth rate in the convex model, and Bi,j(Xi→) denotes the bandwidth provided for content j at edge cache i under the caching strategy Xi→. The concave utility model uses ln⁡(1+Bi,j(Xi→)) to ensure the function is well-defined for Bi,j(Xi→)=0. This formulation provides a consistent and flexible representation of utility, aligning with the diverse requirements of caching systems while avoiding potential confusion with previously defined terms such as bi,j.

In this section, we describe the process of generating a synthetic dataset for evaluating VoD caching systems. The dataset models two key aspects: the total number of requests across time intervals and the dynamic popularity of videos and their versions.

The normalized number of requests Nkreq in time interval k varies over time following a diurnal access pattern, reflecting realistic user activity [26]. This pattern is modeled using a normal distribution 𝒩(μkreq,σkreq), where μkreq and σkreq represent the mean and standard deviation of the normalized number of requests in time interval k, respectively. This approach effectively captures realistic temporal variations in user demand throughout the day [26].

Using these models, a synthetic dataset is generated with Nsample samples per version. The dataset, 𝒟, is represented as an Nsample×Nvideo×Nver matrix, where each entry captures the version-level popularity. For each video i, the average number of concurrent requests, denoted as Niavg, is represented as a set of Ni,javg, where Ni,javg is defined as the product of the duration of segment Vi,j and its average access probability over all samples in the dataset 𝒟.

These modeling components are grounded in prior literature on caching workload simulation [26,27], enhancing the realism and reproducibility of the generated dataset. Furthermore, by varying model parameters such as request skewness and version-level diversity, researchers can customize the dataset to evaluate a broad range of caching algorithms under diverse operational scenarios.

It is worth noting that the dataset is constructed solely from content-level popularity metrics and request distributions, without involving any user identifiers, behavioral traces, or session-level logs. As a result, the training and evaluation processes remain fully compliant with privacy-preserving principles in edge computing.

Let us first define the normalized concurrent request count Ni,j,kcon, which represents the expected number of concurrent requests for version Vi,j during interval k. Using Little’s Law [28], Ni,j,kcon is given as: (3)Ni,j,kcon=pi,j(k)⋅Nkreq⋅Li,where Li represents the average duration (in appropriate time units) for which video i is streamed.

Next, the weighted utility Ui,j,kweight(Xi→) for version Vi,j during interval k is defined by multiplying its utility Ui,j(Xi→) by the normalized concurrent request count Ni,j,kcon, as: (4)Ui,j,kweight(Xi→)=Ui,j(Xi→)⋅Ni,j,kcon.

For each video i, the request-driven utility is calculated as the sum of the weighted utilities across all intervals k and versions j: (5)Uirequest(Xi→)=∑k=1Ninterval∑j=1NverUi,j,kweight(Xi→).

The optimization problem can be formulated to maximize the expected popularity-weighted utility, considering the stochastic nature of video popularity and the number of requests per interval. The storage size of version Vi,j is denoted by Si,j, and the total storage capacity is Slimit. The objective is to maximize the expected popularity-weighted utility under the storage capacity constraint: (6)MaximizeE[∑i=1NvideoUirequest(Xi→)],subject to∑i=1Nvideo∑j=1NverXi,jSi,j≤Slimit, 1.

Objective Function: The objective function computes the expected total request-driven utility over all intervals k.

This formulation optimizes caching strategies to maximize the expected popularity-weighted utility while adhering to storage capacity constraints. DRL is the most suitable approach for addressing this problem, as it can learn optimal policies that adapt in real-time to highly dynamic environments, effectively handle stochastic elements, and balance both short-term performance and long-term efficiency.

We propose a DRL-based cache determination algorithm (CDA) that uses proximal policy optimization (PPO) due to its ease of implementation, sample efficiency, and reliable convergence [29,30]. PPO stabilizes training through clipped probability ratios, making it particularly suitable for dynamic environments with fluctuating content popularity [29,30].

The algorithm operates in two phases: training and decision. In the training phase, the agent learns its caching policy over Nepisode episodes by interacting with a simulated environment. In the decision phase, the agent applies the trained policy to determine caching actions without further learning.

We model the caching process as a sequential decision-making problem, where the agent selects which bitrate versions of a video to cache, processing one video at each decision step. Therefore, the time step in the DRL formulation corresponds directly to the video index, denoted as i. That is, the agent processes videos in order, and each decision step is associated with selecting bitrate versions for video i. This sequential structure naturally follows the framework of a markov decision process (MDP), where each decision influences the subsequent state transitions.

At each step i∈{1,…,Nvideo}, the following operations occur: 1.

The agent observes the current state of the environment, which includes features such as storage usage and popularity estimates for video i.

The DRL algorithm is built upon three main components: 1.

Action space: The action space A is defined as the set of all possible versions an agent can select: A={1,2,…,Nver}, where Nver is the total number of versions.

The valid action space Aivalid is dynamically updated based on the remaining storage capacity. Versions are evaluated sequentially, and those that would exceed the capacity are excluded.

The observation state at each video index i is: Siobj={i+1,Uistr,Niavg,Ni+1avg},where Uistr denotes cumulative storage used until step i, and Niavg is the average concurrent requests for each version of video i.

The reward function is designed to promote caching decisions that maximize utility while ensuring that at least one version is cached for every video. At each decision step i, the agent receives an immediate reward ri, computed as the ratio of the achieved request-weighted utility Uirequest(Xi→) to a reference utility Ugreedy, obtained from a baseline greedy algorithm. This ratio is scaled by a factor λ to normalize the reward magnitude: (7)ri=λ⋅Uirequest(Xi→)Ugreedy.

To penalize scenarios in which no version of a video is cached—thus resulting in zero utility—a penalty term is applied. For each video i, if the selected action ai results in no cached versions (i.e., an empty set), an indicator function adds a penalty of 1. The total penalty across all videos is expressed as: (8)rpenalty=∑i=1NvideoI({j∣j∈ai∩{1,…,Nver}}=∅).

The total reward for an episode is then computed by summing the individual rewards ri across all videos and subtracting the cumulative penalty. This final reward is used as the optimization objective for PPO training: (9)repisode=∑iri−rpenalty.

This reward structure encourages the agent to not only prioritize caching versions that yield higher utility but also to ensure that at least one version is cached per video. As a result, the agent learns a balanced policy that improves overall utility.

To visually clarify the agent’s behavior during training, Fig. 2 illustrates the decision-making process at each step i. At this step, the agent observes the system state—including video popularity, cache utilization, and video index—then selects a multi-version caching action based on the current policy. The environment returns a reward that guides the learning process.

Algorithm 1 describes the training phase of the proposed CDA algorithm. In this phase, the agent learns caching strategies by interacting with a simulated environment over multiple episodes. At the beginning of each episode, a sample scenario is selected from the dataset 𝒟, which defines the access probabilities and request patterns for the video versions. Based on this sample, the concurrent request counts Ni,j,kcon are computed using Little’s Law.

The agent processes the videos sequentially. For each video i, it first identifies the valid action set Aivalid, considering the remaining cache capacity. The agent then selects an action ai, indicating which bitrate versions of the video to store. The caching decision Xi→ is updated accordingly, and the reward ri is computed using Eq. (7). This process continues until either all videos are processed or the storage constraint is violated, at which point the cumulative reward for the episode repisode is computed using Eq. (9). The PPO policy model 𝒫 is then updated based on the collected experience to improve future decisions.

Algorithm 2 presents the decision phase of the CDA algorithm, during which the trained PPO model 𝒫 is used to determine the actual caching decisions without further learning. This phase is applied in a testing environment, where the agent makes one-shot decisions based on the learned policy.

Given the trained model and a test dataset, the concurrent request values Ni,j,kcon are first computed to reflect the workload scenario. For each video i, the agent determines the valid action set Aivalid based on current storage availability and uses the trained policy to select an action ai. The caching decision vector Xi→ is updated accordingly. This process continues for all videos, or until no further actions are valid due to storage constraints. The resulting caching configuration {X1→,…,XNvideo→} defines the final output of the CDA algorithm.

To optimize the performance of the PPO-based caching agent, we employed Optuna [31], a state-of-the-art hyperparameter optimization framework that uses a Bayesian optimization strategy. The objective was to maximize the average episode reward during training by identifying the most effective hyperparameter configurations for the PPO algorithm. The search space was defined over three key hyperparameters: the learning rate, the clipping range, and the value function coefficient.

Specifically, as shown in Table 2, the final PPO hyperparameter settings were as follows. The learning rate was set to 0.0003 to balance convergence speed and policy stability. Each policy update used 2048 steps to collect sufficient experience, and the batch size was set to 64. We trained the model for 10 epochs per update. The discount factor was set to 0.99 to account for long-term rewards, while the clip range was fixed at 0.2 to stabilize the policy updates. Lastly, the value function coefficient was set to 0.5, balancing the optimization of the policy and the accuracy of the value function estimation.

Optuna conducted a total of 100 trials using the Tree-structured Parzen Estimator (TPE) as the sampling algorithm. Each trial trained the PPO agent for a fixed number of episodes and reported the mean episode reward as the objective metric. The best-performing configuration was selected based on its ability to maximize long-term utility across diverse caching scenarios.

We conducted simulations to evaluate the proposed scheme, with the DRL algorithm implemented using the PyTorch framework. The experimental parameters were configured with Nvideo=500 and Nver=7, corresponding to seven recommended bitrates (30, 24, 15, 12, 10, 6, 4 Mbps) as suggested by YouTube [26]. The video lengths were randomly assigned within a range of 30 to 90 min, and Ninterval was set to 24. The value of μkreq was set between 0.3 and 0.8, while σkreq was fixed at 0.05.

The utility model parameters were set as follows: α=0.33, β=0.19, γ=3.28, and r=0.1. The experiments varied five parameters: the cache-to-total video size ratio, version popularity, Zipf skewness, rank-change groups, and the utility model. The default values were set as follows: cache ratio 5%, μver=4, σver=1.5, Zipf skewness 0.3–0.7, Ngroup=5, and a linear utility model.

To evaluate the performance of CDA, we adopted the total utility—normalized to the utility achieved by CDA—as the primary performance metric. We compared CDA against six baseline caching strategies, including four greedy heuristics and two predictive models based on long short-term memory (LSTM) networks. In all cases, video popularity is quantified by the number of concurrent requests: 1.

Initial Popularity-Capacity Ratio (IPCR): Videos are cached based on their initial popularity (i.e., number of concurrent requests) divided by their size, prioritizing those with the highest ratio.

Fig. 3 illustrates the impact of the cache-to-total video size ratio on overall utility. CDA consistently outperforms the benchmark algorithms, achieving utility improvements ranging from 3.32% to 26.73% (with an average improvement of 11.73%). These results highlight CDA’s ability to effectively learn from training datasets and optimize utility across varying cache capacity constraints. The performance gap between CDA and the benchmark algorithms is more pronounced under lower cache capacity conditions and gradually narrows as capacity increases. These findings validate CDA’s capability to optimize resource allocation and maximize total utility, even under stringent cache capacity limitations.

Fig. 4 illustrates the impact of varying μver, the mean value of the version popularity distribution, where higher values indicate increased popularity for high-bitrate versions. CDA consistently outperforms the benchmark algorithms, with utility improvements ranging from 3.32% to 135.87% (average 28.47%). The performance gap is more pronounced when low-bitrate versions are more popular (lower μver) or when μver is randomized, indicating unpredictable version popularity. These results demonstrate CDA’s ability to adapt effectively under conditions where version request patterns favor low-bitrate versions or exhibit high variability.

Fig. 5 illustrates the impact of popularity bias on performance, where the Zipf skewness parameter indicates the degree of bias in the popularity distribution. Higher values reflect stronger bias, while lower values represent a more uniform distribution. CDA consistently outperforms the benchmark algorithms, achieving utility improvements ranging from 3.06% to 50.71% (average 24.23%). The performance gap is most pronounced at moderate skewness levels (e.g., around 0.5) and diminishes when the bias becomes either excessively strong or weak.

Fig. 6 shows the impact of varying Ngroup. As Ngroup decreases, the number of videos per group increases, leading to more dynamic ranking changes. CDA consistently outperforms the benchmark algorithms, with utility improvements ranging from 3.32% to 69.49% (average 23.26%). The performance gap is most significant when Ngroup=4, demonstrating CDA’s efficiency in handling dynamic environments with abrupt ranking changes.

Fig. 7 illustrates the impact of utility types on performance, categorized as linear, convex, or concave. CDA consistently outperforms the benchmark algorithms, with utility improvements ranging from 3.32% to 47.34% (average 23.84%). The performance gap is most pronounced for concave utility types, where the utility growth rate slows down as resources increase. This pronounced gap in concave utility types may occur because greedy algorithms tend to allocate cache storage inefficiently in scenarios with diminishing returns. In contrast, CDA, leveraging its learning-based approach, adapts more effectively by optimizing caching decisions to maximize utility. Notably, even when compared with LSTM-based predictors (LSTM-P and LSTM-PCR), which aim to forecast content popularity, CDA achieves consistently superior performance across all utility types. This result demonstrates that CDA not only outperforms static heuristics but also surpasses predictive approaches by directly learning caching policies under varying utility dynamics.

Fig. 8 illustrates the impact of the total number of videos on caching performance. CDA consistently outperforms the benchmark algorithms across all tested video counts, achieving utility improvements ranging from 1.02% to 55.87% (average 23.81%). These results demonstrate that CDA effectively prioritizes high-utility versions even as the content scale increases, highlighting the strong adaptability of the DRL model.

To assess CDA under realistic conditions, we leveraged video popularity information from a VoD access trace [32], focusing on the top 500 most requested videos. Fig. 9 shows the normalized utility of each method relative to CDA. CDA consistently outperformed all baselines, achieving utility gains ranging from 0.05% to 41.9%, with an average improvement of 14.2%. These results demonstrate the practicality and robustness of our DRL-based caching strategy under realistic content dynamics.

CDA requires an average of 26 min to execute 1 million timesteps during the training phase. In the decision phase, the inference time depends on the total number of videos, as each video corresponds to a single decision step. As shown in Table 3, the overall decision time increases with the number of videos. These results were obtained in an environment with an 8-core CPU running at a clock speed of 4.05 GHz.

To further validate the learning stability of the proposed DRL model, we plotted the average episode reward across training. As shown in Fig. 10, the learning curve stabilizes after sufficient training episodes, confirming convergence of the PPO-based caching agent.

We highlight the following major findings:

The proposed CDA algorithm consistently outperforms both conventional greedy-based algorithms and LSTM-based prediction-driven strategies across various scenarios. It achieves significant utility improvements under limited cache capacity, high popularity of low-bitrate versions, skewed content popularity, and dynamic rank variations. In addition, CDA maintains robust performance across different utility models, demonstrating its adaptability to diverse system environments.

This paper presented a deep reinforcement learning-based caching algorithm designed to optimize content placement in edge caching systems under dynamic video popularity and resource constraints. Unlike traditional greedy or static strategies, our proposed cache decision algorithm (CDA) leverages the proximal policy optimization (PPO) framework to learn adaptive caching policies through experience-based exploration and feedback. We introduced a novel reward structure that integrates both cache utility and video quality degradation, enabling the agent to make balanced and informed decisions that align with provider-centric utility goals.

To rigorously evaluate the algorithm, we developed a comprehensive synthetic dataset generator that models real-world access behaviors including diurnal variations, Zipf-based popularity dynamics, and probabilistic bitrate version preferences. In addition, we conducted an experiment using the top 500 most popular videos from a real-world VoD trace to validate the algorithm under realistic access patterns.The experimental results confirm that CDA significantly outperforms both greedy baselines and LSTM-based prediction-driven caching strategies across a wide range of scenarios. Notably, CDA exhibits strong performance under severe cache constraints, unpredictable version preferences, and when the utility model reflects diminishing returns, such as in concave cases. The utility gains are most pronounced in highly dynamic environments where traditional and prediction-based methods typically struggle.

In addition to utility performance, we validated the computational efficiency of CDA, showing that the model can be trained within practical timeframes and deployed with minimal inference overhead. This makes our approach suitable for real-time edge deployment on low-power platforms. Future work will extend this model to multi-agent cooperative caching scenarios across federated edge nodes and explore transfer learning techniques to reduce training overhead under non-stationary popularity shifts. In addition, collaborative filtering techniques may be incorporated in environments where user-level data is available to enhance personalized caching.