As commonly understood, the cloud computing paradigm [16] often faces the challenge of geographical distance between a multitude of terminal devices and the centralized cloud center, and cloud computing also consumes substantial bandwidth from the backbone network, leading to network congestion and elevated bandwidth expenses. Mobile cloud computing, as an inherent progression and enhancement of cloud computing, addresses these challenges, but it still encounters challenges related to network congestion and significant computational expenses [17]. Subsequently, fog computing [18–20] was introduced. Compared with mobile cloud computing, fog computing is closer to the user than mobile cloud computing. It can not only provide the resources needed for computing but also provide some other services, such as image recognition. It also faces a lot of security and privacy issues [21,22]. Mobile edge computing (MEC) [23,24] bears a resemblance to fog computing, wherein a versatile computing server is deployed at the network edge, and tasks are likewise transmitted to nearby high-performance servers via wireless links. The difference is that MEC supports user location awareness, and MEC servers belong to mobile operators and are generally deployed on the base station side [25], while fog computing nodes are generally managed by individuals. From the perspective of privacy protection, MEC is relatively more reliable. From the design of MEC paradigm, if the terminal nodes and edge nodes in MEC are not under load for a long time, the computing and communication resources will be idle. Currently, resource allocation has grown into a crucial branch of edge computing.

Liu et al. [26] explored the application of non-orthogonal multiple access (NOMA) technologies to enhance connectivity in IoT networks, enabling energy-efficient mobile edge computing (MEC). Additionally, Yang et al. [27] presented a method of resource allocation in multi-access MEC systems that are designed to minimize mobile device energy consumption while meeting competing deadlines. The scenarios consider a scenario in which several mobile devices with different computing capabilities and energy restrictions offload their MEC tasks to a nearby server. Moreover, Chen et al. [28] analyzed the coordination optimization of computation allocation and resource allocation in edge computing, and proposed a new method: a time attention-deterministic strategy gradient (TADPG) for solving the problem. Zhu et al. [29] presented a synthesis of hybrid NOMA and MEC techniques in their work. They solved the optimization problem with respect to power and time allocation for each group and alleviated latency and energy consumption. The method utilizes a matching algorithm for optimizing power and time allocation to cut down the complexity of the method further. Otherwise, Ali et al. [30] posed the Pow Mig Expand algorithm, based on a comprehensive utility function, which allocates the requests received to the optimal server. In addition, they also introduced the Energy Efficient Intelligent Splitter (EESA) algorithm, which utilizes deep learning to allocate the required energy savings to the most suitable servers. Proposed methods are all using fixed infrastructure and lack scalability, making them unable to flexibly respond to sudden traffic surges or damage caused by disasters or other factors. In order to improve the flexibility of computing networks, recent attention has been focused on mobile device-assisted work. Sun et al. presented a novel system architecture in their work [31] that utilizes unmanned aerial vehicles (UAVs) and multiple Industrial Internet of Things (IIoT) nodes. The proposed architecture enables the direct transmission of resources collected by IoT nodes to the UAV for processing. The main goal of this system is to optimize response time for forest fire monitoring considering some constraints between nodes, i.e., minimize the maximum time. The system proposes a Markov random field-based decomposition (MRF) method and a learning-based collaborative particle swarm optimization (LCPSO) scheme to achieve the optimal allocation. These techniques work together to allocate the resources optimally in the UAV system, facilitating efficient forest fire monitoring. In their work, Chai et al. [32] considered users with task computing needs who may choose to perform edge computing offloading, local computing, or device-to-device offloading. The authors proposed a joint optimization method which wants to minimize costs by dividing the task into small data segments and enabling various computational modes to execute them simultaneously. On a similar note, Wang et al. [33] utilized MEC and D2D communication to facilitate offloading from the core network. They studied the optimal resource allocation strategy in the ICWN (Cognitive Wireless Network Internet) scenario. However, it is worth noting that the assistive technology may involve additional costs. In addition, the above research overlooked the potential untapped resources within the network and the achievement of utilizing neighboring computing nodes. If these factors are considered, it may enhance the efficiency and performance of the system.

Following the emergence of MEC, DCOMP was introduced as a complementary solution, aiming to address this gap and harness the untapped computing resources effectively. The objective is to establish a dispersed and distributed network ecosystem with heterogeneous resources, maximizing the utilization of all available resources within the network and delivering users with enhanced efficiency and reliability in computing services. Dispersed computing possesses distinct advantages compared to other paradigms. The objectives of conventional computing paradigms involve task offloading to either the cloud, which possesses greater computing power, or edge nodes with shorter communication distances. This is done to minimize transmission delays and energy consumption. However, central nodes like fog computing nodes, MEC nodes, or service gateways in cloud computing exhibit limitations [34,35].

With a massive increase in users and tasks, the central node fails, services will be interrupted and tasks will be lost, resulting in a poor user experience. Dispersed computing is an architecture for dispersed resources. In this architecture, nodes with computing power are abstracted as general computing points, and the DCOMP intermediate layer is responsible for security authentication, privacy protection, task decision-making and data routing transmission between nodes [36], and there is no centralized scheduling center. In addition, in a dispersed computing environment, the joining and leaving of nodes are very efficient and transparent. Simultaneously, it also addresses the drawbacks of the cloud computing paradigm, including high delay and high cost, while resolving the limitations of edge computing, such as the limited coverage area of edge nodes and the inability of terminal nodes to collaborate directly.

Indeed, in contrast to mobile edge computing, there has been growing attention to resource allocation in dispersed computing. Several studies [37–39] have emerged in this area, taking into account the presence of idle devices and limited resources within the network. A noteworthy example is the work of Rahimzadeh et al. [37], who introduced SPARCLE. SPARCLE is a scheduling scheme for dispersed computing systems and is designed especially for dispersed computing environments which are provided with a network-aware algorithm for allocating tasks and resources for polynomial time flow processes. The proposed schemes are very useful to optimize resources and improve overall system performance. The collaborative management model for the task resource in a dispersed computing system is proposed by Zhou et al. [38] and focuses on the quantification of dynamic dependencies between the resources occupied in a dispersed computing system and the consumed resources of task requests. This can better understand the resource utilization modes and achieve effective resource allocation. A kind of intelligent controlling mechanism to solve the problem of the overburdened load is brought into the collaborative management model of task resources. These intelligent control technologies aim to dynamically allocate and manage resources to avoid overload and ensure system stability. Through intelligent control, this model can optimize resource allocation and improve the whole performance and reliability of dispersed computing systems. The load-balancing scheduling algorithms and the prioritization-aware resource allocation strategies of the open-source dispersed computing system are studied by Paulos et al. [39]. These strategies are very important and have been integrated into a core feature in middleware. However, the research on resource allocation in dispersed environments is relatively limited to the field of dispersed computing. It is necessary to conduct further exploration and investigation to deepen our understanding of resource allocation in this situation. In addition, considering the limited energy resources of devices and the need to optimize energy consumption, energy cost has become a crucial indicator, especially for mobile devices. Therefore, when designing resource allocation strategies in a dispersed computing environment, special attention should be paid to addressing energy efficiency.

Considering a dispersed computing environment including one dispersed computing platform, N tasks published on the platform by L mobile users, and M networked computation points (NCPs) that can complete the tasks published on the platform using their computing abilities. The platform can publish and assign tasks for the mobile users, and can schedule computing resources of NCPs for various tasks. In the dispersed computing environment, the set of mobile users is denoted by U={u1,u2,…,uL}. Each mobile user publishes the tasks in the platform and requires the computing resources from the NCPs. Assuming the tasks published by user should be completed within a time duration T. In the researched dispersed computing environment, the position of mobile users is assumed to be unchanged with in the time duration for tasks to be completed.

The set of NCPs is given by P={p1,p2,…,pM}, which can provide computing abilities to the tasks in the platform. For each NCP, the available computing resource of NCP j for j=1,2,…,M that can be allocated for the tasks is denoted by cj. By using these computing resources, the NCPs can complete the tasks published on the platform by the mobile users.

The set of tasks published on the platform is denoted by V={v1,v2,…,vN}, which is running on the NCPs. For each computing task vi with i=1,2,…,N, the resources scheduled by the platform is denoted by ri, for i=1,2,…,N. Generally, the tasks wk published by user k will be uniformly treated as the related tasks in the set V={v1,v2,…,vN}. The maximum number of tasks that each mobile user can publish on the platform is given by δ, where 1≤δ≤N, and wk=∑i=1δvi.

Assuming ci,j to denote the computing resources of NCP pj allocated for the task vi. We design a binary variable bi,j to indicated whether the task vi is running on the NCP pj. If bi,j=1, the task vi is executed on NCP pj. If bi,j=0, the task vi is not executed on NCP pj. Then we have ci,j≥∑i=1Nbi,jci,j. In order to achieve load balancing among different NCPs, we assume that the maximum number of tasks that is computed on NCP pj is εj, where εj≥∑i=1Nbi,j. Meanwhile, the maximum number of NCPs working on the same task vi is assumed to be λi, and we have λi≥∑j=1Mbi,j.

In the researched dispersed computing environment, we assume all the tasks published by the mobile users should be completed only by the NCPs. The mobile users should publish the computation requirements on the dispersed computing platform. Once the requirements are answered, they should upload the related tasks to the NCPs and wait for the NCPs to complete the corresponding computing tasks. Therefore, for each mobile user, the whole-time duration can be divided into two parts using TDMA. The first part is to upload the computation tasks to the NCPs, where the second part is to complete the computation tasks by the NCPs. The diagram is given in the following figure.

Assuming the transmit power of mobile user is p, and the upload transmission is undertaken on the frequency band with channel bandwidth β. To avoid co-channel interference among different users, we assumed that the task could be uploaded onto the NCP using orthogonal frequency patterns by different users. Then the transmission time for the tasks upload can be written by [32] (1)τi=vi[βlog(1+pσ2)]

Once the tasks are upload to the NCPs, it will be computed by the NCPs. The NCPs should use the limited time to complete the tasks to ensure the real-time satisfaction. The limited time available for computation of task vi in the NCPs is T−τi. Assuming the workload of task vi is given by wi, and the related time for computing the tasks is wi/ci. In order to complete the tasks, the NCPs should contribute their computing abilities, which would bring energy consumption for task due to the control variable ci,j. Therefore, the total cost for computing the task vi in the NCPs can be given by (2)Ji=(T−τi)ciei=∑j=1M(T−τi)bi,jci,jei,where ei denotes the unit energy consumption of computation. The gross energy cost for the entire dispersed computing environment can be given: (3)J=∑i=1N∑j=1M(T−τi)bi,jci,jei.

It follows that we aim at minimizing the consumption of a distributed computing environment and that the minimization problem can be drawn out mathematically as follows: (4)minbi,j∑i=1N∑j=1M(T−τi)bi,jci,jei,s.t.C1:∑i=1Nbi,jci,j≤cj,C2:∑i=1Nbi,j≤εj,C3:∑j=1Mbi,j≤λi

To eliminate this problem (4) and solve it, the optimal solution to bi,j of the discrete offloading decision is found. In this way, the task offloading in inter-slot time is coordinated under a limited time constraint, and the total computational energy consumed is minimized for a network computing node. It is worth noting that bi,j is a binary variable which is the offloading decision variable and is why the network computing nodes can choose user tasks intelligently. With the increase of mobile users and computing tasks, traditional methods may not adapt to deal with complex scenarios with dynamic characteristics and make intelligent decisions. A method of solution for a certain problem based on reinforcement learning is explored.

State space: The state in the MDP model represents the space of the dispersed computing paradigm. For available resources, on time slot t, the state is decided by ci,j−∑i=1Nbi,jci,j and λi−∑j=1Mbi,j in our proposed system, where the former is the residual available computing resource of network computing nodes and the latter is the maximum number of offloaded nodes. These two parameters are observed to maintain the constraints of computing and resource capacity of network computing nodes. Additionally, if it reaches the suitable state, it is important to consider the energy consumption state Etotali at each time slot t to determine. Therefore, the state vector is defined as (Etotali,cj−∑i=1Nbi,jci,j,λi−∑j=1Mbi,j) at the time node T.

Action: The objective is to select the best action by mapping the state space to the action space. In our proposed dispersed computing environment, agents are responsible for determining the resource allocation of users’ computing tasks in each time period.

Reward: The actions of each agent are motivated by rewards. In the dispersed computing system, the reward function is given as follows to minimize energy consumption. We believe that the best action is produced with the lowest task reward value.

The MDP problem is effectively solved through dynamic programming or linear programming. However, when dealing with unknown and time-varying state transition probabilities and immediate rewards in discrete time steps, reinforcement learning methods become more suitable for tackling such problems. MDP is commonly used as the theoretical framework for reinforcement learning [40].

We approach the optimization problem by formulating it as an MDP. The agent interacts with the environment through iterative processes and adaptive learning to determine the optimal actions. The agent finds the current state at each time step t which is written as s(t) and chooses an appropriate action a(t) based on the policy π. The action space a is finite, it is the same for state space s. The policy π maps the current state to the corresponding action, providing guidance for decision-making under different states. Subsequently, the agent switches to its next condition s(t+1) based on a switch probability p which receives a reward R. Over the long term, the state value function V represents the cumulative discounted reward R of the agent in state s(t). It quantifies the long-term impact of the policy π in the current state and serves as a measure of the state-value pair.

Now, let us consider the state after executing action a(t) as s(t+1), where the transition probability from s(t) to s(t+1) is denoted as p. By employing the MDP model, the state value function is expressed using temporal differences through the Bellman equation [41]: (5)V(s(t))=Eπ[(∑t=0∞φRt)|s0=st]=Eπ[(R(st,at)+∑t=1∞φRt)|st=st+1]=R(st,at)+φ∑st+1p(st+1|st,at)Vπ(st+1)

Building upon the aforementioned process, the agent strives to develop an optimal control strategy π that maximizes the expected cumulative discount reward at the current state. This essentially transforms the optimization problem into an optimal state function problem V, as depicted below [41]: (6)V(s(t))=maxπ[R(st,at)+φ∑st+1p(st+1|st,at)Vπ(st+1)]s.tC1−C3.

Therefore, the optimal strategy is obtained by solving the above problems. The optimal action in state s(t) can be given as [41] (7)at∗=argmaxatVπ(st,at)

In practical scenarios, the curse of dimensionality often arises when dealing with high-dimensional states and action spaces. To mitigate this issue, this paper proposes a novel DRL algorithm that deviates from traditional Q-learning methods.

In the conventional implementation of a DQN, a single Q network is employed to evaluate actions and is similar to multiple action-value functions, which can be numerous. However, this approach may lead to significant errors and an overestimation of the action-value function, ultimately affecting the choice of the optimal strategy. To address this concern, we adopt the DDQN algorithm. The fundamental principle behind DDQN is to decouple action selection from evaluation by constructing distinct action-value functions.

By updating DNN’s parameters, the DDQN-based approach enhances the approximation of the optimal Q-values. In DDQN, the Q-value expression can be formulated by (8)Q(s,a)=Q(s,a,θ)

In the DDQN framework, the primary neural network’s weight parameters are denoted as θ. Additionally, it will be discussed for a target network later.

Unlike the general DQN algorithm, the DDQN algorithm incorporates an experience replay buffer. The agent converts relevant experiences into memory tuples (st,at,Rt,s(t+1)) and stores them in the replay buffer at each time step t. The agent utilizes the stored experiences from the buffer which is to train the neural network. It randomly selects a small batch of experiences to fine-tune the neural network and adjust its parameters. This process allows DDQN to leverage randomly chosen previous experiences during each update. Several studies, such as those cited in [41,42], have demonstrated that experience replay improves training efficiency, reduces correlation between training data, and expedites the convergence of DDQN.

In the DDQN algorithm, a target Q-network is introduced. The target Q-network and the Q-network employ different neural network parameters although they have the same neural network structure. The temporal difference signals from the Q-network are received by the target Q-network, while the latter serves as an evaluator for the Q-function.

However, the target Q-network updates its weights periodically instead of continuously. This approach enhances stability and accelerates the learning process.

The agent randomly selects a small batch of sample D with size R from the experience replay buffer to remove temporal correlations in the training data. The loss function L is continuously minimized to approach the target value and improve the accuracy of Q-function approximation during each iteration, as shown below [34]: (9)L(θ)=E[R(st,at)+φQ(st+1,a∗;θ)−Q(st,at;θ)]2

The structure process and training process of DDQN are illustrated in Fig. 1. The initialization phase involves setting the weights of the neural network and the experience replay buffer’s size. In DDQN, the Q-network and the target Q-network initially share the same weights, denoted as θ=θ′. Next, a neural network is employed to establish the relationship between its corresponding value function Q and each state-action pair. For further details, the decision maker needs to preprocess the allocation strategy of the distributed computing platform adequately.

During the training process of each episode, the agent first views the distributed computing platform’s state. Then, to choose the action a for execution, the ε-greedy algorithm is applied. With a probability of ε, the agent randomly selects an action. Otherwise, the state-action pair’s Q-value is estimated using the Q-network to select action a.

Next, the agent accepts the reward value R and the next status by executing action a. It constructs an experience tuple (s,a,R,s) and stores it. The Q-network randomly selects a small batch of samples from the replay buffer using an experience replay mechanism in the training process. Through minimizing the loss function θ, DDQN updates the network parameters. This involves updating the Q-network parameters using Stochastic Gradient Descent (SGD) and calculating the target Q-value. In Algorithm 1, the specific process is provided.

Then, the agent gets the reward value R and the next status by executing action a builds the experience tuple (s,a,R,s), and saves the experience tuple. The Q-network randomly selects small batch samples through an experience replay mechanism in view of the training process. Through minimizing the loss function θ, DDQN renews the network parameters, that is, the algorithm updates the Q-network parameters by taking the Stochastic Gradient Descent (SGD) method for the loss function θ after calculating the target Q-value. Finally, the specific implementation process is provided in Algorithm 1.

We consider the dispersed computing environment as a smaller area with multiple mobile users and three network computing nodes. We consider user devices that are not computationally capable or cannot meet the demand of computational tasks. Each user has a different size task. Here, we consider two cases: (i) each mobile user has only a single task. The size of tasks per user is randomly distributed in the range of 1–2.5 MB; (ii) each mobile user has multiple tasks. Each mobile user has multiple computing tasks of different types and sizes. The number of tasks per mobile user is randomly distributed in the span of 2–4, and the task’s size is distributed in 0.1–1 MB randomly. The user’s transfer rate is equal to 1 MB/s. In the approach of this paper, the experience replay buffer’s capacity is C = 200 and the selected small batch of samples’ capacity is U = 32. Set The learning rate parameter to lr = 0.01 and set the reward decay parameter to φ = 0.9. The energy consumption of users is decided by the task size of users, i.e., larger tasks consume more energy.

First, the convergence property of the resource allocation method based on DDQN is verified. Fig. 2 illustrates the convergence property of the DDQN-based method which is proposed with a total of 24 user tasks. For the DDQN-based resource allocation method the energy consumption of the system gradually drops while the number of training sets rises. In general, the DDQN-based method with multiple iterations of training is able to converge quickly to a relatively stable value.