Before applying the MADRL algorithm, the problem is formulated as a distributed partially observable Markov decision process (Dec-POMDP). In this framework, the static security preventive control of the power system is represented as a six-tuple: (S,Oi,Ai,Ri,P,γ), where S is the system state space; Oi is the local observation set of agent i, containing only partial information from the global state; Ai is the action set of agent i; Ri is the reward function of agent i, which satisfies S×A→R; P is the state transfer function; and γ is the discount factor. At each time step t, agent i observes the operational state oit of its region and selects an action according to its policy πi(ai|oi). The system then transitions to the next state st+1∼P(st+1|st,at) based on the current state st and the joint actions at of all agents, providing immediate rewards rit to each agent. Subsequently, each agent receives a new local observation and stores the current state, action, reward, and next state in the replay buffer for network updates. This process repeats until a termination condition is met and a training episode is completed. (1)

The observable state of the agent

Due to limitations such as communication constraints and privacy protection, each agent’s observation is restricted to its local measurement data. Each agent makes decisions solely based on its local state, without requiring complex communication devices to obtain information from other agents. The observable state is defined in Eq. (9). (9)oi={ηl,PG,Vn,Pload,Qload} (2)

The action of the agent

The range of real-time control decision variables is mapped to the interval [−1, 1] according to the generator ramping constraint in Eq. (3), and the mapped variables constitute the agent’s action space. (10)ai={ΔPG,ΔUG}where ΔPG And ΔUG denote the generator active power and the change in terminal voltage, respectively. (3)

Rewards for agents

The control objective of each agent is to minimize violations of static security constraints. The reward generation process is illustrated in Fig. 1. After the joint actions of all agents are executed in the environment, system security constraints under normal conditions are first evaluated, followed by N-1 contingency verification. The resulting immediate reward is then fed back to each agent. When the trend does not converge, a greater negative reward K₁ is given.

The training goal of MADRL is to find the optimal strategy to maximize the cumulative return. To this end, the static security overstepping problem is transformed into a penalty item and a reward function is introduced. The reward function design is shown in Eq. (11): (11)ri={−K1Post-decision power flow non-convergence−K2Power flow fails to converge for an N-1 contingencyFadd−FI−FUk−K3−K4Limit violations occur during the N-1 contingency analysis1.5The system passes all N-1contingency checks without any violations where K1 is the penalty for current non-convergence after agent i’s action; K2 is the penalty for non-convergence during the N-1 process; K3 is the penalty for violations of balancing machines; K4 is the penalty for violations of regulating units; Fadd is the positive reward assigned when the total number of voltage over-limit Fv(k) and current over-limit Fi(k) in the system is small after the static security verification, as shown in Eq. (12); FI and FUk are the penalties for the agent’s violation of static security constraints. (12)Fadd=K5Fv(k)+Fi(k)where K5 denotes the reward coefficient.

The reward function is designed to address the requirements of static security preventive control in power systems, combining a severity measurement method based on utility theory with a discrete reward mechanism to guide agents in accurately perceiving and responding to static security risks. The penalty items FIUk and FUk for each agent’s violation of static security constraints are defined as follows: (13)FI=∑k=1ΩL⁡{min[a(efIk−1),K6],K6]} (14)FUk=∑k=1ΩL⁡{min[a(efUi−1),K6],K6]}

Among them: (15)fUi={Umin−Ui,Ui<UminUi−Umax,Ui>Umax0, other (16)fIk={Ik−Ik,max,Ik>Ik,max0, other

In summary, the reward function consists of three distinct components that guide the learning process. The penalty terms, governed by coefficients K1−K4, serve to penalize power flow non-convergence and operational constraint violations, ensuring all actions yield feasible system states. The guidance reward Fadd provides dense, positive feedback for control actions that reduce the number of security violations. Furthermore, a significant sparse reward is granted only when a fully secure state (zero violations) is achieved, collectively steering the agent toward robust N-1 static security compliance.

In order to solve the problem of instability in multi-agent environments, Multi-Agent Deep Deterministic Policy Gradient(MADDPG) adopts a centralized training-decentralized execution framework; in the centralized training phase, each regional agent first observes the regional state information, feeds it into its policy network to generate actions, and then all actions are combined to act on the environment. The environment feeds back the rewards of each agent and transfers them to the next operating state, and at the same time stores the experience in the experience replay buffer. In the update phase, small batches of data are randomly sampled from the experience replay buffer and aggregated into joint experiences (s,a,r,s′). When the Critic network is updated, it will consider the other agents’ policies and actions, and use the joint experience to update the network parameters, so as to achieve coordinated decision-making between agents. The Actor network updates its policy based on the Q-value given by the Critic network. After the network update is completed, each agent in the online execution phase makes decisions solely based on local observations, and there is no need to communicate with other agents. This method takes into account the communication restrictions in the actual scenario, making it suitable for the coordinated prevention and control of complex power systems in the sub-regions.

If the environment contains N agents, the system maintains 2N Actor–Critic networks. Among them, the policy of the i-th agent is denoted as πi, with parameters θi. The cumulative expected return of the i-th agent is given by: (17)J(θ)=E[Ri]=Es−pπ,ai−πθi[∑i=0∞γtri,t]

Then the deterministic strategy gradient of the i-th agent is expressed as: (18)∇θiJ(μi)=Eo,a∼D[∇θiμi(ai|oi)∇aiQiμ(s,a1,…,aN)|ai=μθi(oi)]where oi denotes the local observation of the i-th agent; D is the experience replay buffer, which stores the experience trajectories of all agents, and each sample is a quadruple, where: s=[o1,o2,…,on], a=[a1,…,aN], r=[r1,r2,…,rn]; Qiμ is the joint action value function, that is, a centralized state–action value function, each sample not only contains the local observation state and the execution action, but also incorporates the state-action information of other agents. This data structure effectively addresses the technical bottlenecks of traditional reinforcement learning in the field of multi-agents, so that agents can not only perceive changes in their own state, but also obtain global action strategies. Through the state-action joint coding in the process of continuous strategy update, the Markov nature of environmental interaction is preserved, and the problem of environmental instability caused by partial observability of traditional methods is effectively mitigated. Even if the strategy is continuously updated, the environment maintains stability. The update of Qiμ is updated according to Eq. (19): (19)L(θi)=Es,a,r,s′[Qiμ(s,a1,⋯,aN)−yi)2]where y_i is obtained by Eq. (20): (20)yi=ri+γ⋅Qiμ′(s′,a1′,…,aN′)|ai′=μi′(oi′)

In the MADDPG algorithm, the target actor network and the target critic network adopt a “soft update” strategy. This strategy can make the current network too similar to the target network, which hampers the effective separation of action selection from policy evaluation. It can also lead to an excessive pursuit of maximizing long-term discount returns, resulting in overestimation of the true Q-value. With multiple rounds of iterative updates, this overestimation tends to accumulate during policy exploration, increasing both deviation and variance. Consequently, the agent’s ability to explore globally optimal solutions and make optimal decisions in the current state may be impaired, which may eventually undermine the effectiveness of the learning strategy.

In order to solve overestimation in DRL, this paper incorporates the AC framework from the Twin Delayed Deep Deterministic Policy Gradient (TD3) algorithm into MADDPG. Two independent critic networks Q1μ and Q2μ are introduced, along with their corresponding target networks Q1μ′ and Q2μ′. The minimum of the two Q-value estimates is used as the target value. This approach effectively reduces overestimation deviation, thereby improving online learning stability. The Q-value calculation in Eq. (20) is therefore replaced by the formulation in Eq. (21). (21)yi=ri+γ⋅minm=1,2Qi,mμ′(s′,a1′,…,aN′)|ai′=μi′(oi′)

Prioritized Experience Replay (PER) speeds up convergence by prioritizing experiences with large temporal-difference (TD) errors, which are more informative for learning. However, such prioritization introduces a sampling bias, which is corrected by applying importance sampling weights Eqs. (25) and (26) to ensure unbiased gradient estimation.

In the MADDPG framework, the update of the action-value function Q(s,a) depends on the gradient transfer of the TD error. The TD error directly reflects the discrepancy between the agent’s estimated and actual experience values. Based on this characteristic, this study uses |δ| as a quantitative indicator of the importance of experience, as shown in Eq. (22): (22)δ=(y−Qμ′(s,a1,a2,⋯,aN))2

The larger the δ, the greater the discrepancy between the target network’s prediction and the actual experiential return. The sampling frequency of such experiences should be increased to accelerate convergence between the target network and the current network, thereby improving training efficiency. Therefore, the sampling probability of each experience is defined as Eq. (23), with its priority given by Eq. (24). (23)P(j)=pja∑kpka (24)pj=|δj|+ε

In Eq. (23): When α = 0, it corresponds to uniform sampling. In Eq. (24), ε∈(0,1) is a small constant that avoids probability collapse. Meanwhile, the TD error of newly added experiences is initialized as δmax.

Although PER improves learning efficiency by replaying experiences with high TD errors more frequently, it can also lead to data distribution deviations and training instability. Oversampling of high-priority samples changes the empirical data distribution, resulting in a biased gradient expectation during policy evaluation. To address this issue, importance sampling is introduced to compensate for the non-uniform sampling probabilities while maintaining the convergence properties of stochastic gradient descent. The corresponding importance sampling weight formula is shown in Eq. (25). (25)ωj=(1N⋅1P(j))βwhere N is the current capacity of the replay buffer, and P(j) is the sampling probability, and β is the weight coefficient, which controls the influence of the importance sampling weight in the learning process, and linearly anneals to 1 over the course of training. In order to improve the stability during convergence, the weights are normalized to stabilize the gradient update scale and suppress oscillations during training. Therefore, the Eq. (25) is normalized to obtain the formula: (26)ωj=(N⋅P(j))−βmaxi(ωi)

In the MADDPG algorithm, the parameters of the strategy network depend on the selection of the value network, and the parameters of the value network are updated by the loss function of the value network. Combined with the above-mentioned importance experience playback and improved Actor-Critic strategy, the loss function in Eq. (19) is modified to Eq. (27): (27)LPER(θi)=12∑m=12Es,a,r,x′[ωj⋅(Qi,mμ(s,a1,⋯,aN)−yi)2]

This section thus incorporates prioritized experience replay with importance sampling into the proposed CTDE-based multi-agent framework. The implementation process and network structure of the algorithm proposed by this research institute are shown in Fig. 2.

In the reinforcement learning framework, Curriculum Learning (CL) has emerged as an effective paradigm to improve the efficiency of policy learning. Its core principle lies in organizing training tasks in a “from simple to complex” manner, thereby guiding agents to progressively acquire stable and efficient decision-making capabilities. In each subsequent stage, load and generator disturbances are gradually increased, enabling the agents to learn from basic stability control to complex system coordination. By structuring the learning environment with progressive levels of difficulty, agents can rapidly acquire fundamental control strategies under low-complexity conditions and subsequently adapt to increasingly challenging scenarios. Through this gradual exposure, the learned policies not only enhance adaptability to multi-source load fluctuations while maintaining robustness and stability under severe disturbances.

In this work, CL is integrated into the preventive control training process of power systems. At the early stage, the focus is placed on ensuring basic stability control, allowing agents to understand and internalize the fundamental operational dynamics of the grid. During the intermediate stages, progressively broader load variations and disturbance conditions are introduced, thereby enhancing policy transferability and generalization. Ultimately, the agents are trained to respond rapidly and reliably to large-scale disturbances within highly complex operating environments, thereby achieving both robustness and practical applicability. This hierarchical design ensures stability and efficiency during training and establishes a solid foundation for subsequent optimization of cooperative control strategies.

To provide an intuitive overview of the proposed training framework, Algorithm 1 summarizes the pseudocode of the CL-PER-MADRL algorithm. This algorithm integrates CL with PER within an MADRL framework. The pseudocode illustrates the key training process, detailing the curriculum-based progression with increasing difficulty levels and the specific performance criteria required to advance to the next stage.

In this paper, the proposed method is simulated and verified by the IEEE 39-bus transmission system which is partitioned into three regions, as shown in Fig. 3. The observation space of each agent includes the active power of the generator, the line load rate, the bus voltage magnitude, and active and reactive power at load buses. The action space is defined as a continuous high-dimensional space for the adjustment of the generator’s output, and its adjustment range is constrained to 60%–100% of the rated output. During training, seven load levels (85%, 90%, 95%, 100%, 105%, 110%, and 115% of the reference load) are considered, with random source–load disturbances applied. Static N-1 verification requires traversing all transmission lines.

Based on the results of power grid partitioning, a distributed decision-making architecture is developed, where an independent multi-layer fully connected neural network is assigned to each region as the agent model, enabling autonomous regulation and decision-making. The number of neurons in the input layer of the agent actor network corresponding to regions 1, 2, and 3 corresponds to the dimensionality of the regional operating state. The output is the active power of the generator in each region and the adjustment of terminal voltages with the tanh function used as the activation function. Based on experimental evaluation, the number of hidden layers in both the actor and critic networks is set to three; the fully connected layers use the ReLU activation function, and the number of neurons in each layer is (512, 512, 256) for the actor and (1024, 512, 256) for the critic. Each neural network is updated once after every load adjustment. Both networks adopt the Adam optimizer for training, and Gaussian noise is employed for exploration. The specific parameter settings are summarized in Table 1.

Table 2 presents the components that exceed operational limits in the static N-1 verification conducted prior to adjustment. Static N-1 verification requires traversing all transmission lines individually; for each contingency, the system is checked for limit violations. The columns report the total number of violations accumulated over all N-1 contingencies: Voltage violations, Current violations, and Slack generator violations respectively denote the number of buses, transmission lines, and slack generator power outputs that exceed permissible limits in any contingency. Numerous voltage, current, and slack generator elements in the system exceed permissible thresholds, thereby violating static security requirements. Therefore, it is necessary to adjust the system operating point by applying preventive control and related measures to ensure static security compliance.

The power flow calculation program of the simulation example system, as illustrated in Fig. 3, is developed on the Pandapower platform, which provides the simulation environment for multi-agent training. The proposed algorithm is implemented using PaddlePaddle and is employed to train the multi-agent model. The hardware platform consists of an NVIDIA RTX A4000 GPU, an Intel(R) Core(TM) i7-13700 CPU, and 16 GB of RAM.

During the training process of the MADRL agent, each load is explored 12 times per round, and the power system is statically checked for safety after each current adjustment. The training process adopts intensive CL to reduce the difficulty of training, and the course is divided into four stages for learning. The training scenario settings for each stage are shown in Table 3.

The regional agents in this study are fully cooperative, and the total cumulative rewards obtained by the three agents in each round are shown in Fig. 4. Both the reward curve and the actor–critic loss converge smoothly, indicating that the policy has reached a stable equilibrium. The total training duration for the proposed MADRL model was approximately 1 h and 55 min. To ensure reproducibility, random seeds were fixed for all experiments, including neural network initialization, noise generation, and environment randomness.

In the first stage, the weight parameters of each neural network are randomly initialized, which fluctuate greatly during early training but gradually converge as training progresses. During the first 45 rounds, only the experience buffer was filled without updating the network, resulting in low reward values. From rounds 45 to 100, network updates commenced, causing the reward curve to rise rapidly as the agent gradually learned to satisfy the static security and stability requirements of preventive control measures. At round 140, the training entered the second stage. During the second stage, the load fluctuation range was increased, yet the control success rate remained high, indicating that the proposed MADRL method exhibits strong generalization ability. At round 190, the third stage commenced. The substantial increase in initial generator output disturbance caused the agent to partially exceed limits at the beginning of this stage; however, it stabilized after training. Upon entering the fourth stage, the agent continued to maintain high control performance even under greater disturbance intensities.

To further verify the robustness of the learning framework to hyperparameter variations, an additional experiment was performed in which the learning rate was increased by 20% compared to the value listed in Table 1. The resulting reward curve, as shown in Fig. 5, maintained a smooth and stable convergence pattern similar to the original setting, indicating that the proposed MADRL algorithm exhibits strong robustness and stability against moderate parameter fluctuations.

Before validating the proposed MADRL-based preventive control strategy, a conventional Optimal Power Flow (OPF) method was tested under the same system configuration to examine whether traditional optimization can ensure static security compliance. Although the OPF successfully converged to an economically optimal operating point, the resulting solution failed to satisfy N-1 static security criteria. Across multiple load levels, the OPF results exhibited dozens of cumulative voltage and current limit violations after N-1 contingency verification, indicating that OPF-based optimization, which does not explicitly consider contingency constraints, cannot guarantee static security in large-scale grid operations. This highlights the necessity of developing intelligent preventive control approaches that explicitly account for N-1 contingencies.

The effectiveness of the proposed algorithm in enhancing the static security of the power system was evaluated by deploying the trained agents in a simulation environment. During the testing phase, each agent performs real-time control actions based on the grid’s operating state under the specified load conditions.

As an illustrative case, at a load level of 85%—which represents the most severe limit violations—Figs. 6 and 7 illustrate the bus voltages and line loading rates before and after control through N-1 contingency verification. The results indicate that initially, multiple bus voltages and line currents exceeded their operational limits, met static security requirements. Specifically, the lower voltage limit was violated 94 times, and current overloads occurred 253 times. When line 14–15 were disconnected, the voltage at bus 14 dropped to 0.874 p.u., while the load rate of line 2–3 peaked at 308%. clearly indicates static insecurity.

Specifically, at this load level, the overlimit conditions summarized in Table 4 required five consecutive control adjustments to restore static security, with N-1 verification conducted after each adjustment of the system operating point. Through progressive tuning of operating points, the system gradually approached compliance with static security standards. Specifically, after the first adjustment, the removal of line 14–15 reduced the peak load rate of lines 2–3 from 308% to 258% (a 16.2% decrease); the minimum voltage at bus 14 increased from 0.874 p.u. to 0.916 p.u. (a 4.81% increase), with no overvoltage observed. After the second adjustment, the load rate of line 2–3 further decreased to 214%, and the voltage at bus 14 increased to 0.974 p.u. marking the first time all bus voltages satisfied safety limits. After the third adjustment, lines 3–4 were removed, shifting the overload-dominant line to lines 5–10 (load rate of 147%), while all bus voltages remained within safe limits. Following the fourth adjustment, lines 14–15 and 2–3 were identified as key overloaded lines (132%), with voltage levels remaining within normal limits. After the fifth adjustment, N-1 verification confirmed the elimination of equipment overloads and voltage violations, indicating that static security had been fully restored.

After completing training, the agent was deployed to evaluate each typical load condition. The results indicate that under various random operating scenarios, line outages—whether due to failures or maintenance—did not result in system voltages or currents exceeding operational limits.

To verify the effectiveness of the proposed MADRL-based static security preventive control method, MADDPG was employed for comparison, using the fourth stage as the target training benchmark. Fig. 8 presents the total reward curves of agents for each algorithm. As shown in Fig. 8, the MADDPG reward curve exhibits substantial fluctuations, indicating poor performance across varying load and source fluctuation scenarios.

The proposed method also began neural network updates at round 45 but incorporated a PER mechanism to prioritize under-learned experiences via importance sampling, thereby accelerating strategy convergence and enabling the agent to better extract local observation features and retain historical state information. Furthermore, the improved Actor-Critic network structure, combined with a well-designed curriculum, allows the agent to initially accumulate successful experiences in stable control under low disturbance conditions, facilitating a smooth transition as task difficulty gradually increases, and enabling the early-learned control strategies and features to transfer to more complex scenarios. This shallow-to-deep training approach effectively enhances strategy adaptability and robustness across varying load and source fluctuation conditions, maintaining high control performance under severe disturbances. Compared to direct training on challenging tasks, the proposed method substantially improves training stability and convergence speed, mitigates excessive reward fluctuations, and enhances strategy adaptability and robustness under varying disturbance intensities.

Fig. 9 depicts the cumulative number of N-1 limit violations for each algorithm following sequential load adjustments in each round. As illustrated, the proposed method significantly reduces limit violations after network updates, while maintaining a consistent downward trend as task complexity increases. These results demonstrate that the CL strategy effectively reduces static security constraint violations in complex disturbance environments and fosters a more robust multi-round control strategy.

To evaluate effectiveness, 1000 random scenarios from the fourth training stage were selected for testing and compared with the baseline MADDPG algorithm. Evaluation metrics comprised the success rate of static security preventive control, the average decision time per step, and the total number of agent decisions. Simulation results on the IEEE 39-bus system are summarized in Table 5. Results indicate that the proposed method achieves a preventive control success rate of 99.2%, significantly surpassing the 96.2% achieved by MADDPG. Moreover, the proposed reinforcement learning framework exhibits superior performance in both control efficiency and cross-scenario generalization.