In MTD temporal decision methods, dynamically adjusting the triggering timing of defense strategies is a core challenge. Existing MTD approaches often assume symmetric knowledge of network states between attackers and defenders, neglecting the process by which attackers gradually learn defense thresholds through trial and error. This assumption makes it difficult to precisely control the timing of defense strategies, thereby compromising defense effectiveness and system performance. To address this issue, we introduce a belief factor B, capturing the estimation differences between attackers and defenders regarding defense thresholds. The design is based on:

Asymmetric Cognitive Modeling. Defenders know the actual thresholds but estimate attackers’ perceptions, while attackers iteratively approximate thresholds through trial and error. The belief factor dynamically reflects these differences, enabling precise defense timing.

By introducing the belief factor, the proposed MTD temporal decision method effectively addresses the following issues:

Precise Control of Defense Timing. It quantifies cognitive differences, enabling dynamic adjustments to avoid premature or delayed triggers.

We propose a Stackelberg-FlipIt game under the leader-follower paradigm, modeled as a 9-tuple <P,AS,S,T,A,B,R,C,π> to comprehensively account for real-world network attack-defense parameters and variables.

P={PD,PA} denote the participants in the attack-defense game, where PD represents the defender and PA represents the attacker.

The conditions for classifying the four states are as follows: (1){Snorm={S∣s1≤θ1,s2≤θ2,s3≤θ3}Sfra={S∣s1>θ1,s2≤θ2,s3≤θ3}Sdam={S∣s1>θ1,s2>θ2,s3>θ3}Srec={S∣s1≤θ1,s2>θ2,s3≤θ3}where Θ=(θ1,θ2,θ3) is the vector of thresholds for S=(s1,s2,s3). These three thresholds are only transparent to the defender, while the attacker can only gradually approximate them through continuous exploration. This setup aligns more closely with real-world network attack-defense dynamics. Whether these thresholds are exceeded depends on the actions of both the attacker and the defender. For example, a state transition St−1fra→Stdam hints that at time t AS is FLIPPED and controlled by the attacker. This aspect will be further elaborated in the subsequent description of actions and the algorithm.

T=TD+TA, which denotes the total time required for the attack-defense game, which is the sum of the total time the attacker controls the AS (TA=Tdam) and the total time the defender controls the AS (TD=Tnorm+Tfra+Trec). To simplify the analysis, we assume that the attack-defense game unfolds within a finite and discrete time frame, i.e., T={t1,t2,…,ti},i∈N+.

Belief factor BP=(b1P,b2P,b3P) represents the participants’ estimated values of the thresholds Θ perceived by the attacker, as shown in Table 3. It is a crucial factor influencing the temporal decisions of participants in the Stackelberg-FlipIt game. As the game progresses, BA gradually approaches the actual values. Therefore, for the attacker, the primary goal is to quickly converge BA to the actual values, as mastering the true thresholds enables prolonged control over the AS. For the defender, although the thresholds are transparent, waiting until the attacker learns the actual values before taking action would prolong the loss of control over AS, potentially leading to defense failure. Thus, the defender also needs to estimate BA, denoted as BD. The defender determines the timing of strategy triggers based on ΔBD=|Θ−BD|, which represents the difference between the defender’s estimated network state values and the attacker’s estimated thresholds. This ensures targeted and timely implementation of defense measures.

R={RD,RA} denote the reward functions, satisfying: (4)RP=w1⋅RSEC+w2⋅RPERF+w3⋅RAFF−BEwhere w1, w2, and w3 are the weights that satisfy w1+w2+w3=1. BE is the belief error, satisfying (5){BEA=∑ibiA−θiBED=∑ibiD−biA

R mainly rely on three categories of reward evaluation RSEC, RPERF, and RAFF, which evaluate the security, performance, and affordability respectively. The design of BE is primarily aimed at guiding strategy updates. In our presumption, the weights are configurable, so that our method can be applicable to various MTD systems, e.g., a lightweight system that aims to offer modest security and system performance with the least cost, or, a large defense system that pursues as much as possible a highly secure defense without focusing much on cost. This design is motivated by the following two reasons:

A multi-dimensional evaluation framework provides a more holistic assessment of strategy effectiveness, capturing various aspects of performance, security, and cost.

Clarification of reward design in our model addresses it by designing a robust and transparent reward function considering former evaluation metrics of MTD technologies over the past decade. These metrics ensure that the reward function is both comprehensive and grounded in established research, enhancing the practicality and effectiveness of the proposed method.

Assumption 1: The values of each component in the reward function are normalized to the range [0,1]. This design choice is justified because the evaluations underlying the reward function are all derived from probabilities or ratios, which naturally fall within this range. ▪

RSEC ♦

For defender: Availability Probability

To quantify the effectiveness of the defense system in terms of security, we use AP (Availability Probability) [20,21] to measure the probability that the defender controls the AS over a period T. The formula is defined as: (6)AP=TDT ♦

For attacker: Mean Time to Compromise Ratio

MTTCR (Mean Time to Compromise Ratio) [8,22] represents the ratio of the time the attacker controls the AS to the total time, which is essentially the probability of the attacker controlling AS. This metric indirectly reflects the attacker’s ability to fully access and exploit AS. (7)MTTCR=TAT ▪

RPERF ♦

For defender: Escape Probability

EP (Escape Probability) [10,23] represents the probability that the defender transfers the AS after detecting that the attacker has gained control of it. This metric emphasizes the system’s ability to promptly defend itself upon detecting an attack. The formula is expressed as: (8)EP=p(Snorm|Sfra)+p(Srec|Sdam)where p(St′|St,atD,atA) represents the probability of reaching state St′ from state St in which the defender takes action aD and the attacker takes action aA. ♦

For attacker: Successful Attack Probability

SAP (Successful Attack Probability) [8,24] represents the probability that the attacker, after gaining control of the AS, proceeds with the next attack while avoiding detection and triggering an AS transfer by the defender. This metric primarily reflects the attacker’s ability to conduct stealthy attacks. The formula is expressed as: (9)SAP=p(Sdam|Sfra)+p(Sdam|Sdam) ▪

RAFF, the same as C={AC,DC} ♦

For defender: Defense Cost

DC (Defense Cost) represents the cost incurred by the defender when executing defense strategies to counter attacks. This metric measures the expenses associated with defensive actions. Due to the multi-stage nature of the attack-defense design, this indicator indirectly incorporates the time costs of both parties into consideration. Since the costs of strategies vary, no specific formula is provided; instead, the costs are comprehensively calculated after each action. ♦

For attacker: Attacker Cost

Similar to DC, AC (Attack Cost) measures the overhead incurred by the attacker when executing attack strategies. Likewise, it is only necessary to comprehensively calculate the costs after each action.

πP(St,atP) denotes the policy function, representing the probability that participant P takes action atP at a given time t and state St. Its update formula is:

Given the setting of the game model, we model the MTD attack-defense process as a multi-stage Stackelberg-FlipIt game, incorporating the MTD scenario into the attack-defense game evolution through the asymmetric projections of both parties. Finally, the proposed multi-agent WoLF-PHC algorithm is applied to solve the MTD temporal decision-making method.

The IP address is a critical target for attackers launching scanning attacks, DDoS attacks, and other types of subsequent attacks. Therefore, IP hopping defense [28] is one of the key technologies in MTD. Defenders use MTD techniques to randomly change host IP addresses, achieving attack evasion. Existing research has shown that IP hopping is an effective method against scanning attacks. However, blindly triggering IP hopping without perceiving or accurately perceiving the attacker’s behavior can lead to two issues:

It may fail to achieve the desired defensive effect and could provide attackers with more information to counteract.

The experimental algorithm is run on OpenAI Gym, which provides a multi-agent game simulator for simulating complex network attack-defense scenarios. The experimental environment is deployed in an SDN network controlled by an OpenFlow controller [29]. We use Mininet [30] to create an OpenFlow switch network as the data plane of the SDN, while the control plane is constructed and deployed using the Ryu platform to implement the IP address dynamic defense system, as shown in Fig. 2. In this network, 400 hosts are randomly distributed across 20 subnets, with the subnets connected through OpenFlow switches.

The experiment simulates real-world scanning attacks and IP hopping defense techniques. The network system state S=(s1,s2,s3) involved information in the attack-defense process that includes attack frequency, system load, and exposure time of real IP addresses. The system load is influenced by both the attacker and the defender, as the defense system may stop services after receiving a large number of data packets, thereby reducing QoS. The individual threshold parameters of the states are normalized as a middle vector with Θ=(0.5,0.5,0.5) to simplify the multi-agent reinforcement learning process.

The attacker employs a coordinated scanning approach, where a certain number of hosts simultaneously initiate scanning attacks by sending probe packets to the IP addresses of the defense network. In the experiment, we assume 50 to 200 scanning hosts targeting 400 defense network hosts. The attacker is assumed to know the IP address resource pool of the defense network and can divide the entire IP address space of the defense network, evenly distributing it among the scanning hosts. These hosts perform random scans on their assigned IP address segments with a uniform distribution. The attacker’s goal is to detect the IP addresses currently in use by online hosts to build an attack list, providing a foundation for subsequent attacks. Cautious attackers typically avoid rescanning already scanned IP addresses to minimize the likelihood of detection failure [31]. In the experiment, the scanning attack is designed to be propagative, meaning that detected online hosts are compromised and become new scanning hosts. The unscanned IP address ranges are redistributed among the new set of scanning hosts, with an infection time step of 1.

Each scanning host has two attribute characteristics:

Frequency of Scanning Packet (FSP): The average number of probe packets sent per second.

Based on these two characteristics, the attacker’s scanning behavior can be classified into three levels, corresponding to the action classification in the Stackelberg-FlipIt game model, as shown in Table 4.

The defense mechanism assigns each host within the system both a fixed real IP address and a time-varying virtual IP address. IP address alteration operates via two distinct modes: periodic time-triggered IP hopping (low-level defense), and event-driven IP hopping (mid-level and high-level defense). The periodic mode introduces fault tolerance for defensive event adjudication, providing a buffer mechanism to mitigate security risks when event determination errors occur. IP hopping behaviors are hierarchically classified into three tiers based on two metrics as systematically detailed in Table 5:

Hopping Frequency (HF): The frequency of IP address hops.

The experiment employs 1000 episodes with T = 100 steps per episode and 100 independent trials. Hyperparameter configurations are summarized in Table 6, with the following design rationales:

Learning rate α is parameterized with two tiers: ♦

αwin must be sufficiently large to ensure rapid convergence when policy performance is favorable but must avoid excessive magnitudes that induce destabilizing oscillations. Usually, αwin∈[0.1,0.5].

Through a series of preliminary experiments, we rigorously validated the guaranteed convergence of the algorithm within 1000 episodes. These experiments further enabled the determination of optimal hyperparameter configurations to maximize algorithmic performance, as shown in Fig. 3.

As demonstrated in Fig. 3 and Table 7, Config. 3 (0.2, 0.1, 0.1, 0.1, 0.6, 0.2, 0.2) achieves the optimal performance, exhibiting both the fastest convergence and the highest final policy reward.

Configs. 1 and 3 exhibit accelerated convergence due to their larger αwin and αlearn, enabling rapid stabilization of near-optimal policies. Conversely, Configs. 2 and 4 converge more slowly and prolong policy refinement. Despite its rapid convergence, Config. 1 yields the lowest reward because its excessive allocation to w1, which leads to the time-occupancy weight prioritizing adversarial engagement duration at the expense of strategy cost optimization, of which the imbalance diminishes the overall reward efficacy. Hence, Config. 3 achieves superior rewards by balancing weights to harmonize security, performance, and affordability objectives. In actual application, the configuration of reward weight coefficients can be manually changed regarding what the goal of the task is, e.g., a lightweight defense system aiming at modest security and performance with the least cost. Here, we mainly focus on the balance among security, performance, and affordability with the weight coefficient being (0.6, 0.2, 0.2).

In the following experiments, by comparing three IP hopping methods—OF-RHM [32], SEHT [33], and DDS [34]—we comprehensively analyze the results from three perspectives: security, performance, and affordability. As for security analysis, Host Survival Ratio (HSR) and Host Survival Average Time (HSAT) are utilized to evaluate the network security under different algorithms. As for performance and affordability analysis, Actual IP Hopping Frequency (AHF) and Network Channel Resource Occupancy (NCRO) are utilized.