Chaos is a highly irregular and highly sensitive dynamic phenomenon that occurs in deterministic nonlinear systems [11]. Chaotic systems typically exhibit the following basic characteristics, including sensitivity to initial values, non-periodicity, dense periodic orbits, topological mixing, attractors, orbital fractal structures, positive Lyapunov exponent, and so on.

Chaos mapping is an important support for chaotic systems, as its extreme sensitivity to initial values leads to highly complex and unpredictable behavior in deterministic systems. This makes it applied in spread spectrum communication, cryptography, image encryption, and other optimization algorithms [12–14]. Due to these characteristics, they can enhance the security of systems by generating complex, pseudo-random sequences. However, although low dimensional systems such as Logistic chaotic systems [15] and Tent chaotic systems [16] have simple structures and are easy to calculate, they also have problems such as limited pseudo randomness, limited complexity, limited key space, and uneven distribution, and weak resistance to attacks, which limit their performance and security in practical applications.

At the same time, high-dimensional chaotic sequences have larger parameter spaces and more complex dynamic behaviors. However, they also bring problems such as high computational complexity and slow iteration. This restricts the application of high-dimensional chaotic systems, especially in applications that require real-time and efficiency, where speed bottlenecks often affect their practical application effectiveness. Therefore, improving computational efficiency while ensuring sufficient security has always been a key direction in chaotic research.

The behavior complexity and sensitivity of chaotic systems determine their widespread application in image encryption. They are broadly categorized into low-dimensional and high-dimensional chaotic systems [17,18].

Low-dimensional chaotic systems are characterized by their straightforward structure and computational efficiency. Early iterations of image encryption algorithms predominantly leveraged such low-dimensional systems [19,20]. For instance, Zhou et al. [21] employed a combination of multiple low-dimensional chaotic maps for encryption. Feng et al. [22] more recently developed a novel 1D zero exclusion chaotic system that innovatively pays attention to the standard eight-pixel values, leading to enhanced encryption results. However, the simplicity of low-dimensional chaotic systems also means they have a more limited parameter space and reduced complexity in their dynamic behavior. This makes them more vulnerable to cracking under sophisticated attacks.

In contrast, high-dimensional chaotic systems offer a more intricate structure, which translates to a larger parameter space and more elaborate dynamic behavior [23–25]. Yan et al. [26] proposed two scrambling algorithms and one diffusion method by using a five-dimensional hyperchaotic system with memristors. Huang et al. [27] studied the shape dimension of 3D chaotic systems and applied it to the encryption scheme of color images. Despite these advances, the complex architecture of high-dimensional chaotic systems can lead to increased computational complexity and slower iterations. This, in turn, can affect the speed at which key streams are generated and pose challenges to the practical implementation of these algorithms.

In recent times, a plethora of innovative image encryption algorithms have emerged, a testament to the interdisciplinary synthesis of chaos theory with diverse fields [28–30]. Zhang et al. [31] addressed the issue of poor encryption performance of Galois domain multiplication operations for pixels with a value of ‘0’, and integrated DNA computing into Galois domain diffusion, achieving good improvement results. Guo et al. [32] presented a dual-channel encryption algorithm by using dual-channel ciphertexts. Huang et al. [33] presented an encryption method that utilizes the RAS algorithm and wavelet transform. Experimental outcomes have showcased the robust computational capabilities of these algorithms. However, in computer simulations, the efficiency of encoding and decoding processes is often relatively slow, and encryption systems demand stringent equipment and environmental conditions.

Deep learning has extraordinary generalization performance and nonlinearity, and thus it has been used in some encryption methods [34,35]. Man et al. [36] designed a dual-channel encryption method. The method introduces a convolutional neural network (CNN) for generating chaotic pointers. Ding et al. [37] introduced the generative adversarial network (GAN) to construct a private key. These networks are used for generating a random sequence as a key stream for the scrambling and diffusion process.

Maniyath et al. [38] proposed a deep-learning framework for improving image encryption schemes. These image encryption schemes use deep neural networks to produce an unpredictable key stream. The key stream sequence is unpredictable and suitable for image encryption systems. Sang et al. [39] presented a deep-auto-encoder-based encryption method for generating a highly random ciphertext image, which is critical for an encryption scheme. These described schemes use the deep neural network to generate secret keys and yield measurable outcomes, which motivates the exploration of more efficient integration between deep learning and chaotic systems.

The traditional one-dimensional chaotic system is elegantly articulated through its mathematical representation, as illustrated in Eq. (1). (1)xn+1=sin⁡(παxn)

x_n denotes the iterative sequences, α is an essential component.

In response to the security issue of relatively small key space in 1D chaotic sequences, an innovative 2D iterative hyper-chaotic system is presented for image encryption. The mathematical formulation is elegantly depicted in Eq. (2). (2){xn+1=cos⁡(πα⋅xnynβ)yn+1=sin⁡(πα⋅ynxnβ)

x_n and y_n are the inputs, x_n+1 and y_n+1 are the outputs. The iterative sequences x_n and y_n have a value range of (−1, 1). α and β are essential components of the two-dimensional iterative hyper-chaotic system. α ranges over R⁺, while β is constrained to N⁺.

Distinct from 1D systems, 2D hyper-chaotic system unveils a spectrum of more intricate and sophisticated dynamic attributes and patterns. Coupled with an expanded parameter space within chaotic states, it aligns more effectively with the stringent requirements of image encryption.

Chaotic sequence attractors are pivotal in delineating and understanding the inherent dynamics of chaotic systems. The existence and characteristics of these attractors are critical for gauging the stability of chaotic sequences. A stable attractor ensures that the system’s trajectory consistently gravitates towards it, regardless of the chaotic sequence’s complexity within the phase space. Systems endowed with stable attractors tend to exhibit a diminished rate of exploration, as their states revolve around the attractor’s locus. In contrast, systems devoid of stable attractors are marked by an amplified rate of traversal, as they continuously evolve and venture into novel territories within the phase space. For cryptographic systems that require the generation of unpredictable random keys, those that showcase enhanced traversal are particularly advantageous.

Fig. 2 illustrates the behavior of the two-dimensional iterative hyper-chaotic attractors under a variety of control parameters, given the initial values x₀ = 0.69344818165 and y₀ = 0.98443984189. These results compellingly demonstrate the pronounced ergodicity of the two-dimensional iterative hyper-chaotic system, showcasing its attractors uniformly distributed across the phase space. This uniform dispersion indicates that the sequence possesses the desirable quality of randomness, an attribute that is highly beneficial for cryptographic applications.

The average Lyapunov exponent is a crucial quantitative metric for the dynamic characteristics of chaotic systems. It offers the rate at which small disturbances in the system amplify over time. Consequently, the average Lyapunov exponent holds a significant reference value in the analysis of chaotic systems. A positive Lyapunov exponent value denotes chaotic behavior, with its numerical magnitude indicating the level of chaos within the system. A larger Lyapunov exponent generally signifies heightened sensitivity to initial conditions. Table 1 presents the average Lyapunov exponent of the two-dimensional iterative hyper-chaotic system across various control parameters.

As shown in Table 1, the system evinces hyperchaotic traits, with its average Lyapunov exponent notably exceeding those observed in other chaotic systems. This distinction underscores the system’s superiority, suggesting a more pronounced sensitivity to initial conditions and a richer dynamic complexity that sets it apart in the realm of chaotic systems.

The proposed framework leverages a generative adversarial network (GAN) to dynamically synthesize cryptographic keys by fusing chaos theory with deep feature learning. Specifically, the generator (G) employs cascaded convolutional blocks to transform raw image features into chaotic system parameters, which are iteratively refined through adversarial training with the discriminator (D). The optimized parameters then drive a chaotic map to produce provably unpredictable keystreams. These keystreams are subsequently integrated via pixel-level XOR (Exclusive OR) operations to achieve nonlinear substitution.

The architecture of the network is displayed in Fig. 3. The generator consists of three downsample layers, six residual blocks, two transposed convolutional layers, and an ordinary convolutional layer. The discriminator consists of five convolutional layers. In addition, to enhance the generalization performance, instance normalization is applied for all convolutional layers.

The proposed network first uses the generator to produce the keys from the original images. Then the discriminator whether the keys are real or fake. If the discrimination accuracy is nearly 50%, the generated keys have superior uncertainty and unpredictability.

Moreover, the loss function L includes the generator loss function L_G and the discriminator loss function L_D, as illustrated in Eq. (3). (3)L=LG+LD

The loss function L_G and L_D are detailed in Eqs. (4) and (5), respectively. (4)LG=minG(Ex∼pdata(x)log⁡(1−D(G(x)))) (5)LD=maxD(Ey∼pdata(y)log⁡(D(y))+Ex∼pdata(x)log⁡(1−D(G(x))))

G and D are a generator and a discriminator, respectively, x is the original image, and y is the data in the generation domain. The loss function L_G and L_D can achieve adversary during the training process. When the generator and the discriminator achieve an equilibrium state, the training results are optimal.

(1)

The key stream generation phase

 The deep learning network can generate the key streams derived from the original image. As shown in Fig. 3, the paper employs the powerful nonlinearity of the CNN architecture to derive chaotic system keys. Throughout the convolution process, the network utilizes convolution kernels of 7 × 7, 5 × 5, and 3 × 3 for better feature extraction. Moreover, we use 3× residual blocks, to enhance uncertainty, non-replicability, and unpredictability of the key streams. The generated key streams are denoted as K₁, K₂, K₃, K₄.

(2)

The diffusion and scrambling phase

 Unlike conventional diffusion starting from a fixed position, the iterative hyper-chaos encryption method permits the diffusion process to originate from any arbitrary point within the image. This approach significantly amplifies the intricacy involved in decrypting the ciphertext images. The algorithm is described in the subsequent steps.

Step 1: Randomly select the initial diffusion position P_m(m, n).

Step 2: Process keys K₁, K₂, K₃, and K₄ to achieve the initial value by Eq. (6). (6){α=mod(K1,1)×100+fix(K1);β=mod(round(K2×1010),35)+2;x0=mod(K3,1);y0=mod(K4,1).

Then, input the initial parameter α, β, and the initial value x₀, y₀ into function [X, Y] = Chaos(α, β, x₀, y₀, N), where N is the number of iterations.

Step 3: Process X, and Y using equation Eq. (7) to obtain the four matrices required for rotational diffusion. (7){X1=mod(X,256);X2=mod(Y,256);X3=mod(X+Y,256);X4=mod(X−Y,256).

Select a matrix consisting of m elements from point X₁ to form (1, m)^′ and perform XOR operations upwards from point P_m to complete the first step of diffusion; Select matrices of appropriate size in X₂, X₃, and X₄ to complete three diffusion operations to the left, down, and right, respectively.

In addition, for scrambling, Splice X, Y in function [X, Y] = Chaos (α, β, x₀, y₀, N), reorganize it into a matrix of 256 × 256, and sum each column in the matrix to obtain Y₁, Y₂ for scrambling. The scrambling process is shown in Eq. (8). (8){P(i,j)=P(Y1(1,i),Y2(1,j));i∈[1,256],j∈[1,256].

The visualization analysis of encryption methods can evaluate the encryption results through a graphical interface, helping users intuitively understand the effects of the methods.

As shown in Figs. 4–6, visualization results show that our ciphertext image appears complete visual randomness with no discernible patterns or structural features. It is demonstrably completely impossible to achieve any meaningful information from ciphertext. Moreover, the ciphertext exhibits near-perfect visual congruence with its corresponding plaintext, which sufficiently demonstrates the effectiveness of encryption and the accuracy of decryption.

Histogram analysis offers a straightforward representation of the pixel distribution. If the statistical properties are not effectively concealed within the ciphertext, it may allow malicious actors to statistically reconstruct the distribution of information in the plaintext through cryptanalysis. Therefore, the histogram distribution of ciphertext will affect the risk of ciphertext leakage.

Figs. 7–9 provide a comparative visualization of the image histograms, highlighting the transition from the plaintext distribution to the ciphertext distribution. The image histogram of the ciphertext is quite uniform, indicating that the statistical characteristics of the plaintext have been perfectly concealed and eliminated. The comparative results of the histogram underscore the ability to obscure the original statistical features, thereby enhancing information security.

The information entropy can measure the amount and complexity of information. Higher entropy values indicate greater image complexity and enhanced information content.

The information entropy of the proposed image encryption scheme is shown in Table 2. The experimental results show that the ciphertext information entropy is close to 8. Compared to the plaintext image, there is a significant improvement in information entropy, which proves that our scheme has an excellent encryption effect.

Correlation analysis is instrumental in discerning the structural composition and information distribution. Attackers may exploit this by analyzing the correlation to infer patterns, which underscores the necessity for encryption to disrupt such relationships. A strong correlation is typically evidenced by a clustered scatter plot, whereas a weak or disrupted correlation results in a more dispersed pattern. Figs. 10–12 present a comparative correlation analysis conducted on different images.

The plaintext exhibits clustered scatter plots in all directions, indicating that the plaintext exhibits strong correlations. In contrast, the ciphertext exhibits a highly dispersed pattern in all directions. The results reveal that the proposed image encryption scheme can effectively eradicate any discernible correlation, thereby enhancing data security.

A differential attack is a selective plaintext attack. The attack involves subtle modifications to the plaintext to compare the ciphertext before and after encryption. By analyzing these differences, attackers may attempt to obtain encryption keys. We encrypt the images before and after modification using the same key and introduce NPCR and UACI to analyze the differences between the two encrypted images. These two metrics are shown as follows. (9)NPCR=∑q,rT(q,r)W×H (10)UACI=1W×H∑q,r|V1(q,r)−V2(q,r)|255 (11)T(q,r)={1,V1(q,r)≠V2(q,r)0,V1(q,r)=V2(q,r)

W and H denote the width and height of the encrypted images. V₁ and V₂ denote the two encrypted images.

NPCR is used to measure the sensitivity of encryption algorithms to small changes in plaintext or key. A high NPCR value (close to the ideal value of 99.61%) denotes that the algorithm has strong diffusion and can effectively resist differential attacks. UACI reflects the degree of modification of the original pixels. A higher UACI value (close to the ideal value of 33.46%) denotes that the pixel changes significantly after encryption, which can effectively resist differential attacks. However, excessively high values may lead to excessive distortion of image information. Therefore, combining NPCR and UACE can effectively evaluate its ability to resist differential attacks.

To evaluate the resistance to the differential attacks, we compare our proposed method with the existing methods, including Refs. [17,20,24,35,40]. NPCR and UACI of various methods are shown in Table 3. Our method exhibits optimal resistance performance with an NCPR of 99.642% and a UACI of 33.465%. The method obtains high consistency in pixel variation and effectively scatters the intensity of modifications, which is challenging for detection and reverse encryption.

During information transmission, if an attacker is unable to decrypt the ciphertext, they may resort to interference techniques to prevent the receiver from successfully decrypting the message, thereby achieving their objective. Two common methods for such attacks are cropping attacks and noise attacks. Fig. 13 illustrates the decryption effectiveness of our scheme under different levels of cropping attacks, while Fig. 14 shows the decryption effectiveness under various noise attacks. The results illustrate that our method exhibits powerful resistance to different attacks.

The ablation experiments were executed and composed of 2 stages, and each stage is shown as follows.

Only 2D Iterative Hyper-Chaotic module (Module A) is carried out.

Specifically, take a picture of Cameraman, Goldhill, and Peppers as three examples. As shown in Table 4, ablation experiments have demonstrated the effectiveness of each proposed module.