Substitution boxes (S-boxes) are crucial components in contemporary cryptographic systems. An S-box functions as a nonlinear transformation, denoted by S(y):GF(2a)→GF(2b), or equivalently, as a Boolean function g(y)=(g1(y),g2(y),…,gb(y)). The primary importance of S-boxes in symmetric key algorithms is to obscure the connection between the input key and output code [1,2], thereby playing a critical role in ensuring the security of the algorithm.

Attacks, such as parallel brute forcing and linear cryptanalysis, can break an encoder if the quality of the S-box is inadequate. The DES [3] uses eight 6 × 4 S-boxes that are ultimately weak. The AES, which was introduced in 2000 as a successor to DES, uses an S-box that is resistant to linear and differential cryptographic analyses [4]. However, the use of a static S-box in AES remains unchanged despite significant variations in the input, making the algorithm more predictable and vulnerable to ciphertext-mating attacks [5]. Key-dependent S-boxes offer enhanced security; however, designing systematic methodologies for dynamic S-boxes remains an open challenge, which was partially addressed in [6]. However, large design gaps still exist in developing S-boxes that can balance high nonlinearity with strong resistance to differential attacks, necessitating the investigation of advanced metaheuristic techniques to overcome these limitations [7]. Most previous studies on metaheuristic-based S-box designs, including the use of the FA and other optimization techniques, have shown promising results in improving cryptographic properties; however, they often present limitations. Poor search diversity, particularly early convergence, is a major obstacle to achieving optimum cryptographic robustness. As a typical example, although the FA is effective in performing a search in space, it tends to converge prematurely and hence may yield suboptimal S-box designs with lower nonlinearities and weak resistance to differential attacks. In addition, most previous methods cannot balance exploration and exploitation in a complicated search space, which significantly influences the comprehensive security characteristics of the generated S-boxes. These challenges underscore the application of an advanced approach; hence, the proposed ML-F algorithm combines the strengths of the FA with the LOA for better performance in S-boxes. The motivation for this study is the increased S-box design requirements required for stronger robustness against cryptanalytic attacks in terms of higher nonlinear order and stronger resistance properties. Existing metaheuristics, such as FA have demonstrated potential; however, in practice, early convergence reduces diversity and search capability, compromising their eventual results for optimal cryptographic properties. The proposed ML-F algorithm addresses these shortcomings by combining the strengths of the LOA and FA, enhancing search capabilities, and ensuring more reliable S-box performance for secure encryption systems.

The ideal cryptographic S-box remains elusive and requires a balance between the desired properties [8,9]. There are three major design methodologies [10]: algebraic, random search, and metaheuristic methods. Each method has its advantages and disadvantages. A random search is the simplest method; however, it mostly results in S-boxes with inferior cryptographic properties [11]. Algebraic methods yield S-boxes with improved properties; however, generating large sets of strong S-boxes is difficult [12]. Another approach, metaheuristic algorithms, has been widely studied in S-box design and offers good hardware and software performances. Despite these promising approaches, several difficulties still affect S-box design and analysis [13]. For instance, early convergence and lower search diversity were major drawbacks of previous FA applications, making them unfeasible or ineffective in reaching optimality regarding cryptographic properties [14]. Nature-inspired population-based metaheuristics commonly experience poor balancing of global and local search capabilities, which results in slow convergence and premature entrapment into local optima [15]. The study proposes the ML-F algorithm, which constitutes a hybrid of the LOA and FA necessary for addressing issues. The newly developed ML-F introduces fresh search dynamics instigated by the chaotic exploration behavior of the FA and the structured search capabilities of the LOA for better scalability and performance of the S-box design. This study focuses on developing a new and improved methodology for designing an S-box that will further enhance cryptographic robustness and overcome some of the limitations found in previous metaheuristic approaches. This study outlines a strategy for overcoming two significant issues, early convergence and bounded search diversity, which are traditionally encountered in algorithms such as FA. ML-F combines the superior qualities of the LOA in enhancing features, such as nonlinearity and differential resistance, in the S-box design with those of the FA. S-boxes are relevant to secure encryption systems. Thus, the proposed approach generated S-boxes with the best cryptographic properties, possessing higher resistance against differential and linear cryptanalysis. The remainder of this paper is structured as follows: Related works on metaheuristic-based S-box design are presented in Section 2. In Section 3, a cryptographic criterion essential for a good S-box performance is elaborated. The proposed algorithm-namely, including its structure and several enhancements, is presented in Section 4. Section 5 compares the proposed approach with existing approaches, and Section 6 analyzes the cryptographic properties of the generated S-boxes. Section 7 presents the limitations of the study, and Section 8 discusses directions for future research. Finally, conclusions, key findings, and recommendations are presented in Section 9.

Numerous studies have explored metaheuristic algorithms for designing S-boxes, emphasizing the effectiveness of generating S-boxes with strong cryptographic properties. Various optimization techniques have been examined to enhance the security of cryptographic systems using robust S-box designs. SA is an optimization algorithm employed to create S-boxes with high nonlinearity and favorable autocorrelation properties [16]. The iterative nature of SA, with accepted solutions based on improvement or calculated probability, improves the S-box characteristics that are crucial for cryptographic security. However, SA has limitations, particularly in terms of execution time and risk of premature convergence. In addition to these cryptographic criteria, recent research has shown that S-boxes without fixed points, reverse fixed points, or short-period rings are stronger and more resistant to certain types of attacks. In [17], methods for generating S-boxes that avoid fixed points and reverse fixed points were discussed. These methods enhance security by eliminating various types of vulnerabilities caused by repetitive or predictable mappings. This study proposes methods for detecting and removing these properties in S-box designs to avoid weaknesses in cryptographic applications. To address these limitations, researchers have explored combinations of optimization techniques with other methods. One study proposed a method that merges ACO with chaos-based techniques to optimize the S-boxes [18]. This approach leverages the ACO’s ability to find optimal solutions by incorporating chaos for enhanced randomness and security, generating S-boxes that meet the essential cryptographic criteria. The use of ABC optimization alongside chaotic maps has also been investigated to construct robust S-boxes [19]. This method involves the generation of initial S-boxes using chaotic maps, which are subsequently refined using ABC optimization. This process ensures the creation of dynamic S-boxes with superior cryptographic performance compared with existing techniques. Additionally, FA, inspired by the flashing behavior of fireflies, has been explored for S-box designs. The FA was combined with a discrete chaotic map to improve the search capability and performance [20]. The results demonstrate the effectiveness of this approach in generating S-boxes that satisfy various security criteria, thereby enhancing security in symmetric ciphers. Building on these advancements, another study introduced a method that combined chaotic maps with the MSAA for efficient S-box generation [21]. This approach exhibited superior performance compared with existing methods, reducing the execution time and cost while achieving high cryptographic properties. The potential of the GFA combined with chaos theory has also been explored for designing robust S-boxes [22]. This method was evaluated against various cryptographic criteria, demonstrating its effectiveness in generating S-boxes that satisfy the necessary security requirements. Another study integrated discrete chaotic maps with a CS algorithm to enhance the S-box cryptographic properties [23]. The findings suggest that this combined method improves S-box characteristics, such as nonlinearity. Further advancements include the AAHC for S-box design [24], which dynamically allocates agents and uses a swap operator to refine the solutions, and a multiswarm PSO algorithm with a chaotic tent map for S-box design, which results in superior performance [25]. Finally, 3D chaotic maps combined with the WOA [26] for the S-box design showed better results in terms of execution time and cost, along with superior cryptographic properties, underscoring the need for new techniques to build robust and secure S-boxes. In addition to these algorithmic approaches, other studies have explored broader techniques for synchronization and robustness against uncertainties in chaotic systems, thereby providing useful insights into the design of cryptographic systems that are more stable and resilient. For example, the study of the sliding mode synchronization of multiple chaotic systems under uncertainties and disturbances explored techniques for maintaining system stability in the presence of dynamic variations and disturbances [27]. Such methodologies are relevant to the design of S-boxes in cryptographic systems, which inherently require the satisfaction and conservation of secure and robust properties against disruptions. In contrast, chaotic systems introduce controlled chaos into the ML-F algorithm as one of the key ways of improving exploration and convergence properties with integrated chaotic maps; therefore, the stability issues that are pertinent in classic FA and LOA are mitigated here. This agrees well with the results of research on chaotic system synchronization and allows ML-F to balance exploration and exploitation, enhance the resilience of the S-box, and ensure cryptographic robustness.

This study addresses the challenge of designing a robust S-box by evaluating it against established cryptographic criteria: bijective property, nonlinearity, SAC, BIC, DP, and MELP [18,28]. While these criteria are standard in the S-box design, achieving an optimal balance across all criteria simultaneously has proven challenging, as numerous studies have prioritized individual properties rather than holistic optimization. A new assumption of this study is the involvement of controlled chaotic dynamics in the optimization process. Traditionally, chaotic maps have been used either independently or alone in the frames of single algorithms, and chaos is introduced directly into the hybrid ML-F algorithm to enhance both the global and local search capabilities in this study [18,28]. Furthermore, it is assumed that the hybridization of the FA-LOA would reduce the limitations observed in the literature for both algorithms, as the former converges prematurely and the latter may fail to balance exploration with exploitation [8–10]. In this context, a chaotic mapping-based hybrid FA-LOA can result in more consistent performance in the S-box generated through a set of cryptographic criteria, thus reducing the gaps identified in earlier sections regarding the balanced robust cryptographic S-box [29]. Based on these standards, the S-box design creates a bijective S-box that maximizes nonlinearity and BIC, minimizes the SAC offset value, and reduces the maximum DP and MELP values. Previous studies have often focused on improving a single criterion, such as nonlinearity, with other standards serving primarily as filters for the resulting S-boxes [8–10]. However, this study aims to optimize all the criteria using new assumptions integrated into the ML-F algorithm.

Owing to the need for robust cryptographic measures, this study extends the design criteria of S-boxes to not only achieve high nonlinearity, potentially up to values, such as 116, but also ensure that there are no fixed points, reverse fixed points, and short-period rings. The fixed points in an S-box and reverse fixed points reduce the unpredictability of secure cryptographic operations. Short-period rings are pairs that, through successive applications of an S-box, quickly transform an input to its original value, and tend to destroy the effectiveness of resistance to these types of cryptanalytic attacks. In the case of ML-F, the algorithm checks and eliminates them through optimization. This algorithm combines chaotic maps and dynamic feedback systems to iteratively update the entries of the S-boxes such that none of the conditions for fixed points or short-period cycles are satisfied, simultaneously ensuring optimum nonlinearity. This approach not only strengthens the security features of the generated S-boxes but also makes them implementable for environments requiring a high degree of cryptographic applicability.

This section provides an overview of the metaheuristic LOA and FA, followed by a detailed introduction to the proposed hybrid ML-F optimization algorithm. Additionally, a discrete chaotic map, which is used to introduce the randomness required in the ML-F hybrid algorithm, is discussed. Finally, methods for generating S-boxes using these algorithms are presented.

To evaluate the effectiveness of the proposed algorithms, a series of experiments focusing on S-box generation were conducted using different swarm sizes (15, 20, 25, and 50) and iterations (200, 400, and 750) on a system equipped with an Intel (R) Core (TM) i7-8565U CPU operating at 1.80 GHz with 16 GB of RAM, Windows 10, and MATLAB R2013a. Each experiment was repeated 25 times to ensure the reliability.

The performance of swarm intelligence algorithms is highly dependent on the parameter settings, particularly because the focus of this study was to enhance the FA movement strategy. Therefore, the parameters for FA and ML-F were set as follows: β0=0.1, βmin=0.1, γ=1.0, α=0.2, and δ=0.96, in line with the recommendations from previous research [36]. Here is a refined version of the paragraph, aligned with the updated Table 1.

Table 1 illustrates the nonlinearity of the S-boxes constructed using the FA and ML-F for different swarm sizes and iteration counts. The “Best Nonlinearity” column yields the best nonlinearity in each experiment, the “Worst Nonlinearity” column yields the lowest value, and the “Average Nonlinearity” column provides the average of the fitness score over all configurations. At 200 iterations, ML-F outperformed slightly in terms of stability, yielding a higher “Worst Nonlinearity” and more consistent “Average Nonlinearity” for different swarm sizes. When the number of iterations was increased to 400 and 750, the difference in performance between ML-F and FA became evident. For example, at 400 iterations with a swarm size of 25, ML-F reached a “Best Nonlinearity” score of 120, while FA reached only 119.2, which shows the stronger capability of ML-F in achieving high nonlinearity values. Both algorithms improved the nonlinearity with an increase in the swarm size. However, greater scalability was observed with ML-F, which maintained higher nonlinearity values with increased swarm size and iteration count. The best “Best Nonlinearity” value of ML-F was 120 when the number of iterations was 750 with a swarm size of 50. Furthermore, it maintained a stable and reliable performance through configurations, whereas its “Worst Nonlinearity” was 119.5. In contrast, a significant drop in the nonlinearity scores for several configurations was observed in the FA. For example, when the number of iterations was 200 and the swarm size was 15, the worst nonlinearity score for FA was 115.0, whereas ML-F had a minimum score of 117.5, which underlines the robustness of ML-F and its lower tendency toward generating weaker S-boxes. The average fitness values confirm the dominance of ML-F: at 400 iterations and a swarm size of 25, ML-F managed to get an “Average Nonlinearity” of 119.3, against 118.7 obtained by FA. This trend of higher averages in the scores across all iteration counts signifies the capability of ML-F to create S-boxes with consistently better nonlinearity. The best overall results were observed at 750 iterations-run with a swarm size of 50 for ML-F: it achieved a “Best Nonlinearity” value of 120 and “Average Nonlinearity” of 119.8. The results in Table 1 indicate the advantage that ML-F has over FA in generating better and higher nonlinear S-boxes for different swarm sizes and iterations. The performance of ML-F resulted not only in higher best-case nonlinear scores but also was more resistant to worst-case scenarios. Similarly, its capability of having higher average fitness values while providing scalability with a larger swarm size establishes it as more robust and efficient in cryptographic applications.

The results in Tables 2 and 3 indicate that increasing the swarm size improves the quality of the generated S-box to some extent, although the nonlinearity gains in the FA are modest relative to the required increase in swarm size. By contrast, ML-F exhibited substantial improvements, particularly between 200 and 400 iterations. However, further increasing to 750 iterations did not significantly enhance the best S-box nonlinearity.

A robust S-box satisfies the criteria described in Section 2. The performances of the S-boxes generated by the two methods were compared with those of other metaheuristic approaches, such as FA, BFO [39], GA [28], ABC [7], TLO [40], LFM [41], SA [42], ACO [18], and CFA [20], which also used nonlinearity as an objective function.

Empirical research is often subject to various validity threats, both external and internal. External validity threats emerge when experimental results cannot be generalized to real-world applications. In this study, such threats were mitigated by employing widely accepted benchmarks from the literature that reflect realistic and commonly encountered cryptographic configurations. Internal validity threats arise from factors that can inadvertently influence the outcomes of the study, including variations in the population size, number of iterations, and specific parameters used for each algorithm. Considering that the source code for all the methods was not available, it would be challenging to confirm whether the ML-F and LOA methods were evaluated using an identical number of fitness functions. Nonetheless, the comparisons presented in this study are considered valid because the published S-box results were obtained using optimal control parameter settings, thereby minimizing the impact of external factors. In addition, the performance was evaluated based on the averaged results to account for the stochastic nature of each metaheuristic run. Comparing ML-F with methods such as TLO could be misleading because of the inherent differences in their operational mechanisms, such as the dual phases of TLO, which require a greater number of fitness function evaluations. Moreover, although TLO is often cited as parameter-free, it still requires careful tuning of the iteration counts and population sizes. Recent scrutiny of TLO has highlighted concerns about unreported implementation steps that can result in unfair comparisons. Another internal validity threat involves the generation time, which is significantly influenced by the computational environment. A fair comparison of generation times necessitates that all methods are executed under identical conditions.

Limitations

Although the performed study illustrates that the ML-F algorithm provides an effective way of generating highly nonlinear and robust S-boxes, there are specific limitations. The first is that a larger swarm size with an increase in iterations requires more computation time and execution, which is perhaps not feasible in real-time or resource-constrained conditions. Although the algorithm has been tested for a range of swarm sizes and iterations, the configurations are still limited, and further research is required to explore its performance under different environmental parameters or on different hardware configurations. More importantly, although the optimization in this study focused on nonlinearity and resistance to linear and differential cryptanalysis, not all types of cryptographic weaknesses, such as fixed points or reverse fixed points, were considered. The performance has a large remaining margin, which will allow refinement towards fully comprehensive cryptographic strength. The results of this study are obtained from simulations. Real implementations may have completely different performances because of additional factors unrelated to this study, such as hardware variability or specific operating limitations. Future studies may thus overcome these two limitations, with a primary focus on ML-F optimization in terms of computational efficiency and the assessment of its real performance against practical cryptographic requirements.

S-boxes play a crucial role in symmetrical cryptosystems by providing essential confusion, as highlighted by Shannon, to obscure the relationship between the encryption key and plaintext, thereby enhancing the resistance of the cryptosystem to attacks. In this study, we introduced a novel method for generating S-boxes based on a hybrid ML-F optimization algorithm. This approach leverages the exploration and exploitation capabilities of both the LOA and FA, which jointly address limitations, such as early convergence and limited search diversity often observed in standalone metaheuristic algorithms. These experimental results confirm that the proposed ML-F algorithm is satisfactory for some important cryptographic attacks, such as nonlinearity analysis, BIC, SAC, DP, and MELP. The ML-F-based generated S-boxes maintained their nonlinearity scores highly in comparison to the S-boxes generated by FA and other compared algorithms each time while varying the swarm sizes and iteration counts used in ML-F. From these experimental results, it can be concluded that ML-F obtains better S-boxes with higher resistance to both linear and differential crypt analyses.

One of the striking features of ML-F is its scalability, with an increase in iterations and swarm size, yielding robust performance under larger configurations. In contrast, the FA demonstrated greater fluctuation and vulnerability to local optima, particularly for smaller iterations. This consistency in producing high values of best-case, worst-case, and average metrics indicates the reliability of ML-F for cryptographic applications. However, this study has some limitations. One major limitation is that the increased swarm size and number of iterations boost the computational time, which might limit the practice of ML-F in real-world applications with reduced resources or in real time. Moreover, this study focused only on the optimization of important cryptographic properties, such as nonlinearity and resistance against differential attacks. Other possible vulnerabilities include fixed points or reverse fixed points, which are other important aspects of cryptographic strength. These limitations include avenues for further research, in which better computational efficiency and more cryptographic criteria may be integrated to enhance the applicability of ML-F. This implies that, eventually, ML-F is an effective and promising approach toward the optimization of S-box properties because of its persistent improvement in nonlinearity, along with the general cryptographic security of various configurations. Further research might address improving the computational efficiency of ML-F and exploring methods for adaptive tuning to optimize the ML-F performance. In addition, other practical cryptographic applications may be promising avenues to which the ML-F algorithm might be deployed, considering the great potential that this can provide for the robustness and scalability of cryptographic systems. Extending the scope of the application of ML-F by real-time tests and investigating its hardware implementation could provide a more solid basis for establishing the practicality and effectiveness of the proposed MLF on modern cryptography.

This paper presents the ML-F algorithm as an efficient way of generating highly nonlinear and robust S-boxes, although a few avenues for future research remain open. Future work may be undertaken to enhance the computational efficiency of ML-F, especially to optimize its performance for real-time applications and resource-constrained environments. Adaptive tuning techniques for parameters or exploring parallel processing may reduce the execution time without compromising cryptographic strength. In addition, it can be extended further to enhance the robustness of the S-boxes generated by ML-F with more security criteria beyond nonlinearity, bit independence, and the properties of differential approximation. Other possible lines of research work may also include resistance against fixed points, reverse fixed points, and short-period cycles, which are important for several advanced cryptographic applications. Further investigations were conducted to test the ML-F algorithm in practical cryptographic contexts, such as encryption schemes for secure communications, image encryption, and IoT security. In addition, based on recent studies related to the design of S-boxes, such as weighted Loeplitz matrices for data encryption and corner-modified symmetric Toeplitz matrices for image encryption, one can gain insight into some feasible and new applications of the ML-F algorithm in cryptographic infrastructure. Addressing these issues will refine the ML-F approach in future research, thereby expanding its application and further increasing its viability in various cryptographic settings.