The smart grid (SG) represents a transformative leap in energy management, integrating advanced digital technology into the traditional power grid to enhance efficiency [1–3], reliability, and sustainability [4]. As an integral component of this modernization, communication technologies play a pivotal role, facilitating real-time data exchange and control across various grid components [5]. Within this context, non-orthogonal multiple access (NOMA) has emerged as a significant advancement, offering a paradigm shift in SG communications [6]. NOMA stands out by enabling multiple users to share the same frequency resources, thereby drastically increasing spectral efficiency and network capacity [7]. This is particularly crucial in SG environments, where the need to simultaneously connect a multitude of devices, such as smart meters (SMs) and renewable energy sources, is ever-growing. By efficiently managing these dense and diverse communication demands, NOMA not only addresses the scalability challenges of the SG but also contributes to the overall optimization of energy distribution and consumption, heralding a new era of intelligent energy management [8].

Security concerns in SG are paramount, given the critical nature of energy infrastructure and the sensitive data involved in its operation [9]. As SGs become increasingly interconnected and reliant on wireless communications, they become vulnerable to various cyber threats [10–12]. One notable security threat in the SG neighborhood area networks (NAN) is the risk of eavesdropping and impersonation attacks. For instance, an attacker might position themselves as an eavesdropper within the communication range of a neighborhood gateway (NG) and the SMs it controls. By intercepting the communication, the attacker could gain unauthorized access to confidential information, such as consumption data or control commands. More alarmingly, they could impersonate the NG, sending fraudulent signals or commands to the SMs. Such an attack could lead to severe consequences, including the disruption of power distribution, manipulation of billing data, or even causing physical damage to the grid infrastructure. This scenario underscores the critical need for robust security mechanisms in SG communications, to prevent unauthorized access and ensure the integrity and reliability of the energy supply chain [13].

The importance of secrecy performance analysis in designing security schemes for SG communications cannot be overstated [14–16], particularly in the context of emerging technologies like NOMA. Secrecy performance analysis is crucial for evaluating how well a communication system can protect against unauthorized interception and ensure the confidentiality of transmitted data [16–18]. A key metric in this analysis is the secrecy capacity, which is defined as the maximum rate at which information can be reliably transmitted to the intended receiver while ensuring that an eavesdropper gains negligible information [19–21]. In NOMA SG communication, optimizing secrecy capacity poses a unique challenge. NOMA systems are inherently designed to allow multiple users to share the same frequency resources, which increases the complexity of maintaining secure communications [22,23]. The shared spectrum means that the signals intended for legitimate users can be more susceptible to interception by eavesdroppers. Optimizing secrecy capacity in this context involves not only enhancing the signal strength at the intended receivers but also minimizing the information leakage to potential eavesdroppers [24,25]. This requires sophisticated strategies that can dynamically adapt to the varying channel conditions and user positions typical in SG environments, ensuring robust and secure communication against the backdrop of NOMA’s spectral efficiency benefits [26–28].

There is limited research on applying deep learning (DL) and reinforcement learning (RL) models to improve secrecy in NOMA communication systems. Ali et al. [14] developed advanced resource allocation strategies for future communication systems, focusing on maximizing the total transmission rate within a restricted power budget and ensuring a necessary power differential among users for effective NOMA deployment. They introduced a deep neural network (DNN) framework to determine a combined power allocation strategy for both source and relay nodes. To support the training and validation of the DNN, they also obtained an optimal solution using convex optimization methods, which served as a benchmark to evaluate the DNN solution’s effectiveness. It was found that the DNN solution delivers promising outcomes in terms of both sum rate and computational efficiency.

Given the notable gap in SG literature regarding the lack of a robust secrecy performance optimization scheme in NOMA communications, this paper introduces a pioneering approach based on RL to fortify this critical aspect. Recognizing the complexity and dynamism inherent in SG communication systems, especially under the NOMA paradigm, our research proposes leveraging the adaptive and predictive capabilities of RL. RL is selected over other machine learning (ML) methods for secrecy optimization in SG communications due to its distinct capabilities in handling dynamic and complex environments. Unlike static ML models like K-nearest neighbors (KNN), support vector machine (SVM), RL excels in adapting to evolving network conditions by continuously learning optimal policies through interactions with the environment, making it particularly suited for the unpredictable nature of SGs. Additionally, RL’s proficiency in sequential decision making allows it to optimize long-term secrecy performance by considering the future implications of current actions, which is crucial for maintaining secure communication over time. This novel approach is specifically designed to enhance the secrecy capacity rate, a vital metric of secrecy performance, in NOMA communications within SG environments. By employing RL algorithms, our method aims to adjust communication strategies intelligently and dynamically in response to varying network conditions and potential security threats. This allows for the optimization of secrecy capacity rates, ensuring that sensitive data transmitted across the SG remains secure from eavesdroppers and malicious actors. Our research, therefore, stands at the forefront of addressing a critical, yet previously unexplored, aspect of SG communications, offering a significant contribution to the advancement of secure and resilient SG networks.

The training process of a fully connected neural network, commonly used in RL, is a critical phase where the network learns to approximate the optimal policy for decision-making. In RL, a fully connected neural network, also known as a deep Q network (DQN) when used in Q-learning, is often responsible for mapping states to action values. The quality of this mapping directly influences the agent’s ability to make intelligent decisions that maximize the cumulative reward over time. The importance of the training process lies in its ability to capture the complex relationships between the actions, the state of the environment, and the received rewards. Proper training ensures that the neural network generalizes well to unseen states, enabling the RL agent to perform well across the entire state space of the problem. Motivated by the need for efficient and effective training of the fully connected layers in the RL network, we employ an improved chimp optimization algorithm (IChOA) to update the parameters of the neural network, which is inspired by the intelligent hunting behavior of chimpanzees in nature.

The choice of combining RL with IChOA to enhance secrecy performance in SGs is driven by the need to address the complex and dynamic nature of SG communication environments, particularly under the NOMA paradigm. SGs are characterized by their high connectivity and reliance on wireless communication, which inherently increases the risk of eavesdropping and other security threats. RL offers a robust framework for optimizing secrecy capacity by dynamically adapting communication strategies to counteract these threats, ensuring that sensitive data remains secure. However, the effectiveness of RL heavily depends on the efficiency of its training process, where the optimization of neural network parameters plays a crucial role in determining the agent’s ability to make intelligent decisions under varying network conditions. The integration of IChOA into the RL framework is justified by its ability to enhance the training process, specifically by improving the convergence speed and robustness of the learned policies. This combination allows the RL agent to learn more effective policies faster and with greater accuracy, thereby improving the overall secrecy performance. By comparing the proposed IChOA-RL method against other state-of-the-art DL and ML algorithms, the paper demonstrates that this approach not only surpasses traditional methods in terms of scalability, accuracy, and convergence but also provides a more effective solution for the specific challenges of optimizing secrecy in SG communications.

This section delineates the proposed RL technique aimed at optimizing the secrecy rate within the established SG NOMA communication system. In the context of RL, the IChOA is utilized to optimize the weights and biases of the fully connected neural network. It updates the network parameters in a way that the resultant policy maximizes the expected rewards.

The simulation environment is configured to evaluate the secrecy rate performance of an SG NOMA communication system under various ML and RL algorithms. The setup includes an NG transmitting to several SMs in the presence of an eavesdropper. The number of SMs K=3 unless otherwise mentioned. The performance metrics are assessed against the power allocation coefficient (γk in dB) and the total transmission power (P in mW). The algorithms compared with our proposed IChOA-RL in the simulation include traditional ML approaches like standard ChOA, GWO, LSTM, RNN, KNN, SVM, and standard RL model. The selection of comparison algorithms in this study was carefully made to ensure a comprehensive evaluation of the proposed IChOA-RL approach. These algorithms were chosen for their relevance to SG communications, their diversity in representing both traditional ML and advanced optimization techniques, and their proven track records in tasks such as classification, prediction, and optimization. The scope of potential values for each parameter during the simulation process has been extensive; however, due to pragmatic constraints, it is necessary to choose and exhibit a limited set of diverse parameter instances. Table 1 provides a snapshot of this selection, shedding light on the experimental process by emphasizing the specific parameter values that, in certain instances, either enhanced or diminished the performance of the algorithm.

Calibrating parameters for ML algorithms is critical for achieving peak performance and demands careful consideration. It entails identifying the best combinations of parameter values for the algorithms to function efficiently. Establishing these optimal settings is crucial before proceeding with the performance evaluation of the algorithm. In this research, we adopt a systematic trial-and-error approach for parameter tuning, methodically adjusting each parameter separately and monitoring its impact while maintaining all other variables constant. For instance, in an algorithm with multiple parameters such as the number of hidden layers, or iteration, we analyze each parameter independently to assess its effect on the algorithm’s performance. Although there are numerous possible variations for each parameter, practical constraints require us to select and demonstrate a limited range of different parameter scenarios. For our simulations, we utilized OpenAI Gym as the primary simulation environment for training the RL agents. Additionally, we integrated TensorFlow to implement the neural network components of the RL algorithm. To incorporate and evaluate the proposed evolutionary algorithm for updating the fully connected layers in this paper, we employed the distributed evolutionary algorithms in python (DEAP) library.

Fig. 4 presents a detailed analysis of the secrecy rate’s dependency on the power allocation coefficient γk, as represented in decibels (dB), across various algorithmic strategies. In this figure, the proposed IChOA-RL approach consistently outperforms the other methods, showcasing a superior secrecy rate across the entire γk range. This suggests that the IChOA-RL’s optimization process is effectively enhancing the RL agent’s ability to allocate power in a way that maximizes the secrecy rate, regardless of the power coefficient’s magnitude. The RL method alone shows notable improvement over the traditional ML techniques of RNN, LSTM, KNN, and SVM, which indicates the inherent advantage of adaptive learning in dynamic environments. However, KNN and SVM, despite being less dynamic, provide a baseline performance that, while not scaling as well with increased γk, still contributes to our understanding of the impact of power allocation on secrecy. The graph also indicates a diminishing return on the secrecy rate as the power allocation coefficient increases, particularly for KNN and SVM, suggesting a threshold beyond which increasing power does not yield proportional secrecy gains. Overall, the performance trends in Fig. 4 highlight the effectiveness of integrating advanced optimization techniques like IChOA with RL in enhancing secure communications in SG NOMA communication.

Fig. 5 provides an insightful illustration of how the secrecy rate varies under the proposed IChOA-RL approach with the power allocation coefficient γk for different quantities of NOMA SMs, designated as K. The curves represent four distinct scenarios, with K taking on values of 6, 10, 14, and 18, respectively. As can be seen in this figure, when γk increases, a corresponding incremental rise in the secrecy rate is observed for each scenario, which aligns with the theoretical understanding that a higher power allocation coefficient enhances the signal’s robustness against potential eavesdropping, thus improving secrecy.

Notably, the rate at which the secrecy rate increases with γk is more pronounced as the number of SMs, K, grows. This indicates a multiplicative effect of NOMA’s power domain exploitation when more SMs are present; essentially, the system can better differentiate between the intended signal and potential eavesdropping attempts. The scenario with K = 18 SMs achieves the lowest secrecy rate, suggesting that a larger network of SMs can utilize the intrinsic properties of NOMA more effectively, translating into superior secure communication capabilities. This could be due to the lesser complexity and reduced efficacy in channel use when fewer SMs are involved. These trends collectively highlight the effectiveness of NOMA in enhancing secure communications, particularly as the number of participating SMs in the network increases.

Fig. 6 delves into the relationship between the total transmission power, denoted by P and measured in milliwatts (mW), and the resulting secrecy rate, offering a comparative analysis across different algorithmic approaches. As the transmission power increases, all techniques exhibit an upward trend in secrecy rate, indicative of the direct correlation between transmission power and the ability to maintain secure communications. The IChOA-RL technique demonstrates a clear superiority, achieving higher secrecy rates at any given power level. This suggests that the IChOA’s sophisticated optimization algorithm significantly refines the RL agent’s power allocation decisions, leading to more effective secrecy enhancements.

Notably, the slope of the IChOA-RL curve is steeper than that of the other methods, especially in the mid-range of the power spectrum, indicating a more efficient conversion of increased power into higher secrecy rates. This efficiency is a critical advantage in real-world applications where power resources are limited and must be used judiciously. The RNN, LSTM, KNN, and SVM methods, while showing improvements with increased power, plateau sooner than the RL-based approaches, revealing the limitations of static models in leveraging additional power for secrecy. The GWO-RL, I-CSA-RL, ChOA-RL, and RL curves, while outperforming the traditional ML models, still lag behind the IChOA-RL, underscoring the impact of the improved optimization algorithm on RL’s adaptability and performance. Fig. 6 finally illustrates not only the beneficial impact of higher transmission power on secrecy rates but also underscores the enhanced performance that can be achieved by a more powerful algorithm.

In this paper, the results were evaluated using three metrics: accuracy, R2, and RMSE. The coefficient of determination quantifies the correlation between observed and predicted values, with values ranging from 0 to 1. A value of one signifies perfect correlation, whereas a value of zero indicates no correlation between the observed and predicted values. Eqs (19)–(21) provide the formulas for calculating R2, RMSE, and Accuracy. (19)R2=[1N∑i=1N[(Pi−P¯)(Oi−O¯)]σpσo]2 (20)RMSE=(1N∑i=1N[Pi−Oi]2)12 (21)Accuracy=TP+TNTP+FN+FP+TNwhere N is the number of observations; Pi is the calculated parameter; Oi is the observed parameter; P¯ is the average calculation parameter; O¯ is the average observations parameter; σp is the standard deviation of calculations; σo is the standard deviation of observations; TN = true negative; TP = true positive; FN = false negative; and FP = false positive.

Table 2 displays R2, accuracy and runtime results for various evolutionary architectures designed to secure NOMA communication in SGs. The data clearly indicate that the IChOA-RL architecture outperforms the others in terms of both R2 and accuracy, not just in the training set but also in the validation set. The IChOA-RL architecture achieved accuracy levels of 97.41% in the testing set and 98.86% in the training set. When it is stated that the IChOA-RL architecture has the highest R2 value, it implies that this architecture most accurately captures and explains the variations in the problem. In other words, it fits the actual data points best and offers the most reliable predictive power among the architectures evaluated. The I-CSA-RL, GWO-RL, ChOA-RL, and RL models also recorded relatively strong performance. Conversely, the LSTM, RNN, SVM, and KNN algorithms demonstrated lower effectiveness. When comparing the runtime across the different methods, IChOA-RL demonstrates a significant advantage with a runtime of 724 s, making it the most efficient among the advanced algorithms evaluated. In contrast, I-CSA-RL, GWO-RL, and ChOA-RL require substantially more time, with runtimes of 985, 1024, and 896 s, respectively, indicating higher computational demands. Traditional methods like RNN, LSTM, SVM, and KNN show moderate runtimes ranging from 869 to 941 s, with the standard RL method being relatively faster at 659 s. This comparison highlights IChOA-RL’s efficiency in delivering high performance without incurring excessive computational costs.

These results highlight the successful training of these architectures with meta-heuristic algorithms, which have effectively optimized their operational efficiency. Moreover, these architectures consistently demonstrate high accuracy across different hybrid RL structures in both testing and training datasets. This consistent performance suggests that the meta-heuristic algorithms used in the training processes have delivered reliable and uniform results across various models and datasets. The RMSE metric is used to evaluate the performance of the models presented in Table 3. The results clearly show that the IChOA-RL surpasses its competitors, highlighting its effectiveness for the problem at hand. This model enhances the RL network by efficiently updating its weight and bias vectors through the integration of IChOA. The IChOA effectively tunes the parameters, enabling the RL network to more accurately detect and model the patterns and relationships in the data. According to Fig. 7, the IChOA-RL converges more quickly than the others. By the 100th epoch, it almost reaches the lowest RMSE score, while the RMSE scores for the other architectures remain higher. Additionally, the IChOA-RL shows exceptional stability and swift convergence as epoch’s progress. The significant initial drop in RMSE for the model showcases a strong capacity for learning, and its sustained low error rate suggests it generalizes well across the dataset. In contrast, other models gradually improve but fail to achieve the low RMSE scores of the IChOA-RL. For example, SVM and KNN exhibit a slower reduction in RMSE. Other architectures like I-CSA-RL, GWO-RL, ChOA-RL, RL, RNN, and LSTM show moderate learning speeds. They manage to lower the RMSE to a commendable level, yet their convergence trajectories indicate they may need additional epochs to potentially equal the performance of IChOA-RL.

The computational complexity of the proposed RL technique primarily hinges on the intricacies of the RL algorithm itself and the optimization process facilitated by the IChOA. RL, particularly in environments modeled as MDPs, involves substantial computational effort due to the need to explore and learn optimal policies through interactions with the environment. The computational complexity of a DQN technique involves several key components, including the neural network architecture, the number of states, the number of actions, and the number of iterations required for convergence. Integrating the IChOA into this framework adds another layer of computational complexity. IChOA enhances the training process by optimizing the parameters of the RL network, leading to more robust policy learning. The complexity of IChOA, like other meta-heuristic algorithms, depends on the population size, the number of iterations, and the computational cost of evaluating the fitness function. In this paper, the total computational complexity (C) of the proposed IChOA-RL model is calculated as Eqs. (22)–(24). (22)CRL=T.O(B.N.M) (23)CIChOA=O(P⋅D)+G⋅O(P⋅F)+G⋅O(PlogP)+G⋅O(P⋅U)

(24)C=H.CRL+H.CIChOAwhere CRL is the computational complexity of RL, CIChOA is the computational complexity of IChOA, T is the total number of iterations for convergence, B is the mini-batch size used during training, N is the total number of neurons in the neural network, M is the average number of connections per neuron, P is the population size, G is the number of generations, F is the complexity of the fitness function, D is the number of dimensions, U is the complexity of updating positions for one individual, and H denotes the number of times the IChOA process is invoked within the DQL training process. The proposed IChOA-RL technique effectively addresses scalability challenges in smart grid implementations by integrating the IChOA for efficient training, enabling rapid convergence and adaptive policy updates in response to real-time data. This approach optimizes resource use, ensuring the method can operate within the constraints of existing smart grid infrastructure, from high-power servers to lower-power edge devices. Extensive simulations validate the method’s ability to maintain high performance and adaptability across various network conditions and scales, demonstrating its robustness in managing large-scale smart grid networks. This scalability is essential for widespread deployment in complex smart grid environments, where efficient resource management and dynamic adaptability are crucial.

This paper has presented an in-depth exploration of a novel RL-based strategy for optimizing secrecy performance in an SG environment utilizing NOMA communication. By integrating IChOA to adjust the parameters of a fully connected neural network within the RL framework, we have demonstrated a significant enhancement in the secrecy rates across a range of operational scenarios. The IChOA-RL model was compared against eight other ML architectures. The IChOA-RL model achieved the highest accuracy, recording 97.41% on the validation datasets, making it the most effective approach. Our simulation results have conclusively shown that the IChOA-RL method outperforms traditional ML approaches such as RNN, LSTM, KNN, and SVM, as well as standard RL techniques. The robustness of IChOA-RL was particularly evident in its superior performance at higher power allocation coefficients and transmission power levels, showcasing its potential for practical implementation in real-world SG systems. The scalability of the NOMA communication system was also put to the test, giving insights into the relationship of the number of NOMA SMs with the utilization of the power domain for enhancing secrecy rates, as indicated by the higher slopes in the secrecy rate curves as the number of SMs. This finding underscores the importance of considering user density in designing secure SG communications. Furthermore, the study has contributed to the body of knowledge by highlighting the critical role of sophisticated optimization algorithms in RL. The application of IChOA to the training process of the neural network has been shown to significantly accelerate learning and convergence to optimal policies, ensuring efficient use of power resources while maintaining high levels of security.

Implementing the proposed IChOA-RL technique in real-world SG environments faces several challenges. The significant computational complexity and resource demands of the hybrid method require substantial processing power and memory, making real-time applications potentially costly and impractical. Scalability is also a concern, as the SG’s vast network size demands efficient handling without performance degradation or exponential computational increases. Ensuring real-time adaptability and convergence is crucial, as the RL algorithm must quickly adapt to the dynamic conditions of the SG to maintain optimal performance. Integration with existing SG systems poses further challenges, requiring seamless incorporation without disrupting current operations while ensuring interoperability and regulatory compliance. While the paper addresses some of the practical challenges associated with implementing the IChOA-RL approach in SG environments, there are additional considerations that future research could explore in greater depth. These include real-time processing requirements, data quality issues, energy consumption, and security concerns. Addressing these challenges through innovative solutions and rigorous testing will be essential to realize the full benefits of the proposed method in enhancing the security and efficiency of SG communications.

Moreover, the IChOA-RL method may face difficulties in converging to a global optimum in highly complex or non-convex problem spaces, particularly if the initial conditions or parameter settings are not well-tuned. This is a common challenge shared with other evolutionary algorithms and advanced optimization methods like RL, RNN, LSTM, SVM, KNN, GWO, and I-CSA, which also require careful parameter tuning and can suffer from premature convergence or getting trapped in local optima. However, compared to these algorithms, IChOA-RL’s advantage lies in its ability to adapt more dynamically to changing conditions, albeit at the cost of potentially higher computational demands. In summary, while the IChOA-RL method offers superior performance in terms of adaptability and scalability, its limitations include increased computational requirements and the need for careful tuning to ensure convergence, challenges that are also in other state-of-the-art ML algorithms. Additionally, several unresolved questions regarding the IChOA and RL underscore the need for further investigation in this field. Future studies on the IChOA should delve into refining the algorithm’s specific parameters and thresholds. Such research could involve detailed assessments of how parameter variations affect the algorithm’s rate of convergence, the quality of solutions, and computational efficiency. Researchers might consider employing strategies like meta-heuristic parameter tuning or adaptive adjustments to dynamically optimize parameters during the process. Meanwhile, the development of RL models is likely to evolve towards overcoming the challenge posed by the scarcity of labeled data. This shift may lead to a stronger focus on semi-supervised and unsupervised learning methods. Future efforts could also examine the integration of IChOA into these learning frameworks to better leverage unlabeled data, thus enhancing the performance and generalization capabilities of RL models.