Software-Defined Networking (SDN) enables centralized intelligence and programmability across the application, control, and data planes, facilitating efficient management of IoT and IIoT ecosystems [9]. However, the rise of sophisticated IoT botnets such as Mirai and Gafgyt, capable of large-scale DDoS and remote exploitation, has rendered traditional detection techniques inadequate, demanding advanced AI-driven behavioral and network-based defenses [10].

DL-driven architectures remain in their early stages for securing IoT networks [11]. In [12], an LSTM-based botnet detection model using CVUT traffic data achieved 99.9% accuracy. Similarly, reference [13] proposed a Bidirectional LSTM for IoT-botnet detection, attaining 96% accuracy on a Mirai-based dataset. Reference [14] combined CNN and RNN on CTU-13 and ISOT datasets, achieving 99.3%detection, while reference [15] applied LSTM to analyze content and metadata, yielding 98% accuracy. In [16], multiple DL models (LSTM, RNN, CNN) identified malicious domains with a 90% detection rate. Reference [17] emphasized deep learning’s strength in multi-level abstraction and anomaly detection. Further, reference [18] utilized LSTM and SVM for detecting various attacks (DoS, DDoS, Port Scanning, etc.) in SDN-IoT using SDN-IoT and SDN-NF-TJ datasets, achieving 97% accuracy. Likewise, reference [19] presented a DNN-LSTM hybrid for fog-based SDN environments using N_BaIoT 2018, achieving 99.98% accuracy. Reference [20] proposed a GRU-LSTM hybrid for distributed attack detection using NSL-KDD, achieving 87.9% accuracy. In [21], an SDN-based IDS using LSTM achieved 91.4% on CSE-CIC-IDS2018, while reference [22] reached 99.9% with LSTM on N_BaIoT 2018 for binary botnet detection. Our proposed SDN-empowered system aims to enhance IoT security in scalability, performance, and control-plane protection. Unlike existing works, it addresses the lack of next-generation IoT datasets, limited training instances, and absence of comprehensive evaluations using standard and extended performance metrics. Table 1 summarizes the comparison of related work and key characteristics.

The basic architectural description of proposed algorithms and dataset are discussed below.

We utilized the N_BaIoT dataset, developed by the University of Negev, Israel, to address the limitations of outdated botnet datasets [29]. This dataset captures evolving IoT attack patterns, including Mirai and Gafgyt, across nine device types such as security cameras, webcams, doorbells, thermostats, and baby monitors. Table 2 summarizes the dataset, which comprises 7,062,611 records, of which 4,590,136 were used for experimentation, including 292,044 benign and 2,405,593 attack instances. The dataset contains 116 attributes, with 115 features and one label.

Although the experiments were conducted solely on the N_BaIoT dataset, its rich diversity of benign and malicious traffic traces collected from multiple IoT devices under real-world conditions supports strong model generalization. The dataset encompasses dynamic traffic behaviors, device heterogeneity, and a wide range of attack signatures (e.g., Mirai and Gafgyt), making it representative of modern IoT network environments. Consequently, the proposed framework can be readily adapted to other IoT and IIoT datasets with minimal retraining, ensuring scalability and applicability to real-world deployments.

The dataset utilized in this paper contains records of various IoT device. This dataset needs to pre-processed for removing anomalies and the extraction of useful features from data. The preprocessing of the N_BaIoT2018 dataset has been completed. Records having missing, nan, or infinite values were first removed from the dataset. To enhance the quality of the dataset, the normalization is likewise accomplished, which incorporates scaling all values from 0–1 range by using the MinMaxScalar function. The preprocessing of the dataset has been done by importing all CSV’s files. After that, the 6 distinct CSV’s files merged together and labels are encoded against each sample. At last, the merged CSV file is saved.

Feature selection is applied prior to classification to enhance malware detection accuracy while reducing the computational complexity of deep learning model training. Recent studies highlight the benefits of combining multiple feature filtration methods, as individually weak features can collectively improve classifier performance. In this work, features were extracted using five techniques, Variable Clustering Attribute, One-R Attribute Evaluation, Attribute Evaluation, Correlation, and Principal Component Analysis (PCA), as summarized in Table 3. The less common feature selection techniques include the Variable Clustering Attribute method, which groups correlated features to minimize redundancy, and the One-R Evaluation, which employs simple rule-based accuracy to enhance interpretability and computational efficiency. Experiment is performed on Weka [30] for feature filtration. For experimentation, total number of samples are 4,590,136 out of which 292,044 benign and 4,298,092 botnet signatures. From botnet signatures, 1,892,499 Gafgyt, and 2,405,593 are Mirai attack. To achieve efficient and accurate results, we have performed feature filtration techniques to get significant features using 5 distinct techniques, and finally we applied majority voting mechanism to get 74 best features. Besides, 75th feature is a label representing a total of 75 features.

We implemented three deep learning frameworks: Hybrid GRU–LSTM, Hybrid CNN (2D–3D), and Hybrid GRU–DNN. Their detailed architectures, including layers, neurons, activation and loss functions, epochs, batch size, and optimizer, are summarized in Table 4. All models were trained using categorical cross-entropy, a batch size of 128, learning rate of 0.001, and the Adam optimizer, with no dropout layers employed. Training and cross-validation are outlined in Algorithm 1, which describes the execution flow for merging two classifiers. During training, “T” denotes the training dataset, “t” the testing dataset, and “W” the weights for GRU (G) and LSTM (L). Nested loops iterate over epochs (“E”) and batches (“B”) to compute loss and update weights, followed by evaluation on the test set.

The computational complexity of proposed algorithm (i.e., Hybrid GRU-LSTM) is dependent upon two different deep learning algorithms, i.e., (GRU and LSTM) due to its hybrid nature. Both of the parallel algorithms belong to the same Recurrent Neural Network (RNN) family of deep learning models, therefore they have same complexity which is represented in Eq. (1). (1)O=(W)

In Eq. (1), W is the number of the weights. RNN model is local in space and time as a result, the input size has no effect on network storage space. Thetime complexity per weight for each time step is O(1).

The construction of our proposed hybrid deep learning model includes initialization of two parallel deep learning models, i.e., GRU and LSTM following the use of a merge layer for continuation. Consequently, ending the neural network with a simple dense layer as an output layer using the “softmax” activation function. The details of the proposed and other contemporary models are presented in Table 4. The software libraries used for experimentation and evaluation include TensorFlow and sklearn, respectively. An efficient, user-friendly high-level python library famously known as Keras is also utilized. The detail of the software, along with versions and hardware specifications for our experimentation, is elaborated in Table 5.

Deep learning is considered as optimum if it entails a high detection rate of accuracy to correctly identify anomalies with low false alarms. We evaluated the performance of our proposed and comparative models, Hybrid GRU-LSTM, Hybrid CNN2D-CNN3D, and Hybrid GRU-DNN, using standard metrics (accuracy, precision, recall, F1-score) and extended metrics including FNR, FPR, FDR, FOR, NPV, TNR, ROC curve, and MCC.

The deep learning framework is designed to classify three classes: benign, Mirai, and Gafgyt attacks. Experiments were conducted using the proposed hybrid GRU–LSTM model and comparative algorithms (GRU–DNN and CNN2D–CNN3D). We employed 10-fold cross-validation to ensure unbiased evaluation, with results for standard metrics (precision, recall, F1-score, and accuracy) reported in Table 6. Confusion matrices for all models are presented in Fig. 2, demonstrating that the proposed GRU–LSTM effectively distinguishes between benign and attack classes.

To evaluate the performance of the proposed algorithms, accuracy, precision, recall, and F1-score were computed. As shown in Fig. 3, the hybrid GRU–LSTM achieved the highest performance with 99.96% accuracy, 99.77% precision, 99.64% recall, and 99.64% F1-score, outperforming GRU–DNN (99.94%, 99.65%, 99.49%, 99.49%) and CNN2D–CNN3D (99.91%, 99.62%, 99.53%, 99.53%). ROC analysis is presented in Fig. 4. AUC values appear as 1.0 due to rounding, while the actual scores are 0.998163 (GRU–LSTM), 0.997339 (GRU–DNN), and 0.997881 (CNN2D–CNN3D), indicating robust performance without overfitting and effective separation of benign and attack classes. Additionally, True Negative Rate (TNR), Matthews Correlation Coefficient (MCC), and Negative Predictive Value (NPV) were evaluated to assess anomaly detection capability (Fig. 5). The proposed GRU–LSTM achieved 99.98% TNR, 99.93% MCC, and 99.97% NPV, surpassing GRU–DNN (99.97%, 99.90%, 99.96%) and CNN2D–CNN3D (99.97%, 99.90%, 99.68%), demonstrating superior detection of malicious samples and overall classification reliability.

An effective model achieves low False Positive Rate (FPR), False Negative Rate (FNR), False Discovery Rate (FDR), and False Omission Rate (FOR). FPR indicates the proportion of benign samples misclassified as attacks, FNR the fraction of attacks misclassified as benign, FDR the proportion of false positives among predicted positives, and FOR the proportion of false negatives among predicted negatives. As shown in Fig. 5, the proposed GRU–LSTM achieved 0.0002% FPR, 0.0022% FNR, 0.0035% FDR, and 0.0002% FOR, outperforming GRU–DNN (0.0002%, 0.0041%, 0.0034%, 0.0003%) and CNN2D–CNN3D (0.0003%, 0.0045%, 0.0031%, 0.0003%), demonstrating high detection efficiency.

Global performance measures, including Bookmaker Informedness (BM), Markedness (MK), and Threat Score (TS), further validate model performance. As depicted in Fig. 5, GRU–LSTM achieved 99.63% BM, 99.74% MK, and 99.42% TS, outperforming GRU–DNN (99.46%, 99.61%, 99.14%) and CNN2D–CNN3D (99.50%, 99.59%, 99.16%), confirming its superior discriminative power and overall efficiency.

The testing time of the proposed algorithms is shown in Fig. 5. Single-sample inference required 0.0021 ms for GRU–LSTM, 0.0026 ms for GRU–DNN, and 0.0012 ms for CNN2D–CNN3D, measured using optimized TensorFlow GPU execution on a NVIDIA GTX 1050 Ti. These times correspond to single-sample inference and may vary for larger batches or full-pipeline evaluation. Table 7 provides a comparison of our work with the current state-of-the-art in terms of detection accuracy and time efficiency. This table clearly states the difference between existing work and our work by providing significant details that are adequate for comparison.