CNNs are employed as the spatial feature extraction component of the proposed framework. CNN layers capture local correlations among network traffic attributes, such as protocol interactions, flow statistics, and packet-level patterns. Given an input feature vector x, a convolutional layer computes a feature map h_k as follows [21]:(1)hk=σ(Wk∗x+bk)where ∗ denotes convolution, W_k and b_k represent the weight and bias of the k-th convolutional filter, respectively, and σ(⋅) is a nonlinear activation function. Stacking multiple convolutional layers enables the model to learn increasingly abstract and discriminative representations of complex attack behaviors.

LSTM networks are adopted to model temporal dependencies in network traffic sequences. In intrusion detection, such temporal modeling is crucial for identifying time-dependent and multi-stage attacks.

An LSTM cell updates its internal memory state using gated mechanisms, expressed as in (2) [22]:(2)ft=σ(Wfxt+Ufht−1+bf),it=σ(Wixt+Uiht−1+bi),ot=σ(Woxt+Uoht−1+bo),where f_t, i_t, o_t denote the forget, input, and output gates at time t, respectively. W_f, W_i, W_o denote the input weight matrices associated with the forget, input, and output gates, respectively, U_f, U_i, U_o are the corresponding recurrent weight matrices, and b_f, b_i, b_o are bias terms. This architecture allows effective learning of long-term dependencies while mitigating vanishing gradient issues.

PSO is a population-based metaheuristic inspired by collective swarm intelligence [23]. It searches for optimal solutions by iteratively updating a set of particles according to both individual experience and global knowledge. At each iteration t, the velocity v_i and position x_i, of a particle i, are updated according to (3):(3){vit+1=ωvit+c1r1(pibest−xit)+c2r2(gbest−xit)xit+1=xit+vit+1where ω is the inertia weight, c₁ and c₂ are acceleration coefficients, and r₁, r₂ ∈ [0, 1] are random variables, p_i^best is the personal best, and g^best is the global best position.

Based on the optimization principles of PSO, four deep learning-based intrusion detection algorithms are introduced in the following subsections, in which PSO is progressively integrated, ranging from static offline optimization schemes to a fully co-evolutionary learning framework.

The DL-Base model is a hybrid CNN–LSTM, trained conventionally. CNN layers capture spatial correlations among traffic features. LSTM layers model temporal dependencies across flows. Training uses back-propagation on a fixed architecture. This baseline serves as the reference for evaluating the impact of PSO-based enhancements. Algorithm 1 describes the steps involved in the DL-B model.

In the first proposed Algorithm, PSO is applied offline to select a subset of input features. Each particle encodes a binary feature mask. Fitness is measured via classification performance on a lightweight evaluator. The CNN–LSTM is then trained on the reduced feature set, with no further PSO interaction, reflecting standard preprocessing-based PSO usage in IDS literature.

Algorithm 2 details the DL-PSO-FS steps.

In this Algorithm, PSO optimizes hyper-parameters (learning rate, number of CNN filters, LSTM units) offline. Selected hyper-parameters are fixed during training, improves convergence and stability but does not interact dynamically with latent representations. The steps of the DL-PSO-HP method are summarized in Algorithm 3.

In DL-PSO-S Algorithm, PSO is used to preselect the structure of ADBs. Each particle represents a candidate decision block configuration. The structure is fixed after optimization, providing greater expressiveness than DL-Base but lacking dynamic adaptation to evolving attacks. The different phases of the DL-PSO-S are shown in Algorithm 4.

The proposed DL–PSO–CE framework establishes a bidirectional co-evolutionary interaction between deep representation learning and swarm-based structural optimization. A CNN–LSTM encoder continuously learns latent representations of network traffic, while Particle Swarm Optimization (PSO) dynamically adapts the model structure by controlling the activation of Adaptive Decision Blocks (ADBs).

Each ADB consists of a lightweight fully connected module followed by a nonlinear activation function. Multiple ADBs operate in parallel, forming modular decision pathways whose outputs are aggregated before the final classification layer. Their activation is governed by a binary vector encoded by PSO particles, enabling dynamic and compact adaptation of the decision structure during training.

Unlike conventional hybrid DL–PSO approaches with offline optimization, PSO is integrated directly into the training loop. Latent representations produced by the CNN–LSTM are used to estimate latent-space uncertainty, which is incorporated into the PSO fitness evaluation. PSO then updates ADB activation patterns, modifying the model structure before subsequent gradient-based updates.

Each particle encodes a binary activation vector, where each dimension corresponds to one ADB. This iterative interaction forms a closed feedback loop between representation learning and structural adaptation, enabling progressive refinement of decision boundaries and improved detection of rare or unseen attacks.

To guide the structural optimization process, an uncertainty-aware fitness signal is incorporated into the PSO evaluation. The uncertainty component is derived from the predictive entropy of the class posterior probabilities produced by the CNN–LSTM encoder. This entropy value quantifies classification ambiguity and is combined with performance metrics to guide the selection of ADB configurations that simultaneously improve classification accuracy and reduce uncertainty.

Algorithm 5 summarizes the overall co-evolutionary training procedure.

To ensure methodological consistency across PSO-based variants, a unified fitness formulation is adopted:(4)f=a1Acc+a2F1−a3Uwhere Acc, F₁ and U denote Accuracy, F₁-score and latent-space uncertainty, respectively, and a₁, a₂ and a₃ are weighting coefficients. The performance components of the fitness function (Acc and F₁-score) are defined in Section 4.3. The latent-space uncertainty U is estimated using the mean predictive entropy of the class posterior distributions derived from the CNN–LSTM latent representations:(5)U=1N∑i=1N(−∑c=1Cpi,clog⁡pi,c)where p_i,c denotes the predicted probability of class c for sample i, C is the number of classes, and N is the number of samples.

Predictive entropy is employed as an uncertainty proxy due to its computational efficiency and stability during training. Entropy-based uncertainty estimation has been widely adopted in deep learning models to identify ambiguous or poorly classified samples. Although alternative estimators such as Monte Carlo dropout or ensemble-based uncertainty could be considered, they introduce additional computational overhead that may limit their integration within iterative optimization loops.

To comprehensively evaluate the proposed intrusion detection framework under diverse and realistic network conditions, experiments were conducted on three publicly available and widely adopted benchmark datasets: UNSW-NB15, CICIDS2017, and ToN-IoT. These datasets differ in traffic characteristics, attack diversity, and temporal complexity, enabling a robust assessment of model generalization.

UNSW-NB15: 2.5M instances, 49 features, nine modern attack categories [24].

CICIDS2017: 2.8M instances, 80 flow-based features covering 15 attacks classes [25].

ToN-IoT: IoT and industrial telemetry, including up to 10 classes, described by about 43 attributes [26].

All datasets were subjected to a unified preprocessing pipeline, including the removal of missing and infinite values, one-hot encoding of categorical features, elimination of duplicate and incomplete flows, and min–max normalization of continuous attributes.

To ensure label consistency, related attack subtypes were grouped into unified classes, resulting in 7 classes for each Dataset. Class imbalance, particularly severe in CICIDS-2017 and ToN-IoT, was mitigated using SMOTE [27]. Since uncertainty signals are derived from model predictions rather than directly from input samples, the influence of SMOTE-generated synthetic instances on uncertainty estimation remains limited. Consequently, the use of SMOTE does not introduce significant bias into the co-evolutionary optimization process.

Approximately 230k samples were retained from each Dataset, then split into training (70%), validation (15%), and test (15%) sets using stratified sampling. The normalized class distributions of the resulting datasets are reported in Table 1.

To address the high dimensionality of the datasets, statistical correlation analysis and SHAP-based feature attribution were jointly employed. Highly correlated features (r > 0.9) were removed, and post-training SHAP analysis [28] was used to rank feature importance, retaining the top 24 features to balance detection performance and interpretability.

All experiments were conducted on an NVIDIA RTX A6000 GPU (48 GB VRAM) using PyTorch 2.2. Hyper-parameters were determined via ten-fold cross-validation.

All experiments are conducted under multi-class intrusion detection settings, consistent with the class distributions of the CICIDS2017, UNSW-NB15, and ToN-IoT datasets, provided in Table 1.

The proposed IDS employs a hybrid CNN–LSTM architecture composed of two one-dimensional convolutional layers with 64 and 128 filters (kernel size = 3), followed by max-pooling (pool size = 2) and ReLU activation. Temporal dependencies are modeled using two stacked LSTM layers with 128 hidden units and a dropout rate of 0.3. The classification layer is fully connected with Softmax activation.

Model training is performed using the Adam optimizer with a learning rate of 0.001, a batch size of 128, and categorical cross-entropy loss.

For PSO-based variants, a swarm size of 40 particles is used over 50 iterations, with inertia weight ω = 0.7 and cognitive and social coefficients set to c₁ = c₂ = 1.5.

The fitness weights in Eq. (4) were fixed to a₁ = a₂ = 0.5 for all variants. The latent-space uncertainty term was deactivated (a₃ = 0) for all static PSO-based algorithms and enabled only in DL-PSO-CE (a₁ = 0.5, a₂ = 0.4, a₃ = 0.1).

To evaluate the proposed methods, precision rate Acc_i, Recall R_i and F_1i-score are computed, for each class C_i, as well as global rates such as accuracy Acc, recall R and F₁-score. (6)Acci=Cii∑jCji,Ri=Cii∑jCij,F1,i=2AcciRiAcci+Ri (7)Acc=∑iCii∑i,jCij,R=1K∑i=1KRi,F1=1K∑i=1KF1,iwhere C_ij denotes the number of samples belonging to class C_i predicted as class C_j and K the number of classes. In addition, AUC (Area Under roc Curves), Inference latency (ms/packet), and training time per epoch are considered. Paired t-test and Wilcoxon signed-rank test [29] are used to assess significance of observed improvements. Note that the paired t-statistic and the Wilcoxon Z statistic are computed as follows:(8)t=d¯sd/n,Z=W−μWπW,where d¯, s_d, and n are the mean, standard deviation, and number of paired differences, respectively; W, μ_W and π_W are the sum of signed ranks, its expected mean, and standard deviation under the null hypothesis.

The five algorithmic variants were evaluated on UNSW-NB15, CICIDS2017, and ToN-IoT using Accuracy (Acc), F₁-score, and AUC. The best test performances are summarized in Table 2.

Static PSO-based variants provide consistent but limited gains over DL-B (1%–3%), whereas DL-PSO-CE achieves substantially larger improvements (4%–5% over DL-B and 1%–3% over the best static PSO variant), while reaching near-perfect AUC values (≥0.995) across all datasets. The strongest gains are observed on ToN-IoT, highlighting the robustness and generalization capability of the proposed co-evolutionary mechanism under heterogeneous and imbalanced traffic conditions.

In addition, the per-class performance shown in Fig. 1 confirms that DL-PSO-CE achieves consistently higher per-class precisions, recall and F₁, with gains of up to 1%–5% on attack classes compared to the baseline and static PSO-based variants. The proposed DL-PSO-CE consistently achieves the highest discrimination performance, with AUC values above 0.995 on UNSW-NB15, CICIDS2017, and ToN-IoT. While static PSO-based variants improve upon the baseline, their ROC curves degrade in low false positive rate regions. In contrast, DL-PSO-CE maintains high true positive rates, particularly on ToN-IoT, demonstrating superior robustness in heterogeneous and imbalanced traffic scenarios.

Fig. 2 presents the ROC curves for all datasets. The proposed DL-PSO-CE consistently achieves the highest discrimination performance, with AUC values above 0.995 on UNSW-NB15, CICIDS2017, and ToN-IoT. While static PSO-based variants improve upon the baseline, their ROC curves degrade in low false positive rate regions. In contrast, DL-PSO-CE maintains high true positive rates, particularly on ToN-IoT, demonstrating superior robustness in heterogeneous and imbalanced traffic scenarios.

The high detection performance may raise concerns regarding overfitting or data leakage. To mitigate this risk, strict preprocessing was applied, including removal of duplicate flows and stratified splitting into non-overlapping training, validation, and test sets. Furthermore, evaluation on three heterogeneous datasets (UNSW-NB15, CICIDS2017, and ToN-IoT) supports the generalization capability of the proposed framework.

Statistical significance was assessed using both paired t-tests and Wilcoxon signed-rank tests on UNSW-NB15, CICIDS2017, and ToN-IoT datasets.

To control for multiple comparisons, the Holm–Bonferroni correction was applied to all statistical tests. All comparisons remain statistically significant after adjustment, confirming that the observed performance improvements are robust and not due to multiple testing effects.

As summarized in Tables 3 and 4, the proposed DL-PSO-CE framework consistently outperforms all baseline and static PSO-based variants. The paired t-test results show t-values ranging from 3.87 to 9.91, with all corresponding p-values below 0.01, indicating statistically significant improvements in accuracy across all datasets. These findings are further confirmed by the Wilcoxon signed-rank test, where Z-scores range from −2.04 to −3.92, with all comparisons remaining significant (p < 0.05).

Overall, the statistical analysis demonstrates that the performance gains achieved by DL-PSO-CE are consistent, robust across datasets, and not attributable to random variation, thereby supporting the effectiveness and generalizability of the proposed co-evolutionary framework.

Table 5 summarizes the computational performance of the evaluated methods. As expected, PSO-assisted variants require longer training time due to the additional optimization steps. Among them, the proposed DL-PSO-CE framework exhibits the highest training cost, as it involves iterative PSO updates and dynamic structural adaptation through ADB activation. Nevertheless, the additional overhead remains moderate.

Importantly, inference latency remains below 1 ms per sample across all configurations, indicating that the proposed method remains suitable for near real-time deployment. Further analysis shows that the computational overhead associated with PSO iterations and ADB switching is relatively small compared with the overall training cost, while GPU acceleration significantly reduces training time.

The additional computational overhead introduced by PSO iterations and ADB structural adaptation remains moderate compared to the overall training cost. It should be noted that the reported training time includes both gradient-based model training and PSO optimization steps.

Although the co-evolutionary framework increases training time due to PSO-driven structural optimization, the additional overhead remains moderate relative to the overall training process. Considering the significant improvement in detection performance and the low inference latency (<1 ms per sample), the proposed framework provides a favorable trade-off between computational cost and detection accuracy.

Sensitivity was evaluated for the four algorithms across three PSO parameters (swarm size, iteration count, inertia weight), each assessed on a different dataset (Fig. 3).

The accuracy-based sensitivity curves in Fig. 3 show that the proposed DL-PSO-CE framework consistently outperforms all baseline and static hybrid variants. While DL-B and offline PSO-based approaches exhibit strong sensitivity to inertia weight, population size, and iteration count, the co-evolutionary model maintains stable and high accuracy across a wide parameter range.

This robustness is attributed to the continuous interaction between deep latent representations and PSO-driven structural adaptation, which enables dynamic self-correction during training. These results confirm that performance gains arise from co-evolution rather than from isolated or static optimization. The results further validate the selected PSO parameter (swarm size = 40, 50 iterations, inertia weight ω = 0.7).

To evaluate robustness under distribution shift, a cross-dataset experiment was conducted using a binary setting (Normal vs. DoS/DDoS) consistent with Table 1. Models were trained on one dataset and directly tested on another without fine-tuning, ensuring a strict generalization assessment.

Results in Table 6 show that DL-PSO-CE maintains strong cross-dataset generalization (96.11%–97.12% Accuracy), with limited performance degradation compared to within-dataset results (2.86%–3.87%) confirming its robustness under distribution shifts.

Table 7 compares the proposed framework with several recent intrusion detection methods evaluated on the CICIDS2017, UNSW-NB15, and ToN-IoT datasets.

On CICIDS2017, the proposed DL-PSO-CE framework achieves 99.98% accuracy and 99.97% F₁-score, outperforming several deep and hybrid IDS models, including LSTM-based approaches [15], DNN-LSTM architectures [16], and GA-optimized CNN models [17]. It also improves upon more recent architectures such as NAS-Net [14], which report F₁-scores below 99.6%.

For UNSW-NB15, existing approaches including CNN-LSTM [1], GWO-CNN [3], Cyber-MLP [12] and PSO-1D-CNN [17], achieve accuracies between 92.90% and 99.28%. In comparison, DL-PSO-CE reaches 99.98% accuracy and 99.99% F₁-score, indicating improved detection performance in heterogeneous network environments.

On ToN-IoT, machine learning and deep learning methods such as LSTM [18], XGBoost [19], and ANN [20] report accuracies between 98.30% and 99.44%. The proposed framework consistently achieves 99.97% accuracy and 99.99% F₁-score, demonstrating strong robustness for large-scale IoT traffic analysis.

Overall, the results suggest that integrating deep representation learning with PSO-based structural co-evolution provides improved and consistent detection performance compared with conventional deep learning models and static optimization strategies.

The proposed DL-PSO-CE framework achieves state-of-the-art detection performance across the UNSW-NB15, CICIDS2017, and ToN-IoT datasets, while maintaining low inference latency (below 1 ms per sample). Compared with baseline deep learning models, static PSO-based variants, and conventional hybrid approaches, the proposed method consistently demonstrates improved detection capability and robustness. Sensitivity analyses further confirm the stability of the framework with respect to variations in PSO parameters, while paired t-tests and Wilcoxon signed-rank tests indicate that the observed performance improvements are statistically significant (p < 0.01) across datasets and evaluation metrics.

Unlike static feature selection or hyperparameter optimization strategies, the proposed co-evolutionary mechanism dynamically refines the internal decision structure of the model during training. This adaptive behavior allows the IDS to better capture complex and rare attack patterns by continuously adjusting the activation of Adaptive Decision Blocks based on latent representation signals.

Despite these advantages, several limitations should be acknowledged. First, the co-evolutionary mechanism introduces a moderate training overhead due to PSO-driven structural optimization. Second, the effectiveness of the approach depends on the quality of the latent representations learned by the deep encoder. In addition, the current fitness formulation relies on a weighted-sum aggregation of objectives. Future work could explore alternative multi-objective optimization strategies, such as Pareto-based PSO, to explicitly balance competing objectives during structural adaptation.

Finally, practical deployment in real-world network infrastructures may require addressing additional challenges such as streaming traffic processing, concept drift, and memory constraints in large-scale or resource-constrained environments.