The landscape of network intrusion detection continues to be substantively shaped by advancements in both classical and contemporary computational intelligence paradigms. Research has focused on optimizing classical models and addressing specific data challenges. For parameter refinement, Kolukisa et al. [9] synergized logistic regression with a parallelized artificial bee colony (ABC) algorithm, achieving commendable detection performance. To confront high-dimensional feature spaces, Turukmane et al. [10] integrated multi-layer Support Vector Machines (SVMs) with singular value decomposition. For the complexities of mixed-traffic environments, Rustam et al. [11] introduced the Fully Automated Malicious Traffic Detection System (FAMTDS) framework, using a Moth Flame Optimizer (MFO) for automation. Addressing the critical problem of data imbalance, Hooshmand et al. [12] developed a multi-stage model (K-means (SKM) hybrid, Denoising Autoencoder (DAE), and XGBoost classifier), achieving binary/multiclass accuracies up to 99.57% on NSL-KDD. Finally, for scalable stream processing, He et al. [13] proposed a Bayesian gamma mixture model (GaMM) employing extended stochastic variational inference (ESVI).

The utility of deep learning in modern intrusion detection stems from its capacity for automated hierarchical feature extraction. Research has explored several integration strategies. For instance, novel architectures are being developed to work with limited labeled data; Xu et al. [14] proposed the Time-Space Separable Attention Network (TSSAN), which utilizes depth-wise separable convolution and a time-space self-attention mechanism to extract features in an unsupervised or semi-supervised context. Architectural synergy has also been exploited to model both spatial and temporal dependencies; Liu et al. [15] proposed the Residual Memory Convolutional Neural Network (RMCNN), a hybrid model that first extracts spatial features with CNN layers, then captures temporal dynamics using a Gated Recurrent Unit (GRU), and reinforces critical features with a multi-head attention mechanism. To address the challenge of novel, unseen attacks in few-shot scenarios, meta-learning frameworks have been introduced. Xu et al. [16] proposed Marrying Attention and Convolution-based Meta-learning (MACML), an optimization-based meta-learning method that “marries” a self-attention mechanism (for global dependencies) with a CNN (for local features), enabling rapid adaptation to new attack types with minimal samples. Hybrid dimensionality reduction has also been proposed for LSTM-based models, as seen in the two-stage fusion strategy by Thakkar et al. [17]. Their method, fusing a non-linear Autoencoder (AE) with linear Principal Component Analysis (PCA), yielded a performance gain of up to 3% over a standard AE+LSTM baseline. Similarly, Alsoufi et al. [18] addressed high-dimensional IoT data by employing a Sparse Autoencoder (SAE) for feature dimensionality reduction coupled with a CNN for classification, demonstrating significant accuracy improvements in IoT environments

A prominent research strategy is to couple DL feature engineering with classical classifiers or other learning paradigms. Chen et al. [19], for example, introduced Metric Learning Framework with Dual One-Class Units (MLF-DOU), which uses one-class classifiers and a triplet network-based metric learning stage to maximize inter-class separability, demonstrating superior accuracy and F1-score on UNSW-NB15. Hierarchical designs represent another approach to tackle specific challenges, such as identifying rare attack classes (a form of data imbalance). Wang et al. [20] devised a two-layer model where a CNN-BiLSTM layer (addressing temporal dependencies) performs coarse-grained classification, followed by a Stacking ensemble learner for fine-grained analysis of minority classes, which demonstrated superior recall. Reinforcement learning offers a third path; Hossain [21] ventured into this with their Deep Q-Learning Intrusion Detection System (DQ-IDS). This Deep Q-Network (DQN) autonomously learns classification policies using experience replay and adaptive exploration, achieving a notable 97.18% accuracy and 98.52% F1-score on the real-world CICIoT2023 dataset. Finally, to handle the dynamic nature of threats and avoid “catastrophic forgetting,” Wang et al. [22] introduced CTWA, an incremental learning method that uses a Convolutional Autoencoder (CAE) and Temporal Convolutional Network (TCN) to extract features. This framework applies Weight Alignment (WA) techniques, enabling the model to learn new attack classes without losing knowledge of old ones.

Persistent data imbalance remains a critical challenge. One solution is generative data enrichment; Arafah et al. [23] confronted this with the Autoencoder-Wasserstein Generative Adversarial Network (AE-WGAN), where a Wasserstein Generative Adversarial Network synthesizes high-fidelity minority attack samples, yielding substantial performance gains. An alternative strategy integrates imbalance correction directly into the classification pipeline. The Network Intrusion Detection System-Convolutional Neural Network Random Forest (NIDS-CNNRF) model, proposed by Yang et al. [24], exemplifies this by coupling a standard CNN extractor with a Random Forest (RF) classifier, but first applying PCA and the Adaptive Synthetic Sampling (ADASYN) algorithm to resample the data, which was validated across several benchmarks.

The literature reveals a clear trajectory towards hybrid models, but a persistent limitation is this reliance on standard CNNs for feature extraction. This highlights a key research gap: the need for models that integrate feature extractors capable of capturing more complex, multi-scale hierarchies. The proposed TransNeSt architecture is designed to fill this gap. To our knowledge, no existing NIDS research has adapted split-attention mechanisms (ResNeSt), originally designed for 2D computer vision, for 1D network traffic analysis. TransNeSt directly addresses the multi-scale feature challenge by integrating a state-of-the-art ResNeSt block and couples it with a Transformer encoder to effectively model long-term temporal dependencies, creating a more profound synergy between feature representation and temporal analysis.

This review of the literature, for which a detailed summary is presented in Table 1, highlights three persistent challenges that motivate our work. First, data imbalance is a critical and prevalent challenge, as rare but dangerous attack classes like Remote-to-Local (R2L) and User-to-Root (U2R) are vastly underrepresented, causing models to develop a bias toward majority (benign) traffic. Second, effectively modeling complex temporal dependencies remains an open problem; while LSTMs were an improvement over CNNs, they suffer from computational overhead, and attacks are often sophisticated, multi-stage sequences that are difficult to model. Third, discerning subtle attacks requires capturing a multi-scale feature hierarchy, but the standard CNNs used in many hybrid models fail to capture the granular, hierarchical feature relationships embedded within network traffic.

The overall architecture of TransNeSt follows a two-stage sequential workflow. First, a ResNeSt Block functions as an advanced feature extractor, processing the input sequence to capture fine-grained, feature-based spatial patterns. Subsequently, the sequence of enriched feature representations generated by this block is passed to a Transformer Block, which models the long-range temporal dependencies across the entire sequence.

Formally, the input to our model is a sequence of pre-processed network traffic data, denoted as X={x1,x2,…,xN}, where each xi∈Rd is a d-dimensional vector representing a single network flow or packet. This sequence is first processed by the ResNeSt Block. The principal role of this block is to transform the initial input into a sequence of enriched feature vectors, Z={z1,z2,…,zN}, where each vector zi contains a high-level representation of salient local characteristics. This sequence Z then serves as the input to the Transformer Block. The Transformer employs its self-attention mechanism to conduct a global analysis, modeling the temporal context and intricate inter-flow relationships across the entire sequence. The final output from the Transformer is then processed by a conventional classification head, which consists of a Linear layer and a Softmax activation function, in order to generate the final probability distribution across the designated classes of network activity (e.g., Normal, DoS, Probe).

The ResNeSt block, originally designed for 2D image analysis, is a core component of our model. A key innovation of our work is the adaptation of its architecture to effectively process 1D sequential network traffic data by replacing 2D convolutions with their 1D counterparts (e.g., 1×1 and 3×1). Our selection of the ResNeSt block is predicated on its empirically demonstrated advantages over incumbent architectures like ResNet and Squeeze-and-Excite Network (SE-Net). The seminal ResNeSt paper [8] provides extensive empirical validation, showing that ResNeSt models consistently outperform their ResNet and SE-Net counterparts across multiple benchmarks.

The process within a ResNeSt block begins with an input feature map of dimensions (c,l), which is first divided into k cardinal groups. Each of these groups is then further subdivided into r radix splits, yielding a total of k×r parallel processing paths. These hyperparameters k (cardinality) and r (radix) directly control the model’s complexity and capacity; increasing them allows for more complex representations at the cost of higher computational load. Let the feature map for a single split be Ui∈R(c/k)×l. This split-level feature map is then passed through a sequence of 1D convolutions, comprising a 1×1 convolution that precedes a 3×1 convolution, to produce a transformed feature map Ui′=Conv(Ui). The core innovation lies in the Split Attention Mechanism, which intelligently fuses the information from these transformed splits {U1′,U2′,…,Ur′}.

The Split Attention mechanism, explicitly diagrammed in Fig. 3, directs the flow of information and adaptively recalibrates feature responses within each cardinal group. The mechanism unfolds as follows:

First, the transformed feature maps from the r radix splits are aggregated via element-wise summation, producing a unified representation for the cardinal group: (1)U^=∑i=1rUi′

To generate a channel-wise summary of contextual information, global average pooling (ℱgp) is applied to U^, squeezing its spatial dimension to produce a channel descriptor vector s∈Rc/k. The j-th component of this vector is given by: (2)sj=ℱgp(U^)j=1l∑p=1lU^j(p)

This vector s is then passed through a small sub-network to generate split-specific attention weights.It is first passed through a fully-connected layer with weights W1∈Rc′×(c/k) and biases b1, then Batch Normalization (ℬ) and a Rectified Linear Unit (ReLU) activation (δ), producing the condensed feature vector z: (3)z=δ(ℬ(W1s+b1))where c′ is a bottleneck dimension, typically smaller than c/k. Subsequently, a second fully-connected layer with weights W2∈R(c/k)⋅r×c′ and biases b2 takes z as input and generates attention scores for all splits. This output is reshaped to produce r distinct attention score vectors, {s1,s2,…,sr}, where each si∈Rc/k.

These scores are normalized across the radix dimension for each channel using an r-Softmax function. The attention weight ai(j) for the j-th channel of the i-th split is given by:

(4)ai(j)=exp⁡(si(j))∑m=1rexp⁡(sm(j))

This ensures that for any channel j, the attention weights across all r splits sum to one, i.e., ∑i=1rai(j)=1. The Split Attention block’s final output, Vc, for the cardinal group is a weighted fusion of the split feature maps, where each map Ui′ is recalibrated by its corresponding attention vector ai: (5)Vc=∑i=1rai⊙Ui′where ⊙ denotes channel-wise multiplication, with the vector ai being broadcast along the spatial dimension. The concatenated outputs from all k cardinal groups are then passed through a final 1×1 convolution, and a residual connection is added to the original block input to ensure stable and efficient training [25].

Subsequent to the extraction of localized feature hierarchies by the ResNeSt block, the modeling of temporal dynamics across the sequence of network events is entrusted to a bespoke Transformer Block [5]. A pivotal architectural decision within this block is the normalization strategy. In a departure from the canonical Transformer, we substitute the conventional Layer Normalization with Root Mean Square Normalization (RMSNorm) [26]. This modification is deliberately implemented to enhance computational throughput and training efficiency. As demonstrated in [26], RMSNorm achieves comparable performance to LayerNorm but with significantly reduced computational overhead, observing speedups from 7% to 64% across different models. This efficiency gain comes from simplifying the operation: RMSNorm preserves the essential re-scaling invariance of LayerNorm but eschews the mean-centering operation, which contributes less to training stability while incurring computational costs. RMSNorm thus provides a more efficient drop-in replacement. For an input feature vector x∈Rd, the operation is formally defined as: (6)RMSNorm(x)=g⋅x1d∑i=1dxi2+ϵwhere g denotes a learnable gain parameter and ϵ is a small constant for numerical stability.

Each Transformer block comprises two main sub-layers: a Multi-Head Attention (MHA) mechanism and a position-wise Feed-Forward Network (FFN).

In the MHA sub-layer, the input tensor Z∈RN×d is linearly mapped to form Query (Q), Key (K), and Value (V) matrices. The attention is then calculated using scaled dot-product attention: (7)Attention(Q,K,V)=softmax(QKTdk)Vwhere dk is the dimensionality of the key vectors.

The output of the MHA sub-layer is subsequently processed by the FFN, which consists of two linear transformations with a ReLU activation: (8)FFN(x)=ReLU(xW1+b1)W2+b2

Each of these sub-layers is encapsulated with a residual connection and an RMSNorm layer. To build a deeper representation, this entire four-sub-layer block is stacked for two iterations.

The architectural design of the TransNeSt model culminates in a dedicated classification head, which performs the final, decisive conversion of the deeply learned feature vectors into a final probability distribution over the class labels. Adhering to the established paradigm for Transformer-based classification, the predictive inference is predicated on the terminal hidden state vector corresponding to a special classification token [CLS]. This vector, denoted as hCLS∈Rd, functions as a holistic, aggregated representation of the entire input sequence. It is subsequently projected onto the logit space via a fully-connected linear layer, which is parameterized by a weight matrix W∈RK×d and a bias vector b∈RK, yielding a vector of unnormalized scores (logits), z∈RK, where K signifies the number of target classes.

To transform these raw logits into a valid posterior probability distribution, the Softmax activation function is employed to yield the probability Pj for any given class j, defined as: (9)Pj=exp⁡(zj)∑k=1Kexp⁡(zk)

The model’s final prediction, y^, is obtained by selecting the class index with the highest posterior probability, a process performed using the argmax function: (10)y^=arg⁡maxj(Pj)

The model is trained to perform multi-class classification, as all datasets used in this study involve multiple classes of network activity (e.g., Normal and various attack types). To optimize the model’s parameters, we employ the Categorical Cross-Entropy (CCE) loss function. This loss function is standard for multi-class problems and measures the dissimilarity between the model’s predicted probability distribution P and the true one-hot encoded label vector y. For a single sample, the CCE loss ℒ is defined as: (11)ℒ(y,P)=−∑k=1Kyklog⁡(Pk)where K is the total number of classes, yk is the ground truth (1 if the sample belongs to class k, 0 otherwise), and Pk is the model’s predicted probability for class k from the Softmax output (Eq. (9)).

The feature extraction backbone, a stack of ResNeSt modules, was configured to create a robust and hierarchical feature representation. We selected four sequential blocks to ensure sufficient depth for learning complex feature representations from the input data. Within each block, we further optimized the split-attention mechanism to enhance representational power. Within each block, the split-attention mechanism was configured with a cardinality (k) of 2 and a radix (r) of 2. The feature processing pipeline commences with a 1D convolutional stem that projects the input into a 32-channel feature space. We maintain this channel width across all subsequent ResNeSt blocks to ensure consistent dimensionality for hierarchical feature learning.

In the temporal modeling stage, the Transformer Encoder was designed to effectively capture long-range dependencies. The encoder uses a 4-layer configuration, where each layer contains a Multi-Head Attention mechanism with 4 parallel heads and a Feed-Forward Network with an inner dimension of 256. To enhance regularization, a dropout rate of 0.3 was applied within the Transformer’s sub-layers. As previously justified, standard LayerNorm was replaced with the more parameter-efficient RMSNorm, configured with a numerical stability term ϵ=1×10−6, throughout the encoder.

To ensure consistent experimental outcomes and effectively manage the model’s complexity, we actively prevented overfitting through a combination of architectural, data, and training-level regularization. This included applying Dropout, controlling the model’s depth, employing the Adaptive Moment Estimation (Adam) optimizer with L2 Regularization (Weight Decay), and data augmentation. Furthermore, an Early Stopping was used to halt training at the point of peak generalization. The choice of key hyperparameters was determined through an empirical tuning process on the UNSW-NB15, with results summarized in Table 2. This configuration was found to provide the best balance between model capacity and generalization, avoiding the underfitting of simpler models (e.g., 2 layers) and the diminishing returns or overfitting of more complex ones (e.g., 0.1 dropout). To ensure consistent experimental outcomes and effectively manage this final model’s complexity, we incorporated several key training procedures.

The final architectural parameters derived from this selection process are summarized in Table 3.

To establish the empirical validity and generalization capabilities of the proposed model, our investigation was performed on a curated selection of four canonical benchmark datasets: NSL-KDD, UNSW-NB15, CIC-IDS2017, and the contemporary CICIoT2023. This collection ensures a rigorous evaluation across diverse network topologies, traffic profiles, and adversarial tactics. To ensure a consistent and unbiased evaluation, each dataset was partitioned into a training set (80%) and a testing set (20%) using a stratified sampling approach, thereby preserving the original class distribution in both subsets.

We first utilized the NSL-KDD dataset [27], a foundational benchmark for comparative analysis comprising a 41-feature vector. As detailed in Table 4, the dataset exhibits severe class imbalance, with critical minority attack classes like U2R (0.17%) and R2L (2.50%) being heavily underrepresented.

To assess performance against modern threats, we incorporate the UNSW-NB15 dataset [28], which provides 42 features per record. Its class distribution, shown in Table 5, presents a significant imbalance challenge, with minority classes such as Worms (0.08%) and Shellcode (0.69%) being exceptionally rare.

For a test of real-world applicability, we employed the CIC-IDS2017 dataset [29], noted for its high-dimensional (80+ features) space. As shown in Table 6, this dataset is heavily skewed, posing a challenge in detecting scarce minority attacks like Web and Bot, each representing only 0.07% of the data.

Finally, to evaluate scalability on large-scale IoT traffic, we incorporate the CICIoT2023 dataset [30]. This benchmark consists of 46 features per record. It presents a unique imbalance structure, as shown in Table 7, coexisting dominant attack classes with severe minority classes like BruteForce (0.23%) and Web (0.43%).

We implemented a multi-stage data preprocessing pipeline involving feature selection, categorical encoding, and feature scaling. We performed feature selection to remove non-informative attributes. For NSL-KDD, all 41 original features were used. For UNSW-NB15, the id and attack_cat features were removed, retaining 42 features. For CIC-IDS2017, we removed 6 metadata features (Flow ID, Source IP, Source Port, Destination IP, Protocol, Timestamp), resulting in a 78-feature set. For CICIoT2023, all 46 features were utilized. Subsequently, categorical attributes were transformed using one-hot encoding. This step expanded the NSL-KDD dataset from 41 to 122 dimensions and the UNSW-NB15 dataset from 42 to 196 dimensions. The remaining 78 features of CIC-IDS2017 and all 46 features of CICIoT2023 are numerical and did not require this encoding step.

Data quality was addressed by imputing any missing or non-finite feature values with zero. To mitigate the biasing effects of disparate feature scales, all numerical data were standardized using Min–Max normalization. This transformation was applied per-feature (i.e., column-wise), re-scaling each feature element xi to xi′ within the uniform interval [0,1] via the transformation: (12)xi′=xi−min(x)max(x)−min(x)where min(x) and max(x) represent the global minimum and maximum values for that specific feature across the dataset, respectively. This protocol is critical for stable model convergence.

We evaluate model performance based on the standard confusion matrix outcomes: True Positives (TP), True Negatives (TN), False Positives (FP, or Type I errors), and False Negatives (FN, or Type II errors), which are considered on a per-class basis for our multi-class task. While we report overall Accuracy, its utility is limited in imbalanced datasets where a high score may simply reflect strong performance on the dominant (benign) class. (13)Accuracy=TP+TNTP+TN+FP+FN

Therefore, our primary evaluation focuses on Precision and Recall. Precision measures the fidelity of alerts, quantifying the proportion of predicted attacks that are genuinely malicious; this is critical for minimizing the operational cost of investigating false alarms. (14)Precision=TPTP+FP

Recall (or Sensitivity) measures detection completeness, indicating the proportion of actual attacks the model successfully identifies; this is essential for ensuring threats are not missed. (15)Recall=TPTP+FN

To account for the inherent trade-off between Precision and Recall, we employ the F1-Score. As the harmonic mean of Precision and Recall, the F1-Score provides a single, balanced measure of performance. This metric is particularly well-suited for imbalanced classification tasks (as opposed to Accuracy) because it does not depend on True Negatives, thus focusing evaluation on the positive (minority) classes. (16)F1-Score=2×Precision×RecallPrecision+Recall

To visualize model performance independent of a specific classification threshold, we employ the Receiver Operating Characteristic (ROC) curve. The ROC curve is a graphical plot illustrating the trade-off between two key metrics—the True Positive Rate (TPR) and the False Positive Rate (FPR)—calculated across a range of decision thresholds. The TPR, which is mathematically equivalent to Recall, measures the proportion of actual attacks (positives) that are correctly identified. (17)TPR (Recall)=TPTP+FN

Conversely, the FPR measures the proportion of benign instances (negatives) that are incorrectly classified as attacks. (18)FPR=FPFP+TN

A critical and prevalent challenge in network intrusion detection is the severe class imbalance inherent in realistic datasets, a characteristic clearly delineated in our datasets. On benchmark datasets, minority attack classes like R2L, U2R, and Worms are vastly underrepresented compared to normal traffic and common attacks. This imbalance can bias the model during training, leading to high accuracy on majority classes but dangerously poor recall on rare, yet often critical, threats [31].

To mitigate this issue, we moved beyond conventional oversampling techniques like Synthetic Minority Over-sampling TEchnique (SMOTE) [32] or ADASYN [33], which generate synthetic samples through linear interpolation and may not fully capture the complex, non-linear distribution of attack data. Instead, we employed a sophisticated data augmentation strategy based on a Generative Adversarial Network (GAN) [34], specifically a variant of the Wasserstein GAN (WGAN). This approach is inspired by recent advancements that have successfully used generative models to create high-fidelity synthetic attack data for NIDS [23]. The WGAN architecture was chosen for its training stability and its ability to overcome the mode collapse problem common in standard GANs, thereby ensuring the generation of diverse and realistic minority samples [35].

This process is visualized on a 2D plane, using the Web minority class from the CIC-IOT2023 dataset as a case study. As illustrated in Fig. 4, the visualization provides strong support for our methodological choice. The WGAN-generated samples (blue dots) demonstrate a distribution that closely mirrors the original ‘Real Samples’ (red dots), clustering in similar regions and respecting the underlying data manifold. The SMOTE-generated samples (green dots), while also providing augmentation, show a distribution that does not align as closely with the true data structure compared to the WGAN samples.

The WGAN framework replaces the Jensen-Shannon divergence with the Wasserstein-1 distance (W1), which provides a smoother and more meaningful loss metric, leading to more stable training and higher-quality synthetic samples. Our process for augmenting each minority class, illustrated in Fig. 5, begins by training a dedicated WGAN model. The training follows an adversarial process governed by two distinct loss functions based on the Kantorovich-Rubinstein duality, which expresses the Wasserstein distance as: (19)W(Pr,Pg)=sup‖f‖L≤1(Ex∼Pr[f(x)]−Ex~∼Pg[f(x~)])where Pr is the real data distribution (the original minority class samples), Pg is the generator’s distribution that models Pr, and the supremum is taken over all 1-Lipschitz functions f. In our implementation, the function f is approximated by a neural network with weights w, referred to as the critic. The 1-Lipschitz constraint is enforced through weight clipping, where the weights w are clamped to a small range (e.g., [−0.01,0.01]) after each gradient update.

Our process involved training a dedicated WGAN model for each minority attack class. The training follows an adversarial process governed by two distinct loss functions. The critic is trained to maximize the objective in Eq. (19), which corresponds to minimizing the following loss: (20)LCritic=Ex~∼Pg[fw(x~)]−Ex∼Pr[fw(x)]

Conversely, the generator is trained to produce samples that are indistinguishable from real data, which corresponds to minimizing its loss function: (21)LGenerator=−Ex~∼Pg[fw(x~)]

Training for each per-class WGAN was continued until the Wasserstein distance estimate converged. The trained generator was then used to synthesize the new samples to create a more balanced class distribution for training. This targeted augmentation ensures that our TransNeSt model is exposed to a richer and more varied representation of rare attack patterns, enhancing its ability to learn their distinguishing features.

To experimentally quantify the impact of our WGAN-based augmentation. We evaluated the performance of our TransNeSt model on the minority attack classes across all four datasets under these three augmentation strategies. The results of this comparison are detailed in Table 8.

The data clearly demonstrate the superiority of the WGAN (Ours) approach. Across the vast majority of minority classes, our WGAN-augmented model achieves the highest scores in Precision, Recall, and F1-Score. By addressing the data imbalance at its source, we aim to significantly improve the model’s recall on minority classes without compromising precision, ultimately fostering a more robust and reliable detection system.

We initiated our empirical evaluation on the foundational NSL-KDD benchmark. The comprehensive performance comparison, illustrated in Fig. 6, clearly establishes the superiority of the proposed TransNeSt architecture. Our model achieved the highest performance across all metrics, attaining an F1-Score of 99.04% and an accuracy of 99.05%.

The results also serve as a compelling ablation study, isolating the contributions of our model’s core components. The standalone ResNeSt model, acting as a powerful feature extraction baseline, achieved a remarkable F1-score of 97.30%, demonstrating the inherent effectiveness of the split-attention mechanism in capturing salient traffic features. The standalone Transformer, while also performing well, was surpassed by the ResNeSt, underscoring the importance of robust initial feature representation. Crucially, our integrated TransNeSt model outperformed both individual components, validating the synergistic benefit of combining ResNeSt’s fine-grained feature extraction with the Transformer’s sequential modeling capabilities.

A deeper analysis of per-class performance, detailed in the confusion matrix (Fig. 7), ROC curves (Fig. 8), and performance metrics (Table 10), highlights the model’s robust performance. TransNeSt achieves outstanding F1-scores on majority classes like ‘DoS’ (99.84%) and ‘Normal’ (99.11%). More critically, this strength extends to the challenging minority attack classes, a success largely attributable to our augmentation strategy. As the comparative analysis in Table 8 empirically demonstrates, our WGAN-based approach yields performance gains over both the baseline and traditional SMOTE. For the extremely rare U2R class, WGAN elevates the F1-Score to 0.6538, surpassing the baseline’s 0.4728 and SMOTE’s 0.5617, thereby validating its efficacy in generating high-fidelity synthetic data.

Performance on R2L (89.39% F1) and U2R (65.38% F1) remain comparatively lower than that of the high-support classes. This challenge is attributed to the extreme data scarcity and low feature diversity of these attack instances. It suggests that while WGAN provides a critical performance boost, fully modeling the feature boundaries of these subtle attacks remains the primary difficulty and a key focus for subsequent research.

To assess the model’s efficacy against more contemporary and diverse threats, we conducted evaluations on the UNSW-NB15 dataset. As shown in Fig. 9, TransNeSt once again significantly outperformed all baseline models, achieving a top-tier F1-Score of 91.92% and an accuracy of 91.90%. This F1-Score, while lower than the 99.04% achieved on NSL-KDD, reflects the distinctly higher complexity, greater class imbalance, and more subtle feature patterns inherent in this modern and more challenging dataset.

This dataset provides further evidence of our architecture’s successful design. The standalone ResNeSt again proved to be a potent feature extractor with an F1-score of 88.03%, reinforcing the value of its multi-path, channel-aware attention structure for discerning modern attack patterns. The subsequent integration with the Transformer block in TransNeSt yielded a substantial performance gain of nearly 4 percentage points in F1-score. This improvement highlights the critical role of the Transformer in modeling the temporal relationships between the rich feature sets generated by ResNeSt, a synergy that is essential for handling the complexity of the UNSW-NB15 dataset.

The detailed per-class metrics are presented in Table 11, with corresponding confusion matrix in Fig. 10 and the ROC curves in Fig. 11. The model demonstrates high F1-scores on prevalent classes like ‘Normal’ (97.67%) and ‘Generic’ (97.46%). For the challenging low-support classes, the WGAN-based augmentation provides a clear advantage, as detailed in Table 8. WGAN, for example, raised the F1-Score for ‘Backdoor’ to 0.6427 (from 0.2016 baseline) and ‘Shellcode’ to 0.6794 (from 0.5764 baseline). Despite this improvement, these minority classes (e.g., ‘Analysis’, ‘Backdoor’, ‘Shellcode’, ‘Worms’) still yield the lowest F1-scores (62%–68% range). This indicates that while WGAN provides a critical boost, the low instance count and feature overlap remain the primary challenge for model refinement on this dataset.

A rigorous evaluation was performed on the large-scale and highly realistic CIC-IDS2017 dataset. The results, illustrated in Fig. 12, demonstrate the model’s high performance. TransNeSt achieved an F1-Score of 99.18% and an accuracy of 99.19%, outperforming the other tested models. The ablation analysis on this dataset further solidifies our central thesis. The ResNeSt component, with an F1-score of 97.52%, proves itself to be a highly effective feature extractor even on high-volume, complex traffic. The Transformer component also performs well, but the fusion in TransNeSt again elevates the performance to a new level, surpassing the next-best model (ResNet-Transformer) by over 1.5 percentage points. This margin is significant at such high performance levels and underscores the superiority of our specific architectural combination.

Given the severe class imbalance of this dataset, a detailed per-class analysis is crucial (Table 12, Figs. 13, and 14). TransNeSt exhibits near-perfect detection on dominant classes like ‘Normal’ (99.59% F1) and ‘DoS’ (97.64% F1). Critically, it also shows strong performance on less frequent attack types, such as the 98.05% F1-score on ‘Web’. For the ‘Bot’ class (less than 0.07% of data), the WGAN augmentation strategy proved essential. As shown in Table 8, WGAN raised the ‘Bot’ F1-Score to 0.7769, outperforming both the baseline (0.5381) and SMOTE (0.7522).

The ‘Bot’ class performance, while superior to other augmentation methods, remains the primary area for refinement. The 115 False Negatives identified for this class in (Fig. 13) are a key indicator. These are primarily misclassifications against the high-volume ‘Normal’ class. This highlights the persistent difficulty of distinguishing extremely low-support malicious flows from benign traffic, a challenge that persists even with effective data augmentation.

To validate our model’s scalability and performance on a very recent, large-scale IoT benchmark, we performed a final evaluation on the CICIoT2023 dataset. The comprehensive performance comparison, illustrated in Fig. 15, confirms the superiority of our proposed architecture. TransNeSt achieved the highest F1-Score of 97.85% and an accuracy of 97.85%, again clearly outperforming the other models. This dataset’s ablation results further reinforce our design. The standalone ResNeSt (95.23% F1) and Transformer (95.44% F1) components performed well. However, their fusion in TransNeSt (97.85% F1) yielded a performance gain of over 1.6 percentage points compared to the next-best ResNet-Transformer (96.21% F1), underscoring the synergistic benefit of combining ResNeSt’s fine-grained feature extraction with the Transformer’s sequential modeling.

A detailed per-class analysis is crucial for this highly imbalanced dataset (Table 13, Figs. 16 and 17). The model demonstrates exceptional performance on high-volume attack classes, achieving F1-scores of 99.68% (DDoS), 99.62% (DoS), and 99.87% (Mirai). For the challenging minority classes, ‘BruteForce’ and ‘Web’, the WGAN-based augmentation strategy was essential. As quantified in Table 8, WGAN elevated the F1-Score for ‘BruteForce’ to 0.8242 (from a 0.6283 baseline) and ‘Web’ to 0.7930 (from a 0.6275 baseline), validating its efficacy. While WGAN provides a clear boost, these minority classes remain the primary area for refinement. The metrics in Table 13 identify 618 False Negatives for ‘BruteForce’ and 1252 for ‘Web’. A corresponding analysis of the confusion matrix (Fig. 17) reveals that these misclassifications are not random; they are primarily confused with ‘Spoofing’ and ‘Recon’. This indicates a high degree of feature overlap between these specific IoT attack types, posing a persistent challenge even with effective data augmentation.

To contextualize the advancement offered by our proposed model, we compare its performance against several recent NIDS models on the same four benchmark datasets. This comparison serves to contextualize the performance of the TransNeSt architecture relative to other contemporary approaches. The comprehensive results of this comparative analysis are summarized in Table 14.

On the foundational NSL-KDD benchmark, TransNeSt achieves the highest F1-Score (99.04%), demonstrating a superior balance of precision and recall compared to models like MLF-DOU [19]. This robust management of the precision-recall trade-off is also evident on the more complex UNSW-NB15 dataset, where TransNeSt again secures the top F1-Score (91.92%), surpassing other specialized models [14,19].

The model’s advantage is particularly pronounced on large-scale, modern datasets. On CICIDS-2017, TransNeSt (99.18% F1) establishes a clear margin of over 1.1 percentage points against the next-strongest competitor, NIDS-CNNRF [24]. This robust performance is confirmed on the recent CICIOT2023 IoT dataset, where our model (97.85% F1) again outperforms all contemporary methods, including the recent RMCNN [15] (97.29%) and CTWA [22] (96.45%).

In summary, the consistent outperformance of TransNeSt across these four diverse datasets against strong, contemporary models validates its innovative architectural design. The synergy between the powerful multi-scale feature representation from the ResNeSt module and the global contextual understanding from the Transformer allows our model to set a new standard for performance in intrusion detection.