In abnormal traffic detection, the application of machine learning technology has become a key research direction. For example, Autoencoder (AE), eXtreme Gradient Boosting (XGBoost) and XGBoost combined with Principal Component Analysis (PCA) are advantageous in identifying and processing spatial data features.

In network traffic detection, an AE as a neural network architecture is employed for dimensionality reduction or feature extraction [3–5]. Ieracitano et al. [3] proposed an AE-based Intrusion Detection System (IDS). It combines statistical analysis with AE to extract more optimized and strongly correlated features. Andresini et al. [4] introduced a deep metric learning strategy that learns feature embeddings through triplet networks combined with Hnamte et al. [5] introduced a two-stage deep learning model (LSTM-AE) by combining Long Short-Term Memory networks (LSTM) with AE. This model aims to effectively identify anomalous behaviors in complex network data. In abnormal traffic detection, XGBoost applications address complex data challenges [6–8]. Kasongo et al. [6] used the XGBoost algorithm to reduce the dimensionality of the feature space, which improves the performance of various Machine Learning (ML) models and addresses challenges such as high-dimensional data spaces and dataset imbalance. On the other hand, the model combining particle swarm optimization and XGBoost is proposed by Jiang et al. [7]. This method focuses on improving the parameter settings of XGBoost through the Particle Swarm Optimization (PSO) algorithm, which enables the model to find optimal solutions within a wider parameter space. This approach significantly improves the performance of XGBoost on classification problems. Further research has improved model performance by combining PCA with XGBoost [9–11]. Bhattacharya et al. [9] demonstrated the effectiveness of PCA in spatial feature extraction and dimensionality reduction, especially when it is combined with the Firefly algorithm and XGBoost. This approach lays the foundation for data classification by reducing data dimensions while retaining important information. Pan et al. [10] further expanded the application of PCA on this basis, especially in dealing with class imbalance problems. By integrating with the Adaptive Synthetic Sampling (ADASYN) algorithm, PCA not only reduces the complexity of the data but also enhances the balance of the dataset, which provides a more optimized feature set for XGBoost. With the issues of feature redundancy and the neglect of feature mean, Chen et al. [11] proposed an optimized feature extraction algorithm. This method initially applies Kernel PCA (KPCA) to project the original data into a high-dimensional space, removing redundant and irrelevant features. Subsequently, it utilizes Linear Discriminant Analysis (LDA) to perform secondary feature extraction in the new feature space, taking into account the mean differences between and within classes, thereby improving the effect of feature extraction. Diwan et al. [12] proposed a novel, lightweight feature selection method for IoT intrusion detection, which leverages rank-based chi-square, Pearson correlation, and score correlation to identify key dataset features. Similarly, Jhansi et al. [13] used Ant Lion Optimization, Cuckoo Search Optimization, and Firefly Optimization alongside autoencoders for efficient Application Programming Interface (API) scheduling in malware detection.

Although existing machine learning methods have achieved good results in detecting anomalous traffic, many models highly depend on manually extracted features. This approach is not only time-consuming but may also overlook some important complex features in the data. Furthermore, manual feature extraction can lead to models struggling to adapt to new or unknown attack patterns, limiting their generalization and practicality. Therefore, employing methods that automatically learn and extract features is particularly important to overcome the limitations of current approaches in feature extraction and processing.

Compared with machine learning methods, deep learning performs better in terms of learning accuracy and portability because it does not require manual design of features. Abnormal traffic detection typically always relies on spatial and temporal features, as well as a combination of both. Spatial features are usually extracted using Convolutional Neural Network (CNN) [14–16]. Li et al. [14] proposed a multi-CNN fusion method. This method divides feature data into four parts for processing and fusion, aiming to enhance the precision and efficiency of network intrusion detection. In order to further optimize the problem of feature extraction caused by sample data differences, Shi et al. [15] proposed the Deep Abnormal Network Traffic Detection (DANTD) method for effective spatial feature extraction. This model uses deep convolutional autoencoders for high-order feature extraction and employs Generative Adversarial Networks (GAN) for data augmentation. The extraction of temporal features relies on the temporal convolution model [17,18]. Li et al. [17] proposed a method using dynamic chaotic Cross-optimized bidirectional residual-gated recurrent unit and Wasserstein generative adversarial network with generated feature domains. This approach leverages the strengths of GRU for processing time series data, which optimizes weights to achieve more efficient feature extraction and reduced time complexity. Cai et al. [18] developed a method using Bidirectional Temporal Convolutional Network (BiTCN) and Multi-Head Self-Attention (MHSA) mechanism. The method employs BiTCN to capture bidirectional semantic features of network traffic and uses MHSA to assign varying weights to different subsequence segments. Recent studies highlight the significant advantages of hybrid models that focus on extracting both spatial and temporal features [19–23]. Kanna et al. [19] introduced a model combining an Optimized CNN and Hierarchical Multi-Scale LSTM (HMLSTM). This model employs Lion Swarm Optimization (LSO) to enhance CNN spatial feature extraction, while HMLSTM handles temporal feature extraction. Anitha et al. [20] developed a network integrating Bidirectional Long Short-Term Memory (BiLSTM) with a CNN, where the BiLSTM captures long-term dependencies in time series data and the CNN processes and classifies the data. Zhu et al. [21] proposed a model that combines 1D-CNN and BiLSTM. This method effectively extracts time series and spatial features. It also employs a cost penalty matrix and an improved cross-entropy loss function to enhance the recognition of minority class samples. Wang et al. [22] further proposed a model for spatial-temporal feature fusion, using a simplified CNN for spatial learning and BiLSTM for temporal feature learning, incorporating an attention mechanism for effective feature integration. To address overfitting in training, Hassan et al. [23] developed a hybrid deep learning model that combines CNN with Weighted Decreasing LSTM (WDLSTM) to extract key features efficiently. Since the accurate extraction of spatial features directly affects the sensitivity and accuracy of abnormal traffic detection. It is a core link to ensure network security. We mainly focus on the extraction of spatial features in this paper.

Multi-scale feature extraction methods are crucial for improving model accuracy, generalization, capturing data features at various levels [24–27]. Duan et al. [24] introduced a Multi-Scale Residual Classifier (MSRC) for anomaly traffic detection, utilizing wavelet transform to effectively process multi-scale network traffic features. This enhances the accuracy of detecting network traffic anomalies. Yu et al. [25] developed a high-precision intrusion detection system using a Multi-Scale CNN, which extracts features from disordered data to increase the accuracy of intrusion detection. He et al. [26] proposed a method combining a Variational Gaussian Model with a One-Dimensional Pyramid Depth Separable Convolution (PyDSC) network. This approach simplifies complex features using PyConv with added DSC to reduce network complexity. Addressing high-dimensional and complex datasets, Zhang et al. [27] firstly analyzed spatial features using a Multi-Scale Convolutional Neural Network, and processed temporal features with LSTM. Sathya et al. [28] introduced a classification method, which utilizes a dual weight update mechanism to differentiate between attack and non-attack data in IoT devices. Ravi Kiran Varma et al. [29] proposed a software-defined IoT intrusion attack detection method based on enhanced Elman spiking neural network.

In summary, existing abnormal traffic detection methods have obvious limitations in multi-dimensional data feature extraction, identification and maintenance of high-level features. Specifically, (1) some existing methods usually only focus on feature extraction at a single scale, ignoring other levels of information in the data. This limits the model’s ability to understand and process complex data structures. (2) With the complex high-dimensional data, it is difficult to identify subtle differences and high-dimensional features. Therefore, this paper proposes an MD block to effectively capture and integrate features of different scales in multi-dimensional data. And a MRD block is designed to identify complex patterns in high-level features with subtle differences.

Raw data with pcap format generated by industrial IoT need to be converted to images with grayscale format by method in [30]. This paper adopts the ResNeXt network [31] as its main architecture. The preprocessed data are fed into MD block with different scales to cover more global features. Dilated convolutions are used in the MD block to enhance the receptive field. The output of the MD block serves as the input for the MRD block, which combines the Max-Feature-Map (MFM) with residual blocks and dual-channel pooling for the enhancement of precision and efficiency. The MR block of MRD is used to enhance the extraction of key features and maintain feature continuity in deep networks. Finally, the classification results are obtained through the fully connected layer. The overall structure of the MR-MRD-ResNeXt model is shown in Fig. 1a. Meanwhile the detailed MD and MR block are shown in Figs. 1b and 1c, respectively.

With the introduction of deep learning, feature extraction has been automated and the ability to process complex data has been improved. However, some feature extraction methods in deep learning are still limited to a single scale, which often focuses only on local details or overall patterns and neglects other important dimensions of the data.

Inspired by the multi-scale feature processing method [32], a four-layer multi-scale feature extraction module is used as shown in Fig. 1b. In this structure, a 1 × 1 convolution is used in the first branch to enhance the feature representation capabilities of the model, improve the detail capture capability of the networks, and optimize computational efficiency. In the last three branches, dilated convolution is added to the 3 × 3 convolution. The dilation rate of the dilated convolution is adjusted to 1, 2, and 3. A convolution with a dilation rate of 1 is equivalent to a regular convolution and it can capture the potential local features. Gradually increasing the expansion rate to 2 and 3, the receptive field’s distribution is optimized. It captures multi-scale information without changing the size of the feature map, thereby capturing a wider range of spatial features. Consequently, this approach avoids excessive sparseness in feature maps and enhances model sensitivity to small-scale features. The features extracted at different scales are integrated to ensure them work collaboratively in the final decision-making process. Afterwards, by integrating batch normalization and ReLU activation function in the post-convolution stage, the training stability and nonlinear expression ability of the model are improved.

When dealing with high-dimensional data with multiple characteristics and complex relationships, traditional methods are often difficult to effectively distinguish and maintain these key features. In addition, as the network depth increases, the model may lose sensitivity to important features, which results in insufficient feature recognition and generalization capabilities. In order to avoid the fuzzy identification of high-dimensional data with multiple characteristics, in this paper we propose an MRD block. The feature extraction is optimized by combining MFM and residual blocks. As illustrated in Fig. 1c. MFM selects the most significant feature responses across channels to enhance key features. Assume Xhi,wj,ck represents the value of channel ck at position (hi,wj) in feature map X∈RC×H×W, then MFM(X)hi,wj,ck is calculated as Eq. (1). (1)MFM(X)hi,wj,ck=maxck∈C(Xhi,wj,ck,Xhi,wj,ck+C2)

The residual block helps maintain the continuity and stability of features in deep networks through its skip connections. In each layer of the residual block, through skip connections. In each layer of the residual block, skip connections allow the model to retain the feature information from the previous layer, to reduce information loss, and to improve the accuracy of feature recognition. This combination strategy not only improves the model’s ability to maintain features in complex data, but also effectively reduces overfitting and enhances generalization capabilities.

In addition, because dual-channel pooling can capture features more comprehensively in feature extraction, it combines the advantages of the significant feature response of max pooling and the global information capture of average pooling. Therefore, we adopt it to process key information in complex data and improve the performance of the model in abnormal traffic detection. Assume D(X) and MaxPool(X) represent the feature map obtained after dual-channel pooling and the result after the maximum pooling, then D(X)=MaxPool(X)⊕AvgPool(X). AvgPool(X) is the result after using the average pooling and ⊕ represents the element-wise phase.

Through the combination of MFM, residual block and dual-channel pooling. The MRD block enhances the model’s feature capture capabilities and optimizes computational efficiency to achieve more efficient and accurate abnormal traffic detection. The detailed MRD is shown in Algorithm 1.

The preprocessed dataset is divided the training set and validation set into a ratio of 8:2. The training set is used for the learning process of the model, while the validation set is used for the evaluation of model performance. In the first stage of the model, the MD block is responsible for capturing features of different scales through dilated convolution technology, which helps the model learn more detailed data representation. Next, in the MRD block, we use the MFM method to capture the most significant feature responses in the deep network structure. At the same time, we enhance the ability of the model to maintain deep features through residual connections. This method helps avoid information loss during training, especially when the network depth is large. Additionally, by combining the advantages of maximum pooling and average pooling, it can retain rich feature information while reduce the number of parameters.

During the training process, we choose Adam [32] as the optimizer to adjust and optimize the model weights with its effective adaptive learning rate. Training is performed in small batches (the batchsize is 32), which helps to increase the updating frequency of model and improve the accuracy of gradient estimation. In addition, we use the ReduceLROnPlateau learning rate [33] scheduler to dynamically adjust the learning rate. This strategy determines whether to reduce the learning rate based on the model’s performance in terms of loss on the validation set, which ensures that the model does not get stuck in a local minimum during the training process. Model training is scheduled for 50 epochs to ensure sufficient iterations to train the network. The learning rate starts at 0.001, and after the 46th epoch, the learning rate is reduced to 1∗10−6. After each epoch, the validation set is employed to assess the model’s performance by accuracy and loss metrics. And the classification result is calculated with the average of multiple experiments. Finally, after a series of training processes, our model showed excellent performance on the abnormal traffic detection, verifying the effectiveness of the MD and MRD blocks.

To verify the effectiveness of the MD-MRD-ResNeXt and its variants of the network proposed in this paper for abnormal traffic detection, this section conducts detailed performance comparison experiments. The experimental datasets used are USTC-TFC2016 [30] and ToN-IoT-Network [34]. The USTC-TFC2016 dataset is composed of two segments. The first segment includes a collection of ten varieties of malicious traffic, gathered by CTU (Czech Technical University) researchers in real-world network settings from 2011 to 2015. The second segment comprises a set of ten kinds of benign traffic, obtained through IXIA BPS (Ixia Breaking Point Systems). A total of 202,921 records from the USTC-TFC2016 dataset are used. The ToN-IoT-Network dataset was developed by the IoT lab at UNSW (The University of New South Wales) Canberra in collaboration with Cyber Range. It encompasses telemetry data from connected devices, logs from both Linux and Windows operating systems, as well as network traffic from IIoT systems. This heterogeneous data was collected from a medium-sized IoT network. A total of 260,462 records from the ToN-IoT-Network dataset are considered. The data used in USTC-TFC2016 and ToN-IoT-Network is detailed in Table 1. All these data are converted to images with grayscale format by method in [30].

In evaluating the efficacy of the MD-MRD-ResNeXt model, this paper employs four evaluation metrics: Accuracy (AC), Precision (PR), Recall (RC) and F1 score, which are computed by Eqs. (2)–(5). Among all these evaluation criteria, True Positive (TP) refers to the cases where abnormal network traffic is accurately identified by the MD-MRD-ResNeXt. False Positive (FP) refers to the instances where the MD-MRD-ResNeXt incorrectly labels normal network traffic as anomalous. False Negative (FN) refers to the cases where abnormal network traffic that the MD-MRD-ResNeXt fails to identify. True Negative (TN) refers to the cases where normal network traffic is accurately identified by the MD-MRD-ResNeXt. (2)AC=TP+TNTP+TN+FP+FN

The ratio of the proportion of correct predictions by the model to the total number of predictions. (3)PR=TPTP+FP

The proportion of positive examples predicted by the model that are actually positive. (4)RC=TPTP+FN

Among all actual positive examples, the proportion of positive examples correctly predicted by the model. (5)F1=2∗PR∗RCPR+RC

The harmonic means of precision and recall.