Traditional signature-based methods have undeniable advantages such as effectiveness against known DDoS attacks, low false positive rate, and ability to respond quickly to attacks. However, the signature-based method clearly shows its disadvantages against unknown DDoS attacks, zero-day attacks as well and high dependence on databases. Researchers have recognized the limitations of signature-based systems and have explored other approaches. In recent years, ML and DL technologies have been a research trend in developing IDS systems [9].

Recent research by Maseer et al. [2] synthesized and compared current popular ML algorithms on the CICIDS2017 dataset. In their research, multiple algorithms have demonstrated highly favorable outcomes, including Support Vector Machine (SVM), Random Forest (RF), Naive Bayes (NB), Artificial Neural Networks (ANN), and Convolutional Neural Network (CNN).

Ho et al. [3] introduced a novel CNN approach that surpasses conventional single-class classification methods, delivering exceptional performance in the realm of multi-class classification, particularly in the identification of both known and unknown attacks. Furthermore, many studies have applied CNN-based models, producing highly effective results [10,11].

Kim et al. [4] introduced a DL model that combines CNN and Recurrent Neural Networks (RNN) to detect Denial of Service (DoS) attacks. They optimized the CNN design through numerous experiments and achieved impressive results, particularly in terms of accuracy on two datasets KDD and CSE-CIC-IDS2018.

Kiranyaz et al. [12] surveyed 1D Convolutional Neural Networks (1D-CNN) and their main applications. The 1D-CNN model [12,13] has demonstrated its advantages in a scarce training data environment as well as in some specific fields such as anomaly detection, personalized biomedical data classification, early diagnosis, etc.

Beitollahi et al. [14] proposed an ML approach for detecting DDoS traffic. Their approach employs a Radial Basis Function (RBF) network in conjunction with the cuckoo search algorithm (CSA) to identify Application-layer DDoS attacks. This approach stands out for its effective performance in identifying DDoS traffic compared to several other methods, with notably low error rates and high precision.

Laghrissi et al. [15] presented an IDS that utilizes Long Short-Term Memory (LSTM) networks, in combination with Principal Component Analysis (PCA) and Mutual Information (MI) techniques. This method achieves outstanding accuracy compared to other methods in both binary and multiclass classification.

The Open Set Recognition (OSR) technique aims to classify data into known classes and detect instances that do not belong to any known class, unknown class, or new instances [5].

Recent advancements in OSR techniques include the work of Bendale et al. [16], who introduced the OpenMax technique. This technique has become a crucial method in the field of open-set recognition, where systems need the capability to recognize objects that do not belong to any of the classes learned during training. The fundamental theory behind the OpenMax algorithm involves calculating the “openness” level of a data sample based on the distance between that sample and the nearest samples in the nearest known class. OpenMax allows for the determination of whether a data sample belongs to a known class, an open-set class (a class corresponding to openness), or is unknown (not belonging to any known class). This enhances recognition capabilities in real-world situations where objects that do not belong to any known class may appear.

Ge et al. [17] have published their research results as a development method of OpenMax. Unlike previous methods where unknown classes were inferred based on characteristics or distance decisions with known classes, the authors’ novel approach offers a clear framework and decisive criterion for unknown classes. The proposed technique, known as Generative OpenMax, enhances OpenMax by incorporating Generative Adversarial Networks (GANs) to create images for new classes artificially. Additionally, Yoshihashi et al. [18] have proposed the Classification-Reconstruction learning for the OSR method (CROSR) to enhance the reliable detection of unknown classes without affecting the accuracy of known classes. Research results have shown the superiority of this method over traditional approaches.

Generally, Open-Set Recognition has been receiving the attention and research efforts of many scientists and scholars worldwide. The mentioned papers provided earlier represent only a fraction of the noteworthy contributions within this field.

The task of identifying unknown DDoS attacks is difficult, requiring inventive methods to improve the protection of network infrastructures. In recent years, researchers have made significant progress in the field of unknown DDoS attack detection, employing various techniques and methodologies to identify and mitigate these threats [19]. This section provides an overview of recent advances in unknown DDoS attack detection, highlighting key contributions in this domain.

Extreme Value Theory (EVT) has been utilized to enhance the identification of DDoS attacks [20]. This statistical methodology has proven effective in capturing the extreme behaviors of network traffic, aiding in the recognition of anomalous patterns associated with DDoS attacks.

Gaussian Mixture Models (GMMs) and related methods have been widely used for determining the distribution of network traffic data [21,22]. Chapaneri et al. have explored the use of several GMMs for modeling individual input features in the context of DDoS detection [21]. By modeling the underlying data distribution, this technique can help distinguish between benign and malicious traffic patterns, contributing to more accurate detection. Shieh et al. have adopted DL techniques, including Bidirectional Long Short-Term Memory (BI-LSTM) networks and GMMs, for the identification of unknown DDoS attacks [22]. Their research demonstrates the efficacy of combining deep learning and statistical modeling for robust attack identification.

Yang et al. have introduced the AutoEncoder-based DDoS attacks Detection Framework (AE-D3F), a novel technique for threat detection, which has shown promise in identifying anomalous network behavior [23]. By leveraging autoencoders, they contribute to the arsenal of tools for enhancing DDoS attack detection.

Generative Adversarial Networks (GANs) have demonstrated efficacy in DDoS attack detection [24,25]. GANs are proficient in generating synthetic data that can be used to compare and contrast with real network traffic, aiding in the identification of malicious activity. Lin et al. have presented the IDSGAN framework, which incorporates GAN networks as part of a defense system to protect against DDoS attacks [24]. This approach focuses on the proactive use of GANs to safeguard network resources. Chauhan et al. have harnessed Wasserstein GAN (WGAN) to address training issues in DDoS detection models [25]. By incorporating WGAN, they aim to improve the robustness and reliability of DDoS detection techniques.

Fuzzy C-Means (FCM) clustering is a common unsupervised learning technique used for data clustering and classification. In the context of DDoS attack detection, FCM has been applied to group network traffic data into clusters, allowing the identification of abnormal traffic patterns related to attacks. The work by Wu et al. [26] used FCM clustering to partition network traffic data into distinct clusters and then employed anomaly detection techniques to identify DDoS attacks within these clusters. This approach has shown promising results in the identification of both known and unknown attacks.

In this section, we have discussed various approaches related to DDoS attack detection, OSR techniques, and Fuzzy C-Means clustering. For convenience, we have established a comparison table among recent research studies in machine learning and deep learning. Table 1 summarizes the techniques used, the scope of the problems, and some limitations of the studies.

Recent notable studies have primarily focused on addressing the challenge of Closed Set Recognition (CSR). Some performance evaluations are conducted on outdated datasets (KDD99) or specific datasets such as NSL-KDD or CICIDS2017. Regarding methods for tackling the OSR problem, some approaches exhibit drawbacks, such as complexity and difficulty in real-world deployment. Our proposed method successfully addresses both CSR and OSR problems, with performance validated on both the CICIDS2017 and CICDDoS2019 datasets. This approach achieves exceptionally high performance and flexibility for real-world deployment. Of course, our method still has some limitations, which will be discussed in the section “Discussion” of this article.

In the following sections, we will explain our method in detail, a solution involving a deep neural network combined with the SLCPL and FCM techniques to detect unknown DDoS attacks. This combination is relatively new and remains unexplored.

AlexNet, a Convolutional Neural Network, was developed by Krizhevsky and his team in 2012. It garnered significant acclaim for its exceptional performance in the ImageNet Large Scale Visual Recognition Challenge (ILSVRC) competition [8]. This network architecture marked an important milestone at the time, having a major influence on the advancement of deep learning and the development of the task of image classification.

The main characteristics of AlexNet lie in its combination of depth, convolution, and pooling layers, along with a large number of trainable parameters. Its architecture is as follows:

Convolutional and Pooling Layers: AlexNet utilizes convolutional and pooling layers to extract features. The layers efficiently capture specific characteristics within images and employ pooling layers to decrease the complexity of feature maps while preserving important features.

The success of AlexNet underscores the formidable capabilities of deep learning in tasks like image classification. While more profound neural network architectures emerged in the subsequent years, AlexNet’s role as a starting point in deep learning has cast a lasting influence on the design and development of subsequent models. Fig. 2 describes the architecture of the AlexNet network.

Spatial Location Constraint Prototype Loss (SLCPL) [6] is an OSR technique, a loss function designed for the Convolutional Neural Network model. The SLCPL extends from the Generalized Convolutional Prototype Learning (GCPL) [31] by introducing spatial location constraints to solve a common issue faced by most current training methods, such as SoftMax and GCPL. The problem lies in the concentration of known features towards the center of the feature space, resulting in the overlapping of known and unknown feature distributions. To prevent this phenomenon, a spatial location restriction is incorporated into the loss function while undergoing the prototype learning procedure. This allows SLCPL to manipulate the spatial positioning of the prototypes. Consequently, the known features tend to be located in the periphery of the feature space, while the center of the feature space is typically reserved for unknown features.

We will describe the SLCPL function as follows:

Given a training set S = {(x1,y1), (x2,y2), (x3,y3), ….} with N known classes, and the label xi is yi∈{1,2,3,…,N}. A CNN is a classifier with the parameters θ and the embedding function F. The prototypes O = {Oi,i=1,2,3,…,N} are initialized randomly or with a distribution.

With a training sample (x, y), the SLCPL loss function can be represented by:

(1)Ls=LG(x,y;θ,O)+slc(O)

Here LG(x,y;θ,O) is a GCPL loss function and slc(O) is a spatial location constraint introduced by SLCPL.

LG(x,y;θ,O) can be represented as follows: (2)LG(x,y;θ,O)=l(x,y;θ,O)+λ.pl(x;θ,O)

The function l(x,y;θ,O) will be optimized to cluster different known classes. And l(x,y;θ,O) can be represented as follows: (3)l(x,y;θ,O)=−log⁡p(y=k|x,F,O)=−log⁡e−d(F(x),Ok)∑i=1Ne−d(F(x),Ok),with K = 1,....N

where d(F(x),Ok) is the Euclidean distance between F(x) and Ok.

The param λ and the constraint pl(x;θ,O)is used to enhance the compactness of the cluster. (4)pl(x;θ,O)=||F(xk)−Ok||2,with K=1,2,....N

where xk is the training sample of class k.

The spatial location constraint slc(O) can be represented as follows: (5)slc(O)=1N−1∑i=1N(ri−1N∑j=1Nrj)2

where ri=d(Ok,Oc) and Oc=1N∑i=1NOi.

To clarify, slc(O) represents the variability of the distances ri between the prototype of the cluster and the center Oc.

As presented, the time complexity to compute the SLCPL loss function for each data sample is O(N.D), where N is the number of known classes and D is the dimensionality of the feature space.

Finally, SLCPL is a new OSR technique with the ability to enhance the accuracy for classifying known and unknown classes. And SLCPL and 1D AlexNet are a promising combination for our OSR module.

Fuzzy C-Means (FCM) [7] is a clustering analysis method to partition samples into multiple fuzzy categories. The algorithm considers the degree of membership of each sample to every category rather than just employing traditional hard clustering methods. In this research, we integrated FCM with OSR modules including SLCPL and 1D AlexNet. The 2-feature output from the OSR module will be clustered by FCM for Unknown DDoS attack detection.

We will describe the FCM algorithm as follows:

Given N samples and C cluster centers, each sample is denoted as xi, and each cluster center as vj. Define the fuzzy membership matrix U, where Uij is the degree of membership of the sample xi to cluster center vj. The objective function of FCM is to minimize the following cost function: (6)J(U,V)=∑i=1N∑j=1CUijm||xi−vj||2

Here, m is the fuzziness param that controls the degree of fuzziness in clustering. This objective function combines the weighted total of Euclidean distances between the samples and the cluster centres. The algorithm operates in the following manner:

As presented, the time complexity of FCM typically is O(I.N.C.D), where I represents the number of iterations, N represents the number of data samples, C represents the number of clusters, and D represents the dimension of each data sampling.

The core principle of this algorithm lies in iteratively updating the fuzzy assignment matrix and cluster centers, continuously adjusting the membership degrees of samples and the positions of cluster centers to minimize the objective function. Through the fuzzy assignment matrix, one can obtain the fuzzy membership degrees of each sample concerning each cluster center rather than solely obtaining rigid classification outcomes.

In this research, we utilized the CICIDS2017 [32] and CICDDoS2019 [33] datasets to evaluate the proposed IDS. The CICIDS2017 and CICDDoS2019 are widely used datasets for IDS research and development. Developed by the Canadian Institute of Cyber Security (CIC), they aim to simulate various cyber attacks and common traffic in real-world networks. The simulation environment is set up with complete network topology, actual network traffic, and attack behaviors. They are used for various purposes, such as training, validating, and testing IDS performance. Both are valuable for many tasks like feature selection, model optimization, algorithm comparison, and other intrusion IDS-related research.

The CICIDS2017 dataset was collected from 9 AM on Monday, July 03, 2017, to 5 PM on Friday, July 07, 2017, spanning a duration of 5 days. The CICIDS2017 dataset encompasses a diverse range of attacks, such as Brute Force, DoS, Web Attack, Infiltration, Botnet, Heartbleed, and DDoS. In addition, the CICDDoS2019 dataset includes both benign traffic and the latest prevalent DDoS attacks such as DNS, LDAP, NETBIOS, and SNMP.

We utilized the CICIDS2017 Wednesday dataset to train the model in the context of known attack detection. In addition, we utilized CICIDS2017 Friday and CICDDoS2019 datasets in the context of unknown attack detection. Table 2 describes the structure of the datasets we used in this research.

Data pre-processing is a very important phase in conducting machine learning experiments. Pre-processing ensures the accuracy and reliability of the output. In this research, this phase includes Data cleaning and Feature transformation.

In this research, we utilized a workstation with the Ubuntu 20.04 OS. Our workstation is equipped with an AMD Ryzen 5700X 8C16T processor, 96 GB DDR4 memory, and Nvidia RTX3070 and Nvidia RTX2060 as computational accelerators. Regarding the model framework, we undertake development using PyTorch 1.11.0, Sklearn, and Python 3.9.12.

In this research, we have built the 1D AlexNet model using the Pytorch framework and calculated the evaluation metrics using Sklearn. To evaluate the model’s effectiveness, we trained the model ten times using a different random number each time. We chose Adam as the optimizer. The detailed settings of the parameters are described in Table 3. And Fig. 4 shows the network model of 1D AlexNet.

In machine learning and pattern recognition, evaluation metrics serve as quantifiable measures to gauge model performance and effectiveness. These metrics assess and compare a model’s performance on a specific task or problem. In the context of this research, we used the following metrics:

Accuracy: Evaluates the accuracy of model predictions.

These metrics will measure the model’s performance in distinguishing between different classes. Table 4 is the definition of the confusion matrix in machine learning.(10)Accuracy=TP+TNTP+FP+TN+FN (11)Precision=TPTP+FP (12)Recall=TPTP+FN (13)F1 score=2∗Precision∗RecallPrecision+Recall

For known DDoS detection, we utilized CICIDS2017 Wednesday dataset for the training model. Initially, we trained the 1D AlexNet model and visualized the 2-feature output from the 1D AlexNet module in feature space, as shown in Fig. 5. We observed that the training process made the distribution of the known classes more dispersed. In some cases, the distribution areas of known classes have overlapped with one another.

Therefore, we continued to modify 1D AlexNet several times to improve the result. We have tried to increase the number of convolutional layers of the 1D AlexNet. After numerous improvements, we achieved the desired outcomes. Fig. 6 shows that the result of our final model (Fig. 3) improves significantly compared to the initial model in terms of the distribution of known datasets.

Furthermore, we obtained excellent results as shown in Fig. 7, when evaluating known attacks by using the same CICIDS2017 Wednesday dataset. All evaluation metrics exceed 0.98.

We used a lot of recent research in detecting DDoS attacks for the comparison. Chosen algorithms include Deep Neural Network with Genetic Algorithms (2023) [28], Multilayer Perceptron (2023) [29], and Convolutional Neural Network featuring Geometrical Metric (2023) [34]. All algorithms were evaluated on the CICIDS2017 dataset for known attack detection. Table 9 shows the comparisions of our method with recent ML algorithms.

The performance metrics of our proposed method are slightly larger than DNN-GA and MLP. They have accuracy, precision, and recall of approximately 0.99. Compared with the CNN-Geo algorithm, the proposed method has similar performance with negligible differences.

Next, we continued to compare with other algorithms in detecting unknown attacks. The algorithms we chose to compare include GMM (2021) [21], DBSCAN with Random Forest (DBSCAN-RF) (2022) [27], DBSCAN with SVM (DBSCAN-SVM) (2022) [27], One-Dimensional Deep High-Resolution Network with One-Class SVM (1D-DHRNet-OCSVM) (2022) [30], and CNN featuring Geometrical Metric (CNN-Geo) (2023) [34]. All algorithms are trained on the CICIDS2017 dataset and evaluated on other datasets. The performance metrics of the overall system are listed in the Table 10.

Compared with DBSCAN-SVM and DBSCAN-RF methods, our method has much more outstanding performance metrics. Compared with GMM (2021), 1D-DHRNet-OCSVM (2022), and CNN-Geo (2023), our method has slightly larger metrics. Combining the two cases of known and unknown attack detection, our performance is equal to or more outstanding than other methods. This proves that our research has caught up and surpassed recent outstanding research.

The validity of a study is a concern that we prioritize throughout our research. For internal validity, we are meticulous in applying methods, algorithms, and experimental procedures in the fields of machine learning and deep learning. We have used multiple random seeds, as well as dropout techniques to avoid overfitting, during both model training and evaluation. Regarding external validity, in this research, we utilized two main datasets, CICIDS2017 and CICDDoS2019. The validity of these datasets will significantly impact the applicability of our model in real-world scenarios. These are two well-known and standard datasets in the broader field of network attack detection, including specific DDoS attacks. All of these efforts are aimed at ensuring the reliability of our research.

However, new attack methods and their variants are continually evolving. Deploying an unknown attack detection module along with ongoing support from network experts and machine learning incremental learning techniques forms a robust combination that enhances the system’s capabilities over time. While the defense of our system has demonstrated effectiveness against a variety of DDoS attacks, it is essential to recognize certain limitations. This is particularly evident when confronting Layer 7 (L7) DDoS attacks and the intricate strategies employed in adversarial scenarios.

As mentioned, our proposed model heavily relies on two primary datasets, CICIDS2017 and CICDDoS2019, focused predominantly on DDoS attacks at Layer 3 and Layer 4. This specialization in lower network layers may result in inadequacies in addressing the distinct characteristics of DDoS attacks at the L7 layer, the Application layer. Notably, L7 DDoS attacks pose a unique challenge as they exploit vulnerabilities within applications, services, or protocols, diverging from conventional approaches that target network infrastructure. Detection mechanisms may encounter challenges in discerning benign from harmful activities at this granular level. Techniques such as encryption-based intrusion and complex application-layer exploitation, while effective, may incur significant processing costs on lightweight systems, impacting overall performance and response times during high-intensity attacks.

Furthermore, our system has not been studied to face adversarial DDoS attacks, a major evolving threat. These attacks dynamically adjust in attack types, IP address manipulation, and various attack vectors, deliberately employed to complicate defense mechanisms.

In summary, minimizing L7 DDoS attacks and adversarial DDoS attacks requires a sophisticated and adaptive approach, while acknowledging the limitations mentioned above. Traditional mitigation strategies may not be sufficient, and research on advanced techniques needs to be conducted and implemented.