Fig. 1 shows a diagram of the 5G network, with the core inside and the base station and data network (DN) outside. 5G network has three major components: NG-RAN (RAN), DN, and core. The RAN further comprises the base station gNodeB (gNB) and user equipment (UE), i.e., the smartphone. The user is provided network service by being connected to the core using the N3 interface of gNB. DN is connected to UE using the N6 interface of the core and delivers the data to the user. The core can be divided into the user plane (UP), responsible for data transfer, and the control plane (CP), which controls the user end. The core provides services to all the NFs connected to the SBI.

Among the NFs used in this research, the UPF and NWDAF are the most relevant. The UPF was the NF that mainly comprised the 5G dataset in our research. In addition, NWDAF is an NF for internalizing security analysis functions using ML in 5GC, which we are studying and analyzing.

The 5G dataset was established using the N3 interface connected to UPF. The UPF is an NF that handles UP, which is responsible for packet routing and forwarding, packet inspection, quality of service (QoS) handling, and external protocol data unit (PDU) sessions and also interconnects DN [25]. UPF has three reference points, as shown in Fig. 1: N6, N4, and N3 [24]. N4 is the interface between Session Management Function (SMF) and UPF, N6 is DN and UPF, and N3 is gNB and UPF. The N3 interface undergoes encapsulation that expands the GPRS tunneling protocol (GTP) header in the previous internet protocol (IP) header through the GTP. GTP is a protocol for encapsulating and tunneling the IP packet data provided and received by the user in DN. The experiments to construct the 5G dataset utilized the N3 interface. As a result, Fig. 2 shows a structure where the UPF and RAN are connected via the N3 interface. In addition, as an experiment, IP-based packets were dumped to UEs in the RAN and changed to GTP-based packets in the 5G Network (5GC) as they passed through the N3 interface. We utilized packets from this modified 5G core in our experiments.

To provide security analysis functions to the 5GC, NWDAF, which has the ML function among the 5GC NFs, was studied and analyzed. The current 3GPP standardization defines NWDAF. The major function of NWDAF is to use ML and provide analysis results of the network data [2]. Fig. 3 shows the three steps NWDAF takes to deliver the derived network analysis results: (1) collect network data, (2) analyze network data using ML, and (3) provide the analysis results. In addition, the NWDAF structure consists of the Model Training Logical Function (MTLF) and the Analytics Logical Function (AnLF). MTLF trains the ML models and provides the trained ML models. AnLF performs inferences, derives analytics information, and exposes analytics services. As such, NWDAF of 5GC trains the ML models and uses the trained models to provide the derived network data analysis services. Therefore, we aimed to study the ML training method for detecting IoT DDoS attacks by using and collecting the 5GC traffic to provide the security analysis function using the ML model that was trained in 5GC, similar to NWDAF.

Oversampling is the most widely used technique for overcoming network traffic imbalance due to the intuitive principle that rewards imbalance by increasing the number of traffic samples that belong to the minority class [9–15]. This study used SMOTE, which is the most usual oversampling method. SMOTE generates new data by copying virtual data in the minority class through the distance-based rule. Among previously reported oversampling methods, SMOTE improved overfitting as the disadvantage of random oversampling (ROS) [19]. However, SMOTE does not consider the majority class because new data is generated among the minority class data. Thus, the generated data only reflect the properties of data in the minority class. As a result, they may be prone to noise and unable to augment high-dimensional data efficiently. This research employed scikit-learn, an ML open-source library for SMOTE [28].

Data augmentation is a technique that increases the number of data by applying various transformations to the original data to create new data [29]. This study used GAN, a popular method for data augmentation. GAN, which uses DL for augmenting data, comprises a generator that generates data and a discriminator that classifies the generated data. The two have a feedback relationship; the generator generates data that are improved by learning through feedback provided by the discriminator. GAN is being actively researched and is used for data diversification. Data can be distinguished mainly into sequential and tabular forms. This study used tabular 5GC traffic data to train the ML model, and a CTGAN validated for generating tabular network data was employed [4,30,31]. In particular, this study attempted to solve the data imbalance problem by generating attack data. To use CTGAN, we leveraged a related paper and open-source published on GitHub [32,33].

The kitsune dataset, which is openly offered in Kaggle, was used as the wired network environment dataset to generate and gather 5GC traffic [34,35]. Kitsune consists of nine DDoS attack datasets generated by IoT equipment in a wired network environment. The nine types of DDoS attacks are as follows: OS Scan, Fuzzing, Video Injection, ARP MitM, Active Wiretap, SSDP Flood, SYN DoS, SSL Renegotiation, and Mirai botnet. These attack types were used, excluding two (Video Injection and Mirai botnet). The exclusion was because Video Injection was the only type that included the Ethernet class’s logical-link control (LLC) protocol. The Mirai botnet had 3.4 times fewer attack packets compared with the average of the eight other types.

Another reason for selecting kitsune for this study is that it was suitable for conducting the GPRS Tunneling experiment using the 5G testbed, as the provided data formats were diverse. Kitsune offers the following three data formats for the nine types of attacks: the dataset with the original packet pre-processed (.csv), a labeled dataset indicating the malignity or normality of the packet (.csv), and captured dataset of the original network (.pcap). The original network dataset (.pcap) and labeled dataset (.csv) were used for gathering 5GC traffic.

Building the 5G testbed. The 5G testbed was constructed in a Linux 18.04 environment. To build the 5G testbed, we leveraged two open-source software programs. The open-source utilized UERANSIM for RAN and Open5GS for the core. Finally, we built a testbed with 5G RAN (UERANSIM) + 5G Core (Open5GS). UERANSIM is the simulator of 5G UE and gNB. UERANSIM is open-source software available on GitHub and can be used to test the core and study the 5G system. In addition, it is compatible with Open5GS among the 5GS open projects [36]. Open5GS implements core according to 3GPP release 16, and it was developed in the C language. Open5GS is available on GitHub, offering 10 NFs as of September 22, 2022. Active development updates have been underway for the corresponding project since September 22 [37].

Experiments on 5G testbed. Using the constructed 5G testbed, an experiment for collecting the 5GC traffic was performed. First, the packet was dumped with the UE, and the GTP packet that comprised GPRS Tunneling was captured as it passed through the gNB-UPF (N3 interface). Thus, in the experiment, we transformed the IP-based packet of the wired network into the GTP-based packet of the 5G network. Then, to capture the corresponding packet, Linux tcpdump was used.

Feature extraction. We extracted GTP-based packets, which are 5GC traffic obtained from our experiments, and converted them to CSV for use in the ML model. Table 1 summarizes information about all 57 features used in this experiment. The features are extracted based on the six network protocols used in the collected dataset. The six protocols are Internet Control Message Protocol (ICMP), Transmission Control Protocol (TCP), User Datagram Protocol (UDP), Internet Group Management Protocol (IGMP), IP, and GTP. Wireshark's Tshark was used for feature extraction. The six protocols of the gathered packet were used as the criteria for extracting the packet according to the header. Considering the headers of these protocols, 93 pitchers were extracted, and the pitchers that did not have values were removed after checking the presence of values for all data from the extracted headers.

Fig. 4 is the workflow for this experiment, showing the flow and procedure. It had five procedures, and each procedure is explained as follows.

(1)~(2). After pre-processing the dataset, divide it into training-testing sets.

(3). Configure the training dataset to create three comparison models (Baseline, SMOTE, and GAN-CTGAN).

(3)-a. Baseline is used as the training set.

(3)-b. SMOTE is used for training by oversampling in the training set. The reason for excluding the testing set from oversampling is that the sample data replicated by SMOTE is not included in the testing set for model performance verification.

(3)-c. GAN-CTGAN is used for training by augmenting malicious data by utilizing the training-testing set and including the synthesized malicious data in the training set. The reason for using the testing set for malicious data augmentation is that we have a small dataset to experiment with, so we want to utilize as much data as possible to generate plausible synthetic malicious data.

(4). Train the ML model using the training set for the three comparison models.

(5). For the trained ML model, validation and test results are derived depending on the presence or absence of the testing set.

This time, we describe the procedures in the workflow in more detail, such as the methods used, data counts, and algorithms. Fig. 5 showed five detailed experimental procedures: (1) pre-processing, (2) splitting the training and testing sets, (3) three comparative models (Baseline, SMOTE, and GAN-CTGAN), (4) binary and multi-classification, and (5) model evaluation.

Pre-processing. Pre-processing involves data cleaning and feature scaling. Data cleaning entailed the elimination of missing values by inserting the value 0 where no value existed. Additionally, we transformed hexadecimal numbers into decimal numbers and floats into integers. The feature scaling then went through standardization and normalization to bring the values to a consistent level.

Splitting the training and testing sets. To evaluate the performance of the ML training model in this experiment, we separate the training and testing sets. The total size of the dataset for the experiment was 65,000, and the kitsune dataset rate (7:3) for the (benign/malicious) ratio was maintained (45,500/19,500). Finally, the (benign/malicious) rate was maintained while the data were divided into the training and testing sets (52,000 and 13,000 data, respectively). To ensure that the ML model underwent effective training on the imbalanced class dataset, the training and testing set both maintained their (benign/malicious) ratios.

Three comparative models: Baseline, SMOTE, and GAN-CTGAN. This study analyzed and compared the performance of the three models. The models can be distinguished by their ML model training methods. First, the ML model was trained with the Baseline training set (52,000). Second, SMOTE was oversampled in the training set (52,000), and the ML model was trained. Lastly, the 26,005 attack data were augmented for GAN-CTGAN using the seven types of attack data, with 3,715 data per session. Thus, we added 26,005 data were added via augmentation to the original training set (52,000) and used 78,005 data to train the ML model.

Binary and multi-classification. The kitsune dataset was used with multiple attacks to experiment on binary and multi-classification. First, the data were divided into benign (label 0) and maligned (label 1) for binary classification. Subsequently, multi-classification was performed by dividing the data into benign (label 0) and seven types of attacks (labels 1–7). In addition, ML and the ensemble of machine learning (EML) were used as classifiers. First, ML classifiers were used for SVM, k-nearest neighbor (KNN), and decision tree (DT). Then, EML classifiers were used for random forest (RF), AdaBoost, and XGBoost.

Model evaluation. The stratified k-fold (cv = 5) was used to train the model and evaluate the performance because the dataset for training the ML model was insufficient and had imbalanced classes. In addition, these performed fivefold cross-validation and then generated five trained models. Finally, we utilized the validation set for the five trained models to get five validation results and averaged them to get the final validation result. We also got the same test results.

The confusion matrices used in this study were accuracy and f1-score. Accuracy intuitively refers to model prediction performance. Its definition is expressed as Eq. (1). However, it is unsuitable for use as a metric for imbalanced datasets. For this study, we also selected the f1-score, as it is highly prone to data bias due to the imbalanced classification problem. F1-score is the harmonic mean of recall and precision. Recall is a metric showing the percentage of positives correctly classified as true positives. Its definition is expressed as Eq. (2). precision is a metric showing the percentage of negatives correctly classified as true negatives. Its definition is expressed as Eq. (3). Therefore, the f1-score can cancel out differences even if the ratio of positives to negatives is imbalanced. Its definition is expressed as Eq. (4).

(1) Accuracy=TP+TNTP+TN+FP+FN

(2) Precision=TPTP+FP

(3) Recall=TPTP+FN

(4) F1−score=2×(Precision×Recall)Precision+Recall

A model that does not use either SMOTE as the Baseline oversampling technique or GAN-CTGAN as the data augmentation technique was referenced for comparison. We compared the Baseline’s binary and multi-classification performance results to define and prove the problem. Table 2 shows the validation results, test results, and two value differences for binary and multi-classification. In addition, the results are shown as accuracy and f1-score of six ML classification models. For binary classification, the accuracy and f1-score decreased. The accuracy had a minimum reduction of 5.95% (DT) and a maximum value of 14.89% (AdaBoost), and the f1-score had a minimum value of 16.08% (DT) and a maximum value of 54.72% (SVM). For multi-classification, the accuracy increased, with a range of 1.74–3.51%, excluding KNN and AdaBoost. Additionally, the f1-score increased, with a minimum value of 0.06% (RF) and a maximum value of 3.17% (DT), excluding KNN.

The kitsune dataset, which consisted of data relating to seven IoT DDoS attacks, did not experience generalization errors, as the performance of the validation and test results were not significantly different for multi-classification. However, a significant performance difference between the validation and test results occurred in the case of binary classification, and a generalization error occurred because of overfitting. Therefore, we analyzed the training methods for the ML model to reduce the generalization error for the binary classification of the kitsune dataset.

The method of presenting the binary classification results of the three comparison models confirms and compares the trend of the results from the perspective of validation and test results. Fig. 6 shows the trend of accuracy and f1-score for the six classifiers. The three comparison models’ validation results (blue) and test results (orange) are shown for each classifier. Regarding validation results, SMOTE showed increased performance in both accuracy and f1-score. Regarding test results, SMOTE can confirm the improved performance in f1-score and GAN-CTGAN in accuracy. Table 3 shows the exact values of accuracy and f1-score as shown in Fig. 6. The following compares the results for each of the three comparison models regarding verification and test results for the values in Table 3.

Performance comparison for validation result. The accuracy for the Baseline validation result was between 85.47% and 79.86%, and the f1-score was between 71.88% and 92.38%. The accuracy for the SMOTE validation result was between 95.02% and 97.62%, excluding SVM (62.42%), and the f1-score was between 75.36% and 97.78%. The accuracy for the GAN-CTGAN validation result was between 80.31% and 85.65%, and the f1-score was between 71.85% and 83.70%.

Compared with the Baseline, the accuracy and f1-score tended to be higher for SMOTE (excluding SVM). The lowest accuracy was 4.92% (DT), and the highest was 9.07% (KNN). The lowest f1-score was 14.4% (XGBoost), and the highest was 23.48% (KNN). Additionally, compared with the Baseline, GAN-CTGAN tended to have lower accuracy. The lowest accuracy was 1.5% (KNN), and the highest was 8.47% (AdaBoost). Compared with the Baseline, the f1-score was reduced for SVM, AdaBoost, and XGBoost and increased for KNN, DT, and RF. From the perspective of the validation result, the oversampling technique using SMOTE exhibited higher ML training model performance than the data augmentation using GAN-CTGAN.

Performance comparison for the test result. The Baseline test result had accuracies between 70.65% and 84.22% and f1-scores between 43.37% and 63.10%, excluding SVM (22.39%). The SMOTE test result accuracy and f1-score were 77.79–81.99% and 53.77–68.46%, respectively, excluding SVM (47.64%). Lastly, the test result accuracy of GAN-CTGAN was between 71.69% and 85.39%, and the f1-score was between 42.37% and 62.61%, excluding SVM (20.95%).

In the case of SMOTE, compared with the Baseline, the accuracy was reduced for SVM, KNN, and DT and increased for RF, AdaBoost, and XGBoost. The f1-score tended to increase; the lowest value was 3.64% (XGBoost), and the highest value was 25.25% (SVM). For GAN-CTGAN, compared with the Baseline, the accuracy increased, and the F1-score tended to decrease, excluding RF. The lowest accuracy was 0% (KNN), and the highest was 3.16% (RF). The lowest f1-score was 0.21% (KNN), and the highest was 2.18% (XGBoost), excluding RF (2.35%). Therefore, from the perspective of the test result, the f1-score indicated improved performance for oversampling using SMOTE compared with data augmentation using GAN-CTGAN. However, relative to oversampling with the three classifier accuracies reduced, data augmentation generally exhibited superior performance for the six classifiers.

This section examines the learning curve for the three comparative models. Fig. 7 shows the learning curves for the three comparative models for SVM and RF. The scikit-learn learning curves were used, and the cross-validation value was 10. The graph shows the accuracy for the training (red) and validation (green) sets for the size of the training data sample. The estimated variance was expressed by deriving the standard deviation of the average accuracy for each accuracy and utilizing the fill_between function. In the Baseline, both SVM and RF showed significant differences in accuracy and estimated variance between training and validation. Therefore, the Baseline is likely to be overfitted and exhibit generalization errors.

For SVM, the SMOTE decreased training and validation accuracy and estimated variance compared to the Baseline. The GAN-CTGAN increased training and validation accuracy and decreased estimation variance compared to the Baseline. Compared with that Baseline, the SMOTE and GAN-CTGAN training accuracies were the same for RF, but the validation accuracy was higher, and the estimated variance was lower. Therefore, the estimated variance was reduced for both SMOTE and GAN-CTGAN, with the possibility of the generalization error having decreased by reducing the overfitting.

The difference between the validation and test results is checked to assess the generalization error regarding the three comparison models. Fig. 8 consists of two graphs of accuracy and f1-score, which show the difference trend between the validation and test results for each of the six classifiers. We can also see in Fig. 8 that the model with the smallest value difference among the three comparison models mitigates the generalization error problem. Table 4 shows the exact value of the difference between the accuracy and f1-score value expressed in Fig. 8. The following is a comparison of the results for each of the three comparison models for the values in Table 4. SMOTE exhibited a performance difference: the accuracy compared with the Baseline increased to a minimum of 1.51% (AdaBoost) and a maximum of 10.02% (KNN). The performance difference was also evident from the f1-score, which increased to a minimum of 6.63% (AdaBoost) and a maximum of 12.44% (RF), excluding SVM (31.99% reduction). Additionally, a performance difference arose where the f1-score decreased to a minimum of 0.54% (RF) and a maximum of 7.26% (XGBoost), excluding KNN (11.73% increase) and DT (2.55% increase). Therefore, from the validation and test results perspective, the generalization error tended to increase for SMOTE and decrease for GAN-CTGAN.

The SMOTE studies discussed in Section 2 aim to address the data imbalance problem. In addition, SMOTE is used not only in the field of network security related to this study but also in various research fields. Therefore, three studies introduced in Section 2 were selected and compared with the SMOTE experiment. The parts to be compared can be divided into four categories in Table 6. (1) Dataset, (2) ML/DL models, (3) SMOTE description (research field/purpose/other tech.), and (4) Results. Of these categories, the main category is (3) SMOTE description. Because the research fields using SMOTE and the techniques used together differ for each corresponding category, this was analyzed and compared with a focus on these techniques. The purpose of this comparison is not to compare the performance, as the datasets are different, but to analyze the differences between the previous study and this study and the information related to SMOTE to identify the meaning, differences, or limitations of the experimental results.

As a result of the comparison, recent SMOTE studies use both ML and DL, but it is confirmed that SMOTE performs well in DL. In addition, other techniques (GMM, Encoder, etc.) are used together rather than using SMOTE alone. As such, this SMOTE experiment utilized SMOTE only on ML, not DL, and did not combine it with other techniques. Compared to GAN-CTGAN, it was confirmed that the SMOTE experimental results were not mitigated regarding the generalization error. Therefore, the comparison with previous studies indicated that it is necessary to confirm the experimental results by using DL and other techniques (GMM, Encoder, etc.) rather than the SMOTE application method of this study.

The GAN research discussed in Section 2 is from the field of network security, which is related to this research. This research used GANs to solve various problems, not just the data imbalance issue. Among the studies introduced in Section 2, three were selected for comparison with ours. The purpose of this comparison is not to compare the performance of the datasets but to analyze the differences between the previous study and this study and the GAN-related information to identify the experimental results’ meaning, differences, or limitations. The parts to be compared can be divided into the four categories in Table 7: (1) Dataset, (2) ML/DL models, (3) GAN description (type/purpose/method), and (4) Results. In particular, compared to previous studies, the main category is (3) GAN description (type/purpose/method). The type, purpose, and method of GAN used for each study in this category differ. Focusing on these features, we analyzed and compared previous research.

The comparison revealed that recent GAN research mainly focused on the field of network security, and in particular, GAN is used in various ways as well as data augmentation for data imbalance in this study. Our work was impeded by the availability of insufficient 5GC traffic in tabular form to train ML models. Therefore, CTGAN, which has been proven to generate learning data from network data in tabular form effectively, was used in the experiment [4,30,31]. In addition, owing to network traffic, the IoT DDoS attack was augmented using CTGAN and then included in the training set to train the ML model. Consequently, two perspectives, the evaluation index values and generalization errors were examined as the results of the experiment. The experimental results confirmed that CTGAN did not change significantly in the evaluation index values compared to Baseline, but the generalization error tended to be alleviated by reducing overfitting. Our investigation confirmed that various methods and applications based on GAN had been developed, as well as data augmentation methods in the field of network security.