Also called entropy-based approaches, entropy is the measure of the randomness in a dataset. Since every feature of normal network traffic has a special distribution pattern, for example a balance between the count of source and destination IP addresses in normal flow. This entropy pattern will deviate in attack flows since the number of destination IP addresses greatly increases than the number of source IP addresses. Despite this approach is characterized by a fast response and a low computation overhead of processing a large traffic volume. Its detection accuracy is greatly affected by the proper selection of the threshold value for a certain traffic feature to reduce the ratios of false positive and false negative.

In [19] authors proposed a hybrid system for attack detection that merges both entropy and traffic volume characteristics, which offers good results than using each technique alone. Kalkan et al. [20] introduced a joint entropy-based scoring system (JESS) to defend attacks in SDN environments, by utilizing Joint entropy they can defend even unfamiliar attacks efficiently. Lima et al. [21] suggested a system based on statistical analysis of traffic entropy in SDN environments.

Wang et al. [22] proposed a flow statistics process in SDN switches, then performed lightweight entropy-based detection model executed in the edge switches to reduce the communication overhead between the data plane and controller plane. Ahmed et al. [23] introduced a new structure called application fingerprints to express packet attributes and traffic flow level statistics to differentiate legitimate packets from attack packets. However, this approach is not proper for online systems, since some flow attributes such total bytes, number of packets between source and destination and flow duration cannot compute their statistics while gathering them. In [24] authors introduced new hybrid approaches where flow level statistics or entropy based are combined with some techniques of Machine Learning (ML) or Artificial Neural Networks (ANN) to overcome some limitations of flow statistics approaches. The ML and ANN mechanisms will be introduced in detail in the next subsection.

Buczak et al. [25] introduced summaries for different methods of Machine Learning (ML) and Data Mining (DM) used for attack detection, such Support Vector Machine (SVM), k-Nearest Neighbor (k-NN), Random Forest (RF), etc. over recent years, with the abundance of real network traffic datasets. These methods perform better results in traffic classification field.

He et al. [26] suggested a new source side (active defense) machine learning technique instead of defending attacks at destination side (passive defense) preventing malicious network flows from being sent outside the attacking network. This showed a high accuracy and low false positive rates. Hoon et al. [27] introduced the concept of dataset feature selection which reduces the feature engineering processes and increasing classification accuracy. Few papers addressed the implementation of Deep Learning (DL) approaches in attack detection; authors in [28] combined the entropy-based techniques with DL methods to easily control the problem of setting accurate threshold. Conducted experiments showed a high accuracy. Yin et al. [29] proposed a deep learning attack detection model deploying a Recurrent Neural Network (RNN) to perform binary and multiclass classification. They compared its performance with a set of known Machine Learning (ML) models such SVM, Random Forest (RF) and Artificial Neural Networks (ANN). Their proposed model showed a higher performance from point of accurate classification. Wu et al. [30] introduced a multiclass Convolutional Neural Network (CNN) intrusion detection system, deploying CNN to select highly related features from the massive data gathered to improve the classification accuracy and reduce computation overload. Over traditional ML algorithms, CNN showed better performance. Kwon et al. [31] studied the performance of three different models of CNN (shallow, moderate and deep) to check the effect of depth of the CNN to the detection performance. The models are verified using two datasets NSL-KDD [32] and MAWILab [33] showing that shallow CNN model with a single convolutional layer and single maximum pooling layer performed best.

Authors in [34] converted the suspicious traffic traces into arrays that contain flow features by combining both CNNs and RNNs, testing their model on ISCX2012 dataset [35] showing good results from point of reducing classification error from 7.517% to 2.103% relative to conventional machine learning algorithms. Despite the efficiency of deep learning algorithms for detecting DDoS attacks, they are time-consuming, resource-intensive, and require large numbers of model parameters to be trained.

This layer detects any suspicious flows and raises an alarm to the control layer for accurately classifying whether it is an actual DDoS attack or legitimate burst traffic then carrying out proper response. This way the load on CPU Controller and the traffic overload through southbound interface may decrease drastically, it is designed as two modules:

This layer is composed of two modules, Signature Based Detection and Deep Learning Classification.

The framework is evaluated using three datasets, NSL-KDD and CSE-CIC-IDS2018 are the most traditional network’s datasets commonly used, where the third is SDN specific dataset. First, NSL-KDD dataset contains 4,898,430 records each of 41 features. Feature no. 42 is the records’ labels, which may be Normal or attack; attack records are categorized into 4 types DoS, Probe, R2L or U2R.

Despite its simplicity, NSL-KDD dataset is not the ideal representation of the actual network model, so CIC-IDS2018 dataset is used to accurately evaluate any IDS since it represents real attacks. It has 83 features, of 2,830,540 distinct records, and categorized in the label field to 15 classes.

Although the characteristics of CIC-IDS2018 dataset have some weaknesses, first with this huge number of records (3,119,345) and 83 features each, enormous loading and processing overhead is required. Second, it contains some missing data. The last demerit is class imbalance problem [38], in which different attacking classes do not have equal number of instances, some are represented by a large number (BENIGN = 2,359,087 instances) others have few number (HeartBleed = 11 instances). So, when using this dataset for training classifiers or detectors it will make the classifiers biasing toward the majority class [38] degrading the classifier’s accuracy with a higher false ratio. Alleviating methods for these shortcomings will be introduced in the Data preprocessing subsection.

Since the architecture of traditional network differs from that of SDN, which results in a major difference in the feature sets of the data gathered from both. Mininet emulator [39] software has been used to create a realistic virtual network, 23 features in the designed SDN topology with the total number and different categories used in following experiments.

Network collected dataset may include some error values like duplicate, infinity, missing or categorical data, this may cause classification problems. These error values should be eliminated or mitigated before training and testing phase. Duplicate records should be removed, deleting records having outlier’s values. Techniques like one-hot encoder are used to convert categorical data into numeric values.

Feature scaling methods (Normalization and Standardization) [40]: features having values of varying degrees of magnitude, may hurdle the performance of some machine learning algorithms especially those types using gradient descent as optimization techniques. So, these scaling methods may be used to scale their values between 0 and 1, or +a and –a.

Data imbalance reduction [41]: where some features are highly underrepresented, causing the classifier to bias towards the majority features. Many techniques are designed to handle class imbalance problem, one of them deployed in this paper is class relabeling. By either splitting the majority classes into more classes or merging some minority classes to form one class.

Evaluating the trained model’s performance can be done using the confusion matrix and advanced evaluation metrics are shown in Fig. 5.

True Positive (TP): denotes the no. of positive class correctly judged as positive. False Negative (FN): denotes the no. of positive class mistakenly judged as negative. False Positive (FP): denotes the no. of negative class mistakenly judged as positive. True Negative (TN): denotes the no. of negative class correctly judged as negative.

Three experiments have been conducted to validate the performance of the (HDLIDP) framework.

Experiment 1: (NSL-KDD dataset): using a two hidden layers tuned ANN classifier.

Tab. 4 shows the advanced metrics obtained from the confusion matrix shown in Tab. 3. The average results of confusion matrix when utilizing two hidden layers are 97.903, 91.690, 98.271 and 94.798 for Accuracy, Precision, Recall and F1 Score, respectively.

Tab. 5 and Figs. 6 and 7 show the parameters of the ANN classification in case of single, two and three hidden layers compared with the Genetic Algorithm (GA) and Difficult Set Sampling Technique (DSSTE) algorithm. The DSSTE algorithm employs both Edited Nearest Neighbor (ENN) and K-Means clustering algorithms to reduce the data set’s majority class for improving the classifier’s training stage consequently enhances performance. The results show, using two hidden layers NN each contains maximum of 10 neurons provides best performance and approximately good time.

Experiment 2: (CIC-IDS2018 dataset): using a two hidden layers tuned ANN classifier.

Tabs. 6 and 7 show the average results of confusion matrix as 99.849, 93.333, 99.761 and 96.325 for Accuracy, Precision, Recall and F1 Score, respectively.

Since the framework has been designed to be deployed in real time world, so both performance accuracy and running time should be highly improved. From Tab. 8 and Figs. 8 and 9, show that the best accuracy is obtained when using two hidden layers NN, while the best running time obtained with a single hidden layer NN. According to the results obtained, it is advised to deploy the framework with two hidden layers NN as it has best accuracy and approximately good running time.

In the next experiment, the proposed framework is evaluated using SDN dataset collected from the Mininet emulator and comparing its results with those of the framework introduced in [43].

Experiment 3: (SDN dataset): the experiment has been done on a two hidden layers ANN classifier.

Tabs. 9 and 10 shows the average values of Accuracy, Precision, Recall and F1 Score.

Tab. 11 and Fig. 10 show a comparison in case of one, two and three hidden layers with the GA and Automated DDoS attack detection in SDN [43] framework.