Big data technology extracts knowledge and values from data by statistical analysis. Big data has three features: 1) objectivity, 2) accuracy, and 3) testability. Big data-based methods have become an important approach to network security. Statistical algorithms are typical big data-driven methods. The methods represent a well-established approach to anomaly detection. This type of anomaly detection identifies DDoS attacks by calculating thresholds to flag unusual behaviors. Analyzing daily traffic distributions can pinpoint activities that significantly deviate from the norm as potential anomalies. Hamdi and Boudriga [5] introduced a novel method for the statistical analysis of DDoS attacks utilizing wavelet analysis techniques. By transforming traffic data into the frequency domain through wavelet transformation, they were able to identify specific patterns and features associated with DDoS attacks. This method effectively detects and characterizes these malicious activities by extracting frequency-domain features and incorporating statistical analysis. Tao and Yu [6] proposed a DDoS attack detection method based on information entropy within local area network environments. Under typical conditions, the IP entropy tends to be relatively high. However, during a DDoS attack, the volume of packets directed at the target IP increases significantly, leading to a decrease in IP entropy. This reduction serves as an early warning sign of potential DDoS activity. Fortunati et al. [7] proposed an enhanced anomaly detection method based on covariance analysis. This approach involves constructing a covariance matrix from network traffic data to establish a normal distribution profile. By analyzing this covariance matrix, the method can identify deviations from the expected traffic patterns. Abnormal traffic is detected by setting specific thresholds, allowing for the effective identification of anomalies indicative of potential DDoS attacks or other malicious activities.

Tensors are multidimensional arrays. Tensor models with strong expressive ability can mine abundant intrinsic information contained in massive data, and it is a promising method to solve security problems in a big data environment. In terms of the multimodal data problem in large-scale networks, researchers have presented a considerable amount of work based on tensor models for anomaly detection. In [8], a novel approach is proposed for efficient tracking of intrusion in the normal subspace arising from the decomposition of the Parallel Factor Analysis tensor. The method is based on the extraction of a normal subspace obtained by the tensor decomposition technique, considering the correlation between different metrics. Aiming to address dynamic detection issues, some researchers have proposed a series of online detection methods. In [9], the network data is first represented as a unified tensor. Then, an incremental tensor decomposition is proposed for tensor data dimensionality reduction and denoising. In the end, by combining machine learning algorithms, intrusion detection is completed. The work [10] presents an online anomaly detection system capable of handling operational network traffic of large networks.

In the real world, there is a huge amount of multi-source and heterogeneous data in cyberspace, including network traffic data, network topological structure data, time-related information, and so on [11]. This is the basics of designing a big data-driven intrusion detection system. The data sensing plane is the collecting plane of the intrusion detection system. Some data stream collection tools (Sniffer, NetFlow, probe, and Flow tools) are embedded in the switches, which are a set of distributed monitors. The monitors capture the network data day and night. These data are source IP, port, destination IP, protocol type, data packet length, the number of bytes of traffic between two hosts in a fixed time, the number of data streams, flow entropy, etc. These data have various sources and formats, and the network traffic flow is related to multiple factors. For example, it is influenced by the time. From 9:00 PM to 11:00 PM, the network traffic volume exceeds typical baseline levels, but at 2:00 AM, it is low. Also, the network traffic is influenced by weather. Hence, a holistic integration of these multi-source datasets is essential for robust network security analysis. How to fuse these data and the relationships between them is a huge challenge [12]. To cope with the problem, we use the network unified tensor (NUT) to fuse the multi-source and heterogeneous data as Fig. 2. Firstly, these data are represented as local tensors in the data collection layer. With the transmission plane, distributed local tensors are propagated to the big data processing plane, where they undergo tensor fusion operations to form a unified global tensor representation.

From the viewpoint of big data, how to model these multi-source heterogeneous data is a big challenge [13]. The data models are always complex and need powerful computing methods [9]. Considering the tensor’s multi-model features, in this work, we use the tensor to fuse the data in Section 4.2. Aiming at the data model, a novel tensor decomposition method is proposed. Specifically, the tensor decomposition method effectively reduces noise in network data and performs feature cutting on each mode through the Minimum Description Length Principle (MDLP) rule. Different from previous tensor decomposition algorithms such as HOSVD(Higher-Order Singular Value Decomposition), this algorithm further integrates features at the feature level through tensor mode multiplication, seamlessly achieving feature dimension reduction and denoising.

1) Definition: Inspired by the insights from [14], we developed a multi-modal feature extraction method (HOBISVD) shown in Fig. 3. HOBISVD is a variation of the Tucker-2 decomposition. It decomposes a tensor into a core tensor multiplied (or transformed) by a matrix along each mode except the first mode. The HOBISVD of a third-order tensor sets the first-factor matrices as the identity matrix. For instance, a HOBISVD can be described as Eq. (6): (6)𝒳=𝒢×2B×3C=[[𝒢:I,B,C]],where 𝒢∈RI×Q×R with R = K and C=I, the K×K identity matrix. These concepts extend easily to the N-way case, we can set any subset of the factor matrices to the identity matrix.

Feature truncation is a method to optimize the obtained decomposition, thus obtaining the high-value principal components of each modality, and the result of the truncated decomposition is the approximate solution. In the work, the feature cutting on each mode is based on Minimum Description Length Principle (MDLP) rule. The truncated HOBISVD decomposition is a form of high-order PCA. Thus, in the three-way case where 𝒳∈RI×J×K, the Eq. (7) is got, (7)𝒳≈𝒢×2B×3C=∑q=1Q∑r=1Rg:,qr∘bq∘cr,here B∈RJ×Q, and C∈RK×R are truncated factor matrices in each mode, and their column vectors are orthogonal, which are the principal components (PC) related various mode. The tensor 𝒢∈RI×Q×R is the core tensor.

Assuming that 𝒳∈RI×J×K, the tensor decomposition can be formulated as an optimization problem as Eq. (8): (8)min⏟𝒢,B,C‖𝒳−[[𝒢;B,C]]‖F2,subject to 𝒢∈RI×J×K,and B∈RJ×J, C∈RK×K are columnwise orthogonal.

Next, HOBISVD is described in detail. HOBISVD involves three key stages, i) the tensor is unfolded in two modes I2 and I3; ii) perform orthogonality-constrained matrix factorization on the unfolded matrix to obtain the eigenvector for each independent modality; iii) by using tensor N-model multiplication, feature-level fusion and dimensionality reduction are performed on two modes to obtain a serious of feature tensors.

2) Unfolding of Tensor 𝒳: Unfolding a tensor 𝒳∈RI×J×K in two modes J and K. Thus, two unfolded matrix X2, 𝒳3 can be got as follows: (9)X2∈RJ×(K⋅I), (10)X3∈RK×(I⋅J).

Fig. 4 illustrates the matrixization process of a 3th-order tensor 𝒳∈Rn1×n2×n3 in two modes, in which A(1)∈Rn1×(n2×n3), X(2)∈Rn2×(n3×n1) are the matrices of the two modes.

3) Applying Orthogonality-Constrained Matrix Factorization on the Unfolded Matrix A2 and A3: To achieve the optimization target illustrated in the Eq. (11) and ensure the resulting space remains orthogonal, (11)min⏟𝒢,B,C‖𝒳−[[𝒢;B,C]]‖F2,various methods have been devised by researchers. This article employs the singular value decomposition approach for this purpose. The singular value decomposition enables to calculation of the eigenvalues associated with each mode, which in turn allows for the determination of the optimal truncation strategy for each mode based on these eigenvalues, which will be described in the following. We apply orthogonality-constrained matrix factorization on the two unfolded tensors A2. Three new matrix U2,Σ2,V2 are got as Eq. (12): (12)X2=U2×Σ2×V2=∑jJσj∘u2j∘v2j=∑jr2σj∘u2j∘v2j+∑r2+1Jσj∘u2j∘v2j.

Similarly, the same method is applied to matrix A3, resulting in three matrices U3,Σ3,V3 as Eq. (13): (13)X3=U3×Σ3×V3=∑kKσk∘u3k∘v3k=∑kr3σk∘u3k∘v3k+∑r2+1Jσk∘u3k∘v3k.

Fig. 5 presents an example of orthogonal matrix decomposition applied to an unfolded matrix derived from a 3rd-order tensor. In the context of tensor dimensionality reduction, several parameters need to be specified. Among these, r2 and r3 are particularly important. These parameters represent the number of leading singular vectors that are retained in the tensor bases U2 and U3. The values of r2 and r3 directly influence the final dimensionality of the eigentensor S, which is a crucial component in the tensor decomposition process. The selection of the eigenvector parameters r2 and r3 is based on retaining a specified portion of the original data from tensors X2 and 𝒳3. In the following, the Minimum Description Length Principle (MDLP) based truncated method will be described [15].

4) MDLP Based Truncated Method: Enhancing data quality without significantly compromising its inherent characteristics is essential for effective noise reduction. We opt for the Minimum Description Length Principle (MDLP) as our model order selection (MOS) strategy, which aids in ascertaining the necessary rank for Singular Value Decomposition (SVD) based noise elimination. The MDLP is encapsulated by the following principle: (14)MDL(k)=−log(∏i=k+1pσi1/(p−k)1p−k∑i=k+1pσi)(p−k)N+12k(2p−k)logN,here σi are eigenvalues which are from SVD and i∈{1,2,⋯,p}. The eigenvalues are convergent and sorted in descending order, namely σ1>σ2>⋯>σp. p denotes the number of eigenvalues, and N denotes the sample’s number in the dataset. When MDL(k) gets the minimum value as Eq. (14), the best rank of the matrix A can be ascertained by Eq. (15): (15)rank(A)=argmin(MDL(k))+1,where k∈{0,1,⋯,p−1}. Indeed, as the variable k commences at 0, the aforementioned equation must incorporate an additional 1. Subsequently, only the initial rank(A) eigenvalue is preserved, while all remaining eigenvalues are assigned a value of 0, denoted as σ1>σ2>⋯>σrank(A) and σrank(A)+1=⋯=σp=0.

We construct a canonical sinusoidal signal y=sin(x) as a baseline waveform and superimpose zero-mean Gaussian white noise onto the pristine signal. We then apply singular value decomposition (SVD) for noise reduction. Fig. 6 illustrates the progression of singular values as a rank function. Numerical analysis demonstrates that when the matrix rank satisfies rank≥2, the singular values are diminutive when rank≥2. The data encompassed within the eigenvectors associated with these singular values is noise-related. Hence, we maintain the initial three singular values.

5) Tensor Multiplication Based Feature-Level Fusion and Dimensionality Reduction: According to the above step, we have obtained the two reduced U2, and U3 in I2 mode and I3 mode, which are directly isolated from each other. How to integrate the information of the two modalities is a big challenge. To address this issue, the tensor N-mode multiplication between tensors and matrices is proposed to solve the problem, as shown in Eq. (16). First, the characteristic information on modality 2 is integrated, and then the feature information on the integrated modality is fused. Since a principal component truncation operation is used, dimensionality reduction and denoising are also achieved in the process. (16)S=X×2(U2r2)T×3(U3r3)T,where X is the initial tensor, (U2r2)T is the transpose of the r2-dimensionally reduced U2 tensor, (U3r3)T is the transpose of the r3-dimensionally reduced U3 tensor. The Algorithm 1 shows the process of HOBISVD. In nature, HOBISVD is a mathematical method used for tensor decomposition in high-order dimensions. It can decompose a high-dimensional data tensor into the product of multiple low-rank matrices, thereby extracting important features from the data. Essentially, the method can also be applied to deep learning to reduce dimensionality and extract useful information from input data. By performing HOBISVD decomposition on input features, we can obtain feature subspaces at different levels and directions, each containing strongly correlated features to some extent in the original data.

6) XGBoost Classification Method: Aiming to deal with large-scale data, XGBoost (eXtreme Gradient Boosting) was proposed to improve the stability and accuracy of the large-scale model. It is an ensemble learning algorithm based on gradient-boosted trees (GBT). The key to XGBoost’s success lies in its adaptability across diverse scenarios. XGBoost is a powerful and versatile machine-learning system known for its tree-boosting capabilities. Its influence has been acknowledged in various machine learning and data mining competitions. With tensor mode multiplication, feature-level fusion and dimensionality reduction are performed on two modes to obtain a series of feature tensors. In nature, it is a large-scale feature dataset. Aiming at the feature dataset. Aiming at the series of feature tensors, the XGBoost method is used for DDoS detection through classification. The input is the serious of feature tensors [S(1,,),S(2,,),⋯,S(I,,)], the output is Eq. (17): (17)Y^i=∑k=1Kfk(S(i,,)),fk∈F.

The object is Eq. (18): (18)Obj=∑i=1Il(Y^i,Yi)+∑k=1KΩ(fk).

The primary objective of XGBoost is to push the boundaries of machine learning limitations to offer scalable, portable, and precise solutions. When dealing with huge network data, the distributed version of XGBoost demonstrates exceptional portability, effectively addressing the challenges associated with large-scale datasets [16].

Dataset CICDDOS 2019, curated by the Canadian Institute for Cyber Security (CIC), includes a vast array of network data with 87 features and millions of traffic instances, encompassing various types of DDoS attacks [17]. In our experimental setup, we constructed a representative subset by randomly sampling 40,000 data points from the original dataset, with a stratified distribution of 32,000 normal traffic instances and 8000 malicious attack samples, as specified in Table 1. Additionally, we eliminated several features that were deemed insignificant for the classification process from the initial set of 87. Consequently, we narrowed it down to 64 key features, as outlined in Table 2. The approximate distribution of the used data is shown in Fig. 8.

According to the confusion matrix, we use TP, FP, TN, and FN to represent True Positive, False Positive, True Negative, and False Negative.

In this subsection, a series of comparative experiments will be carried out. Focusing on the CICDDoS2019 dataset detailed in Table 1, which comprises 64 features, the data matrix is denoted as X∈RM×N, where M represents the total number of data points and N=64 signifies the number of features. Within the context of the HOBISVD-based denoising algorithm, the matrix X is transformed into a three-way tensor 𝒳∈RM×N1×N2, with N1=N2=8 and N=N1×N2.