Matrix factorization (decomposition) is the process of breaking down a matrix into the product of several matrices. This can involve techniques such as triangular factorization, full rank factorization, orthogonal triangle decomposition, Jordan decomposition, and Singular Value Decomposition. In our case, we primarily utilize Singular Value Decomposition to process data. Singular Value Decomposition allows for the representation of a relatively complex matrix as the product of smaller and simpler matrices. These smaller matrices describe the essential characteristics of the original matrix. Singular Value Decomposition is applicable to any matrix, making it adaptable to the features of current attribute information. It finds applications in various fields such as signal processing, statistics, natural language processing, and more. In recommendation systems, Singular Value Decomposition is widely applied in collaborative filtering and matrix completion algorithms. Through Singular Value Decomposition, a user-item rating matrix can be reduced to a low-dimensional latent factor matrix, extracting latent features of users and items for recommendation purposes.

Singular Value Decomposition has significant potential value in processing raw data. The singular values decrease exponentially with rank, with early singular values much larger than later ones, as shown in Fig. 1. By applying low-rank Singular Value Decomposition to the data, we can capture the essential features of the data, enhance data representation, and improve the performance of multi-view clustering. Based on this, we have optimized and improved the Singular Value Decomposition method to be suitable for large datasets, providing higher computational efficiency and scalability.

In recent years, contrastive learning has achieved remarkable success in the fields of images [20–23] and graphics [24–26], inspiring extensive research on contrastive deep graph clustering methods [12–16]. The clustering performance of these methods is primarily influenced by three key factors: data augmentation, network architecture, and the handling of positive and negative sample pairs. Taking these factors into account, we summarize the distinctions between our proposed ECN-MF and other contrastive deep graph clustering methods.

In an undirected graph G={X,A}, let V={v1,v2,…,vN} be a set containing N nodes with K classes, and E be the set of edges. X∈RN×D represents the attribute matrix, and A∈RN×N represents the original adjacency matrix. The degree matrix is denoted as D=diag(d1,d2,…,dN)∈RN×N, where di=Σ(vi,vj)∈εaij. Normalizing the original adjacency matrix A to A¯∈RN×N is achieved by computing D(−1)(A+I), where I∈RN×N is the identity matrix. Table 1 summarizes these symbols.

Recent studies have demonstrated the significant effectiveness of Singular Value Decomposition in handling sparse matrices and dimensionality reduction. Inspired by their success, we introduce the Singular Value Decomposition algorithm, treating attribute information as an independent preprocessing step before training. This approach allows for the effective extraction of latent and essential information from the data while filtering out noise present in the attributes. The method is as follows:

Specifically, given a matrix A∈Rm×n, low-rank Singular Value Decomposition decomposes it into the product of three matrices, where m and n can be any integers: (1)A=USVT

Here, U∈Rm×r is the left singular vector matrix, containing the structural information of the original data, where r represents the rank of the low-rank. S∈Rm×r is the diagonal matrix containing singular values, typically arranged in descending order, representing the importance of the data. VT∈Rr×n is the right singular vector matrix, containing the feature information of the original data.

Singular Value Decomposition can be applied to decompose any matrix, making it adaptable to the characteristics of current attribute information. We will rewrite Eq. (1) as follows: (2)Xr=UrSrVrT

Here, Xr∈RN×D represents the low-rank attribute matrix, Ur is the low-rank left singular vector matrix, Sr is the low-rank diagonal matrix, and Vr is the low-rank right singular vector matrix, where VrT is the transpose of Vr.

To adapt to the Singular Value Decomposition in the case of large data, we partition the dataset X={x1,x2,…,xN} into M batches, where each batch batchi consists of B samples, and B is a positive integer, i.e.: (3)batchi={xi⋅B+1,xi⋅B+2,…,x(i+1)B}

Here, B represents the number of samples included in batchi.

The objective of centralization is to set the mean of the data in each batch to zero. Assuming batchi contains B samples with a mean of μi, centralization is applied to each batch batchi, resulting in the centered batch batchicentered. This process can be mathematically represented as follows: (4)batchicentered=batchi−μi={xi⋅B+1−μi,xi⋅B+2−μi,…,x(i+1)B−μi}

Here, xi⋅B+2 represents the 2-th sample in batchi, and μi is the mean of batchi. Performing Singular Value Decomposition on each batchicentered yields different low-rank matrices ui,si,vi : (5)ui∼i+r,si∼i+r,vi∼i+r=SVD(batchicentered)

Here, SVD represents the Singular Value Decomposition algorithm.

We concatenate all low-rank matrices ui,si,vi from each batchicentered to form matrices Ur,Sr,Vr: Ur={u1,u2,…,ui∗r} (6)Sr={s1,s2,…,si∗r} Vr={v1,v2,…,vi∗r}

We perform a descending order sorting on Sr and select the top r values. In other words, we individually sort the real numbers of singular values in descending order and choose the top r singular values. The purpose is to emphasize the significance of singular values, allowing the primary information of attribute matrix X to be represented by a reduced number of singular values or vectors. The aforementioned learning process can be articulated as follows: (7)Slow=descendingsort(Sr) Slow_r=TOP(Slow,r)

Next, we will obtain the corresponding vectors of Vr based on the indices selected indices of the singular values Slow_r and form Vlow_r according to Eq. (8): (8)Vlow_r=f(x)={vi,Slow_r,i match vi0,others

Finally, we reconstruct the low-rank attribute matrix Xr using Ur, Slow_r, Vlow_r according to Eq. (9). (9)Xr=UrSlow_rVlow_rT

In this section, we embed the nodes of both the original and enhanced data into a latent space and design a pseudo-siamese neural network with an encoder that shares the same architecture but has non-shared learnable parameters.

Residual connections allow information to propagate between network layers, aiding in mitigating the vanishing gradient problem, accelerating the training process, and enhancing model performance. In this work, the representations learned by the l-th layer of GCNs can be obtained through the following convolution operation: (10)Z(l)=∅(A¯Zl−1Wv1l−1+Zl−1) Zr(l)=∅(A¯Zrl−1Wv2l−1+Zrl−1)

Here, ∅ is the activation function of the fully connected layer, such as Relu [29] or the Sigmoid function. Zl−1 and Zrl−1 represent the embeddings of two views, and Wv1l−1 and Wv2l−1 are the weight matrices corresponding to the encoder’s l-th layer. Additionally, we denote Z(0) as the original data X and Zr(0) as the enhanced data Xr. As shown in Eq. (10), the representations Zl−1 and Zrl−1 will traverse the normalized adjacency matrix A¯ to obtain new representations Z(l) and Zr(l). It is important to note that the input to the first layer of GCNs consists of the original data X and the enhanced data Xr: (11)Z(1)=∅(A¯XWv11) Zr(1)=∅(A¯XrWv21)

We denote the embeddings output by the last layer of GCNs as Z(L) and ZrL.

To minimize redundancy in embeddings and effectively preserve more discriminative features, we initially optimize the embeddings between cross-view samples using Mean Squared Error (MSE) loss. This ensures that the learned representations are not influenced by irrelevant information, thereby guaranteeing the quality of the latent space for subsequent clustering tasks. The formula is as follows: (12)LMSE=1n∑i=1n(Z(L)−ZrL)2

Here, Z(L) and ZrL represent the embeddings of the last layer of the graph convolutional network.

Next, we merge the embeddings from the two views for each node as follows: (13)Z=12(Z(L)+ZrL)

Here, Z∈RN×d represents the node embeddings for clustering.

Finally, by inputting Z into a one-layer Multi-Layer Perceptron (MLP) with a softmax activation function, we convert it into a K-dimensional clustering space, where K represents the number of clusters. The learning process can be expressed as: (14)Q=softmax(Z)where Q∈RN×K represents the probability matrix indicating the probability of all N nodes belonging to K clusters. We can view Q as a probability distribution.

Following the acquisition of the clustering probability distribution Q, we refine the data representation by focusing on learning high-confidence data, aiming to strengthen the cohesion within the clustering. Specifically, we aim to reinforce the intra-cluster cohesion by emphasizing representations that are closer to the cluster centers. Therefore, we compute the target distribution P as follows: (15)Pij=Qij2/fj∑j′Qij′2/fj′

Here, fj=∑iQij is the soft clustering frequency. In the target distribution P, each assignment in Q is squared and normalized to give higher confidence to the assignments. This leads to the following objective function: (16)LKL(P||Q)=∑xlogP(x)Q(x)

Minimizing the KL divergence loss between the Q and P distributions enables the GCNs module to enhance its ability to learn representations for clustering tasks. This ensures that the data representations around cluster centers become more compact.

In order to aggregate samples within the same cluster while simultaneously separating them from samples in other clusters, we choose to periodically update the top τ probability samples from different clusters. This process enhances the cohesion of positive samples within clusters and improves the discriminative ability of sample pairs, i.e., (17)I(xi,j isamongthetopτelements)={1,if qi,jisamongthetopτelements0,otherwise

Here, I(xi,jisamongthetopτelements)=1 indicates that sample xi is most likely to belong to cluster cj, and the label for this sample is recorded as ykey. The number of key samples in each cluster is Nk, and the total number of key samples is C⋅Nk, enhancing the cohesion of positive samples within clusters.

We use the obtained pseudo-labels ykey to constrain the model’s results, i.e., (18)LBCE(ykey,y^)=−1C⋅Nk∑i=1C⋅Nk(yikeylog(y^i)+(1−yikey)log(1−y^i))

Here, ykey is the label of the key sample, and y^ is the model’s clustering result.

In summary, the overall loss calculation for ECN-MF is as follows: (19)L=αLMSE+βLKL+LBCE

Here, α and β are hyperparameters.

The detailed learning procedure of ECN-MF is shown in Algorithm 1.

We proposed ECN-MF, which was evaluated on six datasets, including CORA [12] CITESEER [12], European Air Traffic (EAT) [30], United States Air Traffic (UAT) [30], Amazon Photo (AMAP), and Amazon Computer (AMAC). Table 2 provides a brief description of these datasets.

All experimental results were obtained on a high-performance server equipped with an NVIDIA 3090 GPU, 64 GB RAM, and the PyTorch deep learning platform.

To demonstrate the superiority of our proposed ECN-MF, we compared ECN-MF with 13 base- lines. Specifically, a classification method, graphMAE2 [34], is considered. Five deep clustering methods, including AE [35], DEC [36] SSGC [37], SDCN [5], and SAGSC [38], utilize autoencoders for node encoding, followed by clustering on the learned embeddings. Two hard sample mining methods, GDCL [39], and ProGCL [40], are employed. Additionally, five deep graph clustering methods for comparison: MCGC [14], MVGRL [13], AFGRL [27], AutoSSL [41] and SCDGN [42], are incorporated. These methods are designed with contrastive strategies to enhance the discriminative capability of samples.

Table 3 reports the clustering performance of all compared methods on six benchmarks. From these results, we can derive four key observations: 1) Our ECN-MF outperforms other deep clustering methods, attributed to the benefits of contrastive learning in implicitly capturing supervisory information. 2) Compared to contrastive methods, our approach demonstrates superior performance, leveraging an information-enhanced clustering guidance module to better preserve the original data information in the embedded data. It ensures better retention of information and improves the discriminative capability of sample pairs by bringing samples from the same cluster closer and pushing away those from different clusters. 3) Our method achieves the best performance on CITESEER, showcasing the effectiveness of utilizing Singular Value Decomposition to capture latent important information of samples, particularly advantageous in handling sparse matrices. 4) Favorable results on AMAP and AMAC highlight the effectiveness of our batched low-rank singular value decomposition algorithm in handling large datasets. In summary, our method outperforms most others on six datasets with four metrics, validating the effectiveness of our proposed approach in addressing unreasonable data preprocessing and handling positive and negative sample pairs.

Firstly, we perform Singular Value Decomposition on attributes, and the time complexity of calculating the low-rank process to obtain the reconstructed attribute Xr in Fig. 2 is O(batch×BD×r), where r is the rank size, B is the batch size, and D is the dimension. Subsequently, during the training phase, we employ a two-layer residual graph convolutional neural network (GCN) model. The time complexity for computing the two views is O(2NDd+4Ndd), where d is the dimension of the graph convolutional neural network, N is the number of samples, and D is the dimension of the original samples. The time complexity for calculating the mean square error loss function is O(Nd), and N is the number of samples. The time complexity for high confidence selection is O(NK), where K is the number of clusters, and N is the number of samples. The time complexity for computing the target distribution is O(NK), where N is the number of samples. The time complexity for the KL divergence loss function is O(NK), and the time complexity for the cross-entropy loss function is O(NK), where N is the number of samples. Therefore, the overall time complexity of our algorithm is O(batch×BDr+NDd+Ndd+NK+Nd).

In this section, we first experimentally validate the effectiveness of our proposed data augmentation method and periodic update strategy, as shown in Table 4. For simplicity, we denote the Batch low-rank Singular Value Decomposition as B and the periodic update as P. Note that in order to replace the B operation, we use a mask to generate different views of the same node, with a mask rate of 0.5. “(w/o)B & P” represents not using the batch low-rank Singular Value Decomposition operation and periodic update, while B + P indicates the usage of both. Table 4 displays the convergence results after running 1000 epochs. Based on the observed results, we conclude that the performance would degrade without B and P, indicating that these two strategies contribute significantly to the performance improvement.

In Fig. 3, we visualize the accuracy throughout the entire training process until convergence using a line chart. The graph demonstrates that our model exhibits robust performance. Overall, the experimental results validate the effectiveness of B and P.

In this section, we will analyze the hyperparameters r, α, and β to demonstrate their impact on the dataset. For the rank r, we set the range of values from 10 to 100, and for α and β, we set the range of values from 0.1 to 1.

To visually showcase the superiority of ECN-MF, we employ the t-SNE algorithm (Maaten and Hinton 2008) to visualize the distribution of the learned clustering embeddings Z in a two-dimensional space. As shown in Fig. 6, ECN-MF can better reveal the intrinsic clustering structure among the data.