High-quality sentence representation is essential for accurate new intent discovery. Multi-task pre-training is conducted using the BERT model to adapt representations for this task, integrating masked language modeling and sentence classification. Through predicting missing words and classifying sentences, the model learns intent-aware representations, enhancing its ability to handle unseen topics and diverse intents. The core of masked language modeling is to mask certain words in a sentence and predict them based on the remaining context. This process enables the model to capture both the semantics of intent-related words and the overall sentence structure. An illustration of this task is shown in Fig. 2.

In Fig. 2, text denotes the text input to the model, the words in the sentence are masked partially using a random masking strategy, and the predicted words are output after model modeling. Predicting masked words in a sentence allows the model to understand the internal structure of the sentence and learn sentence information. The equation of loss LM is shown below: (1)LM=−∑i=1Nmlog⁡P(Wm|Tm)where P(Wm|Tm) denotes the masked prediction probability distribution, Wm denotes the predicted word, Tm denotes the masked sentence, and NM is the number of masked words. The loss in masked tasks is reduced. The ability to predict masked words using sentence context is improved. Sentence dependencies are captured more effectively. The core idea is for the classification task to generate deep feature representations of the sentences. The corresponding sentence label is then calculated. Key features of the text are extracted during the classification process. Sentence comprehension is improved. The illustration of the classification task is shown in Fig. 3.

In Fig. 3, CLS is the output of the model, Model Output is the model output layer, Liner Layer is the linear layer, and Softmax is the normalization layer. The CLS output from the model is fed into a linear layer. Several linear layers reduce the high-dimensional features to match the number of classes. A Softmax layer normalizes the probabilities to a range between 0 and 1. The classification probability Pn,c is obtained. Combined with the class label, it is then processed through cross-entropy loss for calculation. The equation of cross-entropy classification loss LC is shown below: (2)LC=−1Nc∑n=1Nc∑c=1Cφn,clog⁡(Pn,c)where Nc is the number of categorized samples, C is the number of categories, φn,c is a symbolic function (0 or 1), indicating that the true category of sample n is equal to c takes 1. Otherwise it takes 0, which is the predicted probability that sample n belongs to the category. The masked language modeling loss is added to the classification loss, resulting in the total multi-task pre-training loss LMulti. The equation of LMulti is shown below: (3)LMulti=LM+LCwhere LM denotes the loss of the masked language modeling task, and LC denotes the loss of the classification task. Joint training of the two tasks helps prevent the SNID-ENSEF model from overfitting on a single task or data type. A balanced optimization across different tasks results in effective initial sentence representations, providing a solid foundation for subsequent training. The final layer of the SNID-ENSEF model connects to a mean pooling layer, preserving overall semantic information. The representation vectors for each word are averaged, producing a sentence vector that captures the combined semantic information of the words in the sentence. The equation of the pooled representation Ypool is shown below: (4)Ypool=∑iNwFiNwwhere Nw represents the number of word vectors, Fi denotes the feature vector of a word, and i indicates the position of the word vector. Mean pooling is applied to the word vectors by calculating the average representation of all words. The influence of random or irrelevant words is reduced on the overall representation. The overall representation improves stability and mitigates noise to some extent. The process facilitates subsequent training and the discovery of new intents.

Multi-task pre-training allows the SNID-ENSEF model to learn data features from different perspectives. The distinguishability of sentence representations is enhanced, and understanding of intent domain sentences is improved. The pooling layer outputs sentence representations, reducing the impact of noise and reinforcing stability.

After multi-task pre-training, universal intent sentence representations are obtained, but they lack task-specific optimization for new intent discovery. To address this, a syntactic contrastive learning approach is proposed. First, syntactic elimination increases the proportion of valid samples in the positive sample domain by removing invalid ones, improving training efficiency. This is akin to clearing clutter, allowing the model to focus on relevant data. Second, syntactic data augmentation enriches sample diversity, introducing varied representations within the same category. Together, these strategies help the model more effectively locate useful samples and benefit from enhanced sample diversity, improving new intent discovery.

The selection method for positive samples is crucial in contrastive learning. A semi-supervised approach is used to maximize the number of positive samples for training. Supervised information is combined to flexibly adjust the neighborhood radius of positive samples and define the positive sample domain. The elastic neighborhood radius R is chosen to maximize the number of positive sample domains while minimizing the boundary of ineffective samples. The significance of finding the elastic neighborhood radius lies in identifying the optimal region around each sample to balance useful data with minimal irrelevant noise, ensuring more accurate intent classification. Within an appropriate elastic neighborhood radius, only the most relevant data surrounding each example is included. This process is akin to continuously zooming in or out until the optimal level of detail is achieved. The selection of the elastic neighborhood radius R is shown in Fig. 4.

In Fig. 4, R represents the elastic neighborhood radius, N denotes the number of iterations, and MN is a variable indicating whether invalid samples exist in the neighborhood during the N-th iteration. When M is 0, it means there are no invalid samples in the neighborhood, and when M is 1, it means there are invalid samples in the neighborhood. K represents the state change counter. When K is greater than 2, it indicates that the neighborhood radius has undergone a large-small-moderate or small-large-moderate state change, meaning R is the appropriate neighborhood radius. N is equal to 0 and used to check whether the loop has run at least once. During the first iteration of the loop, there is no historical state, so the comparison of states is skipped. The overall process begins by calculating the upper and lower bounds of the elastic neighborhood radius R as the initial input. A binary search method is used to find the appropriate neighborhood radius. If invalid samples are found within the radius R, the radius is reduced until no invalid samples are present. Then, R is increased until invalid samples are just present. If no invalid samples are found within the neighborhood of radius R, R is first increased until invalid samples are present, then decreased until no invalid samples are found. The illustration of the elastic neighborhood radius R is shown in Fig. 5.

In Fig. 5, yellow samples represent valid samples, while green samples represent invalid samples. The orange cross-circles indicate that the radius is too large, causing too many invalid samples in the neighborhood. The yellow dashed circles represent a slightly smaller neighborhood radius, which cannot include as many valid samples as possible. The green circles represent the appropriate neighborhood radius. If the judgment condition is K < 1, the green circle cannot be obtained, and the search will always fall into ranges that are either too large or too small. Therefore, the judgment condition is set to K<2, allowing for an adjustment after the radius becomes too large or too small.

Due to the presence of ineffective samples in the positive sample domain and a more significant number of unknown ineffective samples, supervised information is used to calculate the number of ineffective samples in the selection strategy and to remove the ineffective samples from the supervised portion. The number of ineffective samples is then used to estimate the ineffective sample ratio in the positive sample domain and to calculate the reduction sample rate. It ensures that after the elimination of positive samples, the overall ineffective sample ratio increases, improving the training effectiveness of syntactic contrastive learning. Before the elimination of ineffective samples, the equation of the sample efficiency ε under the initial elastic neighborhood radius R is shown below: (5)ε=Ns−NvNswhere Ns represents the total number of samples, and Nv denotes the estimated number of ineffective samples. The process of eliminating ineffective samples can be understood as extracting L samples from Ns samples. After extracting L samples, the proportion of effective samples in the total can fall into three scenarios. Remaining unchanged, decreasing, or increasing. When L samples are extracted, and Q effective samples are present, the efficiency remains unchanged. The equation for calculating Q is shown below: (6)Q=Ns−Nv−ε×(Ns−L)where Ns represents the total number of samples, Nv denotes the estimated number of ineffective samples, ε indicates the efficiency of the elastic neighborhood samples, and L signifies the number of samples to be eliminated. When L samples are extracted, and the number of effective samples is greater than Q, the efficiency after extraction is less than that before extraction. The equation for calculating the probability PO is shown below: (7)PO=∑e=QLP(X=e)where P(X=e) represents the probability of having e-positive samples in the eliminated samples, and the summation indicates the cumulative probability. When the number of extracted effective samples is less than Q, the efficiency after extraction is lower than that before extraction. The equation for calculating the probability Pn is shown below: (8)Pn=∑e=0QP(X=e)where P(X=e) represents the probability of extracting e-positive samples, and the summation indicates the cumulative probability. After calculating the probabilities, L is taken when the probability Pn exceeds 50%. The negative impact of ineffective samples on training effectiveness far exceeds the benefits of a low proportion of additional effective samples. Therefore, when the extraction probability exceeds 50%, the overall training performance of the model is improved. The equation for calculating the reduction rate p is shown below: (9)p=LNswhere L represents the number of eliminated samples, and Ns denotes the total number of samples. The equation for calculating the positive sample domain Hp is shown below: (10)Hp=Ho(p)where Ho represents the original positive sample domain, and p denotes the reduction rate. It indicates that the positive sample domain is eliminated with a probability of p. After obtaining a suitable positive sample domain, data augmentation is applied to the positive samples using a combination of syntactic data enhancement and random token replacement. Data augmentation increases the diversity of the training data, enhancing the model’s ability to recognize sentences. The positive sample data augmentation is shown in Fig. 6.

In Fig. 6, Ji represents the i training sample, J^i denotes the positive sample uniformly selected from Ji positive sample domain, G indicates the sample after syntactic data augmentation, and S represents the sample after random token replacement. The specific operation of syntactic data augmentation involves two steps. The first step considers syntactic factors in selecting verbs as replacement words, and the second step involves choosing synonyms for substitution. Using the sentence “What steps are taken to transfer many into my account” as an example, it demonstrates how syntactic augmentation selects and replaces syntactic tokens. An example of syntactic replacement is shown in Fig. 7.

In Fig. 7, the original sentence is analyzed using Stanza CoreNLP [33] for part-of-speech tagging, obtaining the part-of-speech for each word in the sentence, such as ‘VB’ for verbs and ‘NN’ for nouns. ’Syntactic Select’ refers to the process of selecting the word with the part-of-speech tag ‘VB’ (the word ‘transfer’). “Similarity search and select” refers to searching for the list of candidate words with the highest semantic similarity to the selected word. Using an open-source online dictionary [34], a semantic similarity search is performed for the word ‘transfer,’ identifying the 15 most semantically similar words as candidates for replacement. One word is randomly selected from this list to replace the word in the VB position. This results in the syntactically augmented sentence. The illustration of random token replacement is shown in Fig. 8.

In Fig. 8, ‘Random select’ refers to the random selection of positions for replacement words, while ‘Random replace’ indicates the process of completing data augmentation using randomly selected words for substitution. By applying both syntactic enhancement and random replacement strategies to the sentences, positive samples for data augmentation are obtained. Subsequently, syntactic elimination contrastive learning is used to train the model. The equation of syntactic elimination contrastive learning loss LCON is shown below: (11)LCON=−1|NS|∑i∈MSlog⁡exp⁡(Sim(Ji,G)/τ)+exp⁡(Sim(Ji,S)/τ)∑k≠iMNegexp⁡(Sim(Ji,J^k)/τ)where NS represents the number of samples, MS denotes the sample index, Ji is the sentence embedding of the i-th sample, MNeg is the index of the negative sample for Ji, S is the sentence embedding of sample Ji after random replacement, G is the sentence embedding of sample Ji after syntactic data augmentation, and J^k is the k-th embedding of the negative pair after augmentation. τ is the temperature parameter, and Sim(.,.) is the similarity function on the normalized feature vectors.

The model optimizes syntactic elimination contrastive learning to cluster sentences with the same intent, reducing representation differences and achieving a more compact distribution in vector space. Conversely, it maximizes differences between representations of different intents, creating a more dispersed distribution. This adjustment of sentence representations establishes a solid foundation for new intent discovery.

In new intent discovery, the distance between samples of the same intent is key to accurate intent identification. Large intra-class distances can separate similar intent samples, while small inter-class distances may cause misclassification. To address this, a neighborhood sample fusion strategy replaces sample vectors with the mean of their neighborhood vectors, reducing noise and outliers. This results in more compact representations, decreasing intra-class distance while increasing inter-class distance, thereby improving intent recognition accuracy. This enhances the accuracy of new intent discovery. The illustration of the neighborhood sample fusion strategy is shown in Fig. 9.

In Fig. 9, the numbers represent sample indices. In Fig. 9a, the samples to be tested are selected as the central samples. The neighborhood sample fusion strategy is applied to the green sample with index 0, selecting the k nearest-neighbors of the 0-th sample vector. In Fig. 9b, the size of the sample neighborhood is determined based on the elastic neighborhood selection strategy. The yellow samples are the neighbors of the green sample at index 0, while the pink samples represent other samples surrounding the green sample. In Fig. 9c, the 0-th sample is replaced with the mean of its neighbor samples. The equation of the mean of the sample vectors XE is shown below: (12)XE=∑n=1kSnkwhere k represents the number of nearest-neighbor samples, n is the index of the nearest-neighbor sample, and Sn denotes the vector of the n-th neighbor sample. The neighborhood sample fusion strategy in creating cohesive clusters is based on the idea of similarity and adjacency, which is akin to placing similar items closer together. This reduces disorder and improves the accuracy of intent detection. The neighborhood can be compared to the classification areas in a library, where books on similar topics are placed in the same area, making it easier to find books on related subjects quickly. This helps the model recognize new intents more efficiently. As a result, the determination of sample similarity becomes more reliable, effectively lowering the difficulty of new intent discovery and aiding in the more accurate identification and definition of new intent categories. After determining the number of new intents, a similarity measurement method is employed to classify the intent sentence vectors into Nc new intent categories. An initial label selection algorithm is then used to choose Nc intent sentences as pseudo-labels for these new intent classes [32]. The remaining sentences are categorized into their respective new intent classes using similarity measures. The equation of the new intent class os is shown below: (13)os=argmini=1Ncd(h^,hi)where argmin denotes the ordinal number of the smallest value in the range in which the function is taken. d(.,.) calculate the spatial distance between the vectors. h^ denotes the vector of unclassified intent sentences. hi denotes the vector of intent sentences for the i-th pseudo-label.

The SNID-ENSEF model achieves the ability to represent intent sentences through multi-task pre-training and fine-tuning with contrastive learning, resulting in a uniform distribution of intent sentences within the vector space. New intent classes are obtained using a similarity classification method.

Model development and experiments are conducted on a cloud server, with an Nvidia GeForce RTX 3090 GPU utilized for training the BERT model. The Adam optimizer is utilized, and experiments are conducted with the Python programming language and the PyTorch framework. The versions are used Python 3.8.18, PyTorch 1.12.0, and CUDA 11.3. The experimental hyperparameter settings are as follows. The learning rate (Lr) is set to 1e-5, the batch size (Bs) is set to 128, the number of training epochs (Ep) is set to 50, and the elimination rate (p) is set to 0.05. The SNID-ENSEF model is tested on three publicly available intent datasets. Banking77 is a dataset of banking dialogues containing 77 intents derived from conversations in the banking context. StackOverflow is a large-scale dataset collected from an online Q & A platform. Clinc150 encompasses a wide range of user intents and scenarios, not limited to specific domains [32].

To evaluate the performance of the models Adjusted Rand Coefficient (ARI), Accuracy (ACC), and Normalized Mutual Information (NMI) are used to evaluate the performance of the SNID-ENSEF model as well as to compare the models [35]. Adjusted Rand coefficients are used to measure the degree of similarity between the categorization results and the real situation. Accuracy is used to measure the proportion of accurate categorization. Normalized mutual information measures the consistency between the categorization results and the real labels. The three evaluation indicators are distributed in [0, 1], with larger values representing more accurate categorization results.

In syntactic elimination contrastive learning, the magnitude of the elimination rate significantly impacts the effectiveness of training samples. To verify the rationale behind the chosen elimination rate, the effects of different elimination rates on the SNID-ENSEF model are examined. Five elimination rates ranging from 0.01 to 0.05 are selected around the optimal elimination rate, with the evaluation metrics displayed for the Stackoverflow dataset. The illustration of elimination rate variation is shown in Fig. 10.

In Fig. 10, the model performs best at an elimination rate of 0.03. Fig. 10a shows the change in the ACC index with the elimination rate. Fig. 10b shows the change in the ARI index with the elimination rate. Fig. 10c shows the change in the NMI index with the elimination rate. Fig. 10d shows the change of the sum of the three indicators with the elimination rate. The choice of elimination rate significantly impacts the effectiveness of the model’s training samples. When the elimination rate is too low, the effectiveness of the training samples may decrease or remain unchanged, failing to eliminate the training interference from invalid samples. Conversely, if the elimination rate is too high, the model may incorrectly remove valid samples, leading to a reduction in effective training data and overall poorer model performance. Therefore, selecting an elimination rate of 0.03 strikes a balance between removing invalid samples and retaining valid ones, providing the model with an adequate training dataset, which helps enhance its performance and generalization ability.

To select an appropriate parameter k for the neighborhood sample fusion strategy, the effects of different k values on the SNID-ENSEF model are examined. Values near the ten best k values are used. The variations in SNID-ENSEF model metrics under different parameters of the neighborhood sample fusion strategy are shown in Fig. 11.

In Fig. 11, differences in model performance are observed under various settings of k. Fig. 11a shows the change in the NMI index with k. Fig. 11b shows the change of ACC index with k. Fig. 11c shows the change in the ARI index with k. Fig. 11d shows the change in the sum of the three indicators with k. When k is too small, the neighborhood sample fusion strategy fails to filter out noise. Conversely, if k is too large, new noise may be introduced, which prevents the stabilization of new intents toward their respective classes, ultimately hindering the achievement of optimal results. Based on the experimental results, an appropriate k value is chosen to achieve effective outcomes.

The SNID-ENSEF model is primarily divided into the syntactic elimination contrastive learning module (SECL) and the neighborhood sample fusion strategy module (NSFS).SECL contains Syntactic augmentation (SA) and Elastic neighborhood elimination (ENE). Different stage combinations are used in the StackOverflow dataset. An ablation study is conducted to analyze the SNID-ENSEF model. The ablation experiments validate the effectiveness of the syntactic elimination contrastive learning module and the neighborhood sample fusion strategy module. The selection of modules for SNID-ENSEF ablation experiments is shown in Table 1.

In Table 1, Experiment 1 involves only using syntactic enhancement. Experiment 2 focuses solely on elastic neighborhood ablation. Experiment 3 combines both syntactic enhancement and elastic neighborhood ablation. Experiment 4 utilizes only the neighborhood sample fusion strategy. Experiment 5 implements a combination of syntactic enhancement and neighborhood sample fusion. Experiment 6 pairs elastic neighborhood ablation with neighborhood sample fusion. Finally, Experiment 7 represents the full SNID-ENSEF model, which employs both syntactic ablation contrastive learning and neighborhood sample fusion strategies.

Experiments 1–3 show that elastic neighborhood elimination outperforms syntactic enhancement in syntactic elimination contrastive learning. While data augmentation enriches sample diversity, elastic neighborhood elimination fundamentally increases the proportion of effective samples, improving data efficiency. Thus, it provides more valid training data. Both methods contribute to expanding training data, and their combination enhances sentence understanding and new intent recognition accuracy. Experiment 4 demonstrates that neighborhood sample fusion significantly aids in recognizing new intents. Experiments 5–6 confirm that combining syntactic elimination contrastive learning with neighborhood sample fusion achieves a more uniform distribution and that elastic neighborhood elimination is more effective than syntactic enhancement. The ARI, ACC, and NMI of syntactic elimination comparative learning are higher than those of the neighborhood sample fusion strategy module. In the absence of syntactic elimination comparative learning, the ACC value decreased by 0.41% compared to the results with both modules. When the density distribution-aware comparative learning module is missing, the ACC value drops by 1.68%. In conclusion, the two modules introduced in the study significantly contribute to the accuracy of new intent discovery. The ablation experiments of random token replacement (RTR) and syntactic token replacement (STR) are shown in Table 2.

In Table 2, the experimental results show that SRT significantly outperforms RRT, while RRT shows a relatively smaller improvement. However, when combined, SRT enhances sentence diversity, while RRT introduces some appropriate noise, improving the model’s robustness and leading to better performance.

To evaluate the performance of the proposed SNID-ENSEF model, comparative experiments are conducted on the Banking77, Stackoverflow, and Clinc150 datasets. The benchmark models for comparison include: Kmeans++ [36]: A traditional clustering algorithm that improves the initialization of cluster centroids. PTJN [37]: A robust pseudo-label training and source domain joint training network. Noisy pseudo-labels are refined using prior knowledge, and a new extractor-generator-corrector architecture is introduced. ELECTR [38]: A Transformer-based language representation pre-training model that draws on the ideas of GANs. It trains the model by distinguishing between real words and “fake” words generated by a small generator model. DPN [39]: An end-to-end deep contrastive clustering algorithm. The algorithm jointly updates model parameters and clustering centers through supervised and self-supervised learning, optimizing the use of labeled and unlabeled data. MPNET [40]: A new pre-training model that improves traditional pre-training methods through a “Masked and Permuted Pre-training” strategy. MTP-CLNN [35]: A multi-task pre-training model for new intent discovery has been proposed. Utilizing self-supervised signals in the representation space to improve the accuracy of new intent discovery. USNID [41]: A new intent discovery model that introduces a centroid-guided clustering mechanism. DWG [32]: A new intent discovery model that employs a novel diffusion-weighted graph framework. This framework uses a weighted method based on semantic similarity and local structure for contrastive learning.

As shown in Table 3, the SNID-ENSEF model exhibits strong performance in terms of NMI, ARI, and ACC across the Banking77, StackOverflow, and Clinc150 datasets. Compared to the highest-performing models (DWG) in terms of NMI, ARI, and ACC from Kmeans++, PTJN, ELECTR, DPN, MPNET, MTP-CLNN, USNID, and DWG, the SNID-ENSEF model shows improvements of 1.17%, 1.15%, and 0.34% in NMI, 0.88%, 1.86%, and 1.07% in ARI, 2.27%, 0.9%, and 0.75% in ACC, respectively. The training of the SNID-ENSEF model utilizes elastic neighborhood boundaries to select positive sample domains, ensuring a high quantity of training data while eliminating ineffective samples to enhance sample efficiency. Additionally, by referencing syntactic information and substituting meaningful words in sentences, the model increases the diversity of training samples, thereby improving training effectiveness. The use of the neighborhood sample fusion strategy reduces noise and decreases the difficulty of the new intent discovery task. By combining these approaches, the SNID-ENSEF model learns high-quality intent sentence representations from limited training samples, enhancing the accuracy of the new intent discovery task.

To validate the generalization ability of the model, two additional datasets were introduced to evaluate its performance across different domains. MCID [42]: An open-source intent detection dataset for COVID-19 chatbots focusing on the healthcare domain. It contains sixteen intents and is used to test the applicability of the model in the medical field. HWU64 [43]: A dataset consisting of 25716 utterances across 21 domains and 64 intents. Compared to Clinc, which has fewer domains, HWU64 enables the testing of the performance of the model across a broader range of domains. The results are presented in Table 4.

As shown in Table 4, the ESEF-SNID model demonstrates an improvement over models such as DWG and MTP-CLNN, exhibiting stable performance across different datasets. This stability to some extent validates the generalization capability of the ESEF-SNID model.

With the rapid development of large language models, an increasing number of task-specific models are being enhanced by these large models. Integrating large language models will further improve the performance of models on specific tasks. For the ESEF-SNID model, leveraging large language models can refine the distinction of previously unknown intents, allowing for a more detailed differentiation of broadly separated intents, thereby increasing the accuracy of intent discovery. Another future direction involves converting newly discovered intents into defined intents. However, this process requires significant human effort and computational resources. Therefore, integrating large language models to assist in defining discovered intents is a crucial area that needs to be addressed in future work.

Virtual assistants are able to respond to users’ questions. The application of new intent discovery in virtual assistants enables them to provide appropriate replies to various user inquiries, allowing them to more intelligently address a wide range of user needs without being limited by predefined tasks. This increase in flexibility has a profound impact on user satisfaction and interaction experience, making conversations more engaging and open-ended. For example, when a home voice assistant encounters a newly introduced term for the first time, it may not provide an effective response because it cannot recognize the meaning of the new term. However, through the discovery of new intent, the assistant can capture this intent, allowing it to provide appropriate replies in the future when the term or its associated intent is mentioned again.

To discuss the ability of the SNID-ENSEF Model to recognize intent meaning overlap and intent sentence similarity, two major overlapping intent categories in the Banking dataset Card and Transaction intents-were extracted into four sub-overlapping intents, resulting in a total of twenty-nine categories. The performance of DWG and SNID-ENSEF was then tested in extreme cases. The dataset labels and intent distribution are shown in Table 5. The model performance is shown in Table 6.

As shown in Table 6, in extreme cases, SNID-ENSEF outperforms the strongest competing model, DWG, in the NMI, ARI, and ACC metrics. This indicates that SNID-ENSEF still retains a certain ability to recognize intents even under extreme conditions.

To test the model’s actual performance, SNID-ENSEF and the DWG model were tested on the BANKING dataset, and real-time performance metrics were recorded for comparison. The experimental results are shown in Table 7.

As shown in Table 7, compared to the DWG model, the SNID-ENSEF model is 50 s slower. However, thanks to the matrix operations used in the proposed method, this time difference is within an acceptable range. The memory usage increased by 10 MB without any trade-off between space and performance. Overall, the SNID-ENSEF model does not have significant disadvantages in terms of time and memory usage compared to the strongest competing model while showing an improvement in performance.

Perform significance testing on the model’s various metrics to verify the performance improvement of the SNID-ENSEF model compared to other competing models. The formula for the t-test is as follows: (14)t=X−μSDnwhere X represents the data point to be tested, μ represents the mean of the other data, SD represents the standard deviation, and n represents the number of data points. Calculate the mean and standard deviation of the metrics listed in Table 3 to compute the t-value. Then, obtain the p-value from the corresponding t-distribution. Set the null hypothesis: There is no significant difference between the SNID-ENSEF model and the competing models. Set the alternative hypothesis: there is a significant difference between the metrics of the SNID-ENSEF model and the other competing models. Reject the null hypothesis if the p-value is less than 0.05. The specific calculations are as follows.

As shown in Table 8, the p-values for all metrics of the SNID-ENSEF model are less than 0.05 compared to the competing models, allowing us to reject the null hypothesis. Additionally, the silhouette scores for the strongest competing model, DWG, and the SNID-ENSEF model are calculated. The silhouette score of the DWG model is 0.6811, while the silhouette score of the SNID-ENSEF model is 0.8755, further validating the performance improvement of the SNID-ENSEF model.

In order to fully demonstrate the initial value selection and parameters of the elastic neighborhood strategy, a set of 669 samples is taken, and the Euclidean distance from the first sample to all other samples is calculated. For the sake of convenience, the Euclidean distances in this paper are scaled by a factor of 1000. The relationship between the samples and their distances is shown in Fig. 12.

In Fig. 12, the distances between the first sample and all other samples are plotted, where the minimum distance is denoted as Rmin. At this point, the neighborhood of the sample contains only one sample, and there are no invalid samples. The maximum distance is denoted as Rmax, at which point the neighborhood contains all samples, and there are certainly invalid samples. This choice of values ensures that the elastic neighborhood strategy will have a solution. The change in the value of R after the judgment process is determined by the following formula: (15)RC=DENSA

In the distance range close to 600, where 600 samples are distributed, RC is approximately 1. Therefore, the value of R is adjusted as R+1 or R−1.