The dataset is divided into safe minority and boundary minority categories, with the following definitions. For a given dataset D∈Rd, it consists of both minority class samples and majority class samples. The minority class dataset is denoted as D+, and the majority class dataset is denoted as D−.

If xi∈D, xj∈D, d(xi,xj) indicates the distance between xi and xj, dmin represents the minimum distance between xi and its nearest neighbor in D, the nearest neighbor of xi in D is denoted as xnn, xnn={xj∈D∣d(xi,xj)=dmin}.

Definition 1 (Boundary minority sample): If x∈D+, xnn∈D−, then x is a boundary minority sample, as demonstrated in Fig. 2a.

Definition 2 (Safe minority sample): If x∈D+, xnn∈D+, then x is a safe minority sample, as demonstrated in Fig. 2b.

For each of the minority class samples xi, xi∈D+, if the nearest neighbor of the minority class sample xnn∈D+, then xi is added to the safe minority class set Dsafe+. Otherwise, if xnn∈D−, then the xi is added to the boundary minority class set Dboundary+.

As can be seen from Fig. 2a,b, boundary minority samples and safe minority samples have heterogeneous characteristics, so multiple strategies should be taken into account for data sample synthesis. We aim to increase the diversity of synthesized samples and expand the boundary of the minority class. Meanwhile, for safe minority class samples, it is necessary to increase the local area as much as possible and avoid generating duplicate examples and noisy samples.

For the safety minority sample synthesis, the location of the nearest majority samples for one safety minority sample should be found. Then, a new safety minority sample is synthesized randomly within the formed circle by taking this safety minority sample as the center and the distance from the nearest majority class sample as the radius. This procedure is illustrated in Fig. 3a.

fgensafe represents the synthesized sample by safe minority class and is computed with Eq. (2). Unlike the linear interpolation synthesis of the safety minority sample in PF-SMOTE, we randomly synthesize a new sample within a local area, in which the minority class sample point is the center of a circle and the radius is the distance from the nearest majority class sample to the minority class sample. (2)fgensafe=rsinθ+xi,θ∈U(0,2π),r∈U(0,d|xi−xnn−|)

For the boundary minority sample synthesis, the boundary toward the majority class sample is extended, i.e., the position of the synthesized sample is biased towards the position of the majority class sample. Specifically, for a boundary minority sample, its nearest majority class sample is first found, and a boundary minority class sample is synthesized by interpolating along the line between boundary minority class sample point and the nearest majority class sample point. The Gaussian process is employed to enhance the diversity of the synthesized samples. This procedure is shown in Fig. 3b. fgenboundary represents the synthesized sample by boundary minority class as given in Eqs. (3) through (6). (3)fgen boundary =12πσexp⁡(−(xi−μ)22σ2) (4)N((xi+gap×(xnn−−xi)),σ2) (5)w.r.t.μ=xi+gap×(xnn−−xi)),gap∼U(0,1) (6)σ=∑i=1∣D+∣(xi−x¯)2|D+|−1where xnn− is the nearest majority sample points of xi. |D+| is the size of data set, and gap follows a uniform distribution. Algorithm 1 provides the pseudo-code for data synthesis.

Noise is inevitably introduced during the synthesis of new samples, potentially degrading the quality of both sample attributes and class labels and adversely affecting model performance. To mitigate this, data cleaning is applied to all samples, including both majority and minority class samples, based on two key aspects: the WENN method and relative data density.

(1) WENN

WENN addresses class imbalance and the small sample problem through a distance scaling function. By applying different distance scaling for positive and negative candidate neighbors, WENN effectively preserves a higher proportion of safe minority and safe majority samples.

In WENN, the distance between two samples is defined using the isomorphic value difference metric, as described in Eq. (7). (7)dHVDM(x1,x2)=∑a=1mda2(x1,a,x2,a)where x1 and x2 represent the feature vectors of two samples. a denotes the attribute index, and m is the number of attributes. The distance scaling function of WENN d is shown in Eq. (8), where N−, N+ and N represent the sizes of the majority class sample, minority class sample, and total sample. (8)d(x1,x2)={ e(N+N)m⋅dHVDM(x1,x2),x2∈D+e(N−N)m⋅dHVDM(x1,x2),x2∈D−

Here, x2 is the k closest sample of x1 (i.e.,k=3). k nearest neighbors of x1 can be found by computing all distance scaling functions of these nearest samples. Ki+ denotes the number of majority class samples in the k nearest neighbors of xi, Ki− denotes the number of minority class samples in the k nearest neighbors of xi. Data samples are cleaned up according to the rules given in Eq. (9). (9){ Ki+>Ki−,xi∈D−Ki+<Ki−,xi∈D+

Fig. 4a shows an unscaled distance sample, where the k (k=3) nearest neighbors of x1 are x2,x3,x4. x2,x3 are minority class samples and x4 is a majority class sample. In this case, if x1 is the minority class sample, then it should be retained. In Fig. 4b, after scaling with distance weights, the k(k=3) neighbors of x1 are x3,x4,x5. x3 is a minority class sample and x4,x5 are majority class samples. If x1 is a majority class sample, then x1 should be retained.

(2) Relative data density

Besides WENN, we design another rule for data cleaning based on relative data density. A reverse k-nearest neighbor is adopted to determine the relative data density. The inverse k-nearest neighbor is defined as: The reverse nearest neighbor of query point q is the set of all data points in data set D whose distance from q does not exceed the kth nearest neighbor of q. It can be noted as RkNN(q,k)={p∈D(p,q)<=dist(q,p′)}, where p′ is the kth nearest neighbor of q in dataset D.

Fig. 5 shows the reverse k neighbor sample diagram. As shown in Fig. 5a, when k = 3, the query sample has 3 nearest neighbors A, B, and D. In Fig. 5b, the query sample has 4 reverse nearest neighbors A, C, B and D for it belongs to the 3 nearest neighbors of these samples. Thus, there are two significant differences between kNN and RkNN: kNN contains a specific number of samples, while RkNN may contain from 0 to an infinite number of samples. Data samples are cleaned up according to the rules given by Eq. (10). Algorithm 2 provides the pseudo-code for data cleaning.

(10){ Ki−>2×Ki+,xi∈D+Ki−<2×Ki+,xi∈D−

In this section, 40 unbalanced datasets from the KEEL repository are used to evaluate the performance of RE-SMOTE. Table 1 provides details of the datasets, including the numbers of attributes (Attr.) and examples (NE), the number of each class (%Class(maj,min)), and the imbalance ratio (IR).

Repeated stratified k-fold cross-validation is employed, using ten replicates of 10-fold cross-validation, resulting in 100 models being fitted and evaluated. The dataset is divided into 10 subsets, each containing 10% of the samples. In each iteration, one subset is used as the test set, while others are used for training. The average of the 10 repetitions is considered as the final performance metric.

While classification accuracy is often used to evaluate algorithm performance, it is not an ideal metric for imbalanced datasets due to the skewed class distribution. Unlike standard metrics, which assume equal importance for all classes, imbalanced classification problems typically prioritize minimizing classification errors in the minority class over the majority class. As a result, performance metrics must focus on the minority class, which poses a challenge due to the limited representation of minority class observations, making it harder to train an effective model. Therefore, we employ two widely recognized metrics, AUC and F1 score [21]. To better explain AUC and F1, the confusion matrix of a dichotomous problem is shown in Table 2, and the corresponding concepts are given as follows:

TP (True Positive). The number of positive class samples that were predicted as positive class.

FN (False Negative). The number of positive class samples that were predicted as negative class.

FP (False Positive). The number of negative class samples that were predicted as positive class.

TN (True Negative). The number of negative class samples that were predicted as negative class.

Based on the confusion matrix, ROC curves can be drawn on different thresholds. ROC curves, also called subject working characteristic curves, are composed of True Positive Rate (TPR) and False Positive Rate (FPR) at different classification thresholds, as given in Eqs. (11) and (12). TPR is the vertical axis and FPR is the horizontal axis. Each threshold corresponds to a (FPR, TPR) point, which is depicted as the ROC curve. The closer the ROC curve is to the upper left corner, the higher the model’s accuracy. (11)TPR=TPTP+FN (12)FPR=FPFP+TN

AUC is the area under the ROC curve. The larger the area under the ROC curve, the better the model. There are two obvious advantages of AUC. First, AUC does not focus on specific scores. It reflects relative results such as ranking relations. Second, AUC is an overall indicator and does not focus on the local characteristics of the model, so it is not sensitive to the sample. AUC can be represented as Eq. (13), where (xi, yi) is the coordinate and n is the number of points. (13)AUC=12∑i=1n−1(xi+1−xi)⋅(yi+yi+1)

The F1 value is the summed average of precision and recall, as given in Eq. (14). It is close to the smaller of these two values. If the F1 value is large, then precision and recall must be large. F1 can reflect the algorithm’s overall performance. (14)F1=2TP2TP+FP+FN

To evaluate the proposed RE-SMOTE model, 3 well-known classifiers are selected in this experiment, which are described in detail as follows:

• Decision Tree (DT) [42,43]. It can be applied to both classification and regression, with the leaf nodes being the final decision result. It starts with a root node containing all the training data and then continuously refines the internal nodes using specific division criteria until the stopping conditions are satisfied. Thus, classification rules can be obtained inductively. The training is to construct a tree based on the given dataset and select the most valuable feature-slicing nodes. Decision tree is easy to understand and interpret. The data preparation is simple. It can be used constructed for data containing many attributes. Moreover, the decision tree scales well to large databases while its size is independent of the dataset.

• Support Vector Machine (SVM) [44]. The basic idea is to find the best-separating hyperplane in the feature space to maximize the interval between positive and negative samples in the training set, and with the power of kernel functions, SVM can also be used to solve nonlinear problems. The SVM classifier can be adapted to small training datasets and easy-to-fit high-dimensional samples. In addition, it can also handle the problem of neural network structure selection and local minima prevention.

• K-Nearest Neighbor (KNN) [45]. KNN is used for classification by measuring the distance between different feature values. It is based on the idea that a sample belongs to a class if the majority of the k most similar (i.e., most neighboring) samples in the feature space belong to that class. k is usually an integer no greater than 20. The neighbors selected in the KNN algorithm are correctly classified objects. It relies only on the categories of the nearest neighbors to determine the classification of the samples. KNN is simple and low-cost, and the training time and space are linearly related to the size of the training data set. It is more suitable for the set of samples to be divided with more crossover or overlap of class domains since it relies on a limited number of neighboring samples around.

To verify the performance of the RE-SMOTE method, three SMOTE variants are selected for experimental comparisons. A brief descriptions of these methods are given as follows:

• SMOTE-ENN [46]: It starts with oversampling samples of minority class using SMOTE, and then performs local data cleaning using ENN. If the predicted label of k-nearest neighbors (KNN) is different from the true label, the sample is considered noisy and deleted, otherwise, the sample is retained.

• SMOTE-WENN [20]: It designs a new data cleaning method WENN. WENN uses a weighted distance function and KNN rules to detect and remove unsafe majority and minority samples. The weighted distance function extends a suitable distance by considering local imbalance and spatial sparsity.

• PF-SMOTE [19]: It is a parameter-free variant of SMOTE that generates a sufficient number of representative synthetic samples based on bounded minority and the safe minority classes while avoiding the generation of interpolated noisy samples.

• SMOTE-RkNN [47]: This method introduces an improved SMOTE hybrid algorithm called SMOTE-reverse k-nearest neighbors (SMOTE-RkNN). The algorithm identifies noise based on probability density rather than relying on local neighborhood information.

Among the 40 datasets in Table 1, we randomly select two datasets for visualization, numbered D1 and D23, for visualization. For these comparative methods, we set the k-nearest neighbors parameter to 3 based on empirical experience. Other parameters will be adjusted and calculated according to the characteristics of different datasets.

The original dataset, the balanced dataset after the SMOTE, and the balanced dataset after RE-SMOTE are listed, respectively. In this way, the regions where RE-SMOTE generates samples can be visually displayed. The final visualization plot of applying SMOTE and RE-SMOTE to a two-dimensional data set is shown in Fig. 6, where blue points represent the minority class samples and red points represent the majority class samples. Original data sample plots are given in Fig. 6a,d. Balanced data sample plots after SMOTE are given in Fig. 6b,e.

We randomly select a minority class sample and interpolate between the nearest minority class neighbors to synthesize a new minority class sample. SMOTE is a linear interpolation to synthesize the new sample. This leads to a more convergent distribution of the synthesized minority class sample than the original one. Balanced data samples performed by RE-SMOTE are given in Fig. 6c,f. In contrast, RE-SMOTE focuses on the diversity of the synthesized samples and favors the synthesis of minority-class samples. In general, it can be concluded that RE-SMOTE is more effective in sample synthesis.

The comparison results on AUC and F1 with DT-based, KNN-based and SVM-based classifiers are shown in Tables 3–12, respectively. Through these results, it can be observed that the proposed RE-SMOTE outperforms other variants in most cases. Specifically, the proportion of best results with RE-SMOTE on 40 data sets is shown in Fig. 7.

Table 3 presents the average results of comparative methods across 40 datasets, tested on three different classifiers. From the table, it is evident that our method consistently outperforms the comparative approaches across all classifiers, demonstrating its superior performance. This highlights the effectiveness and robustness of our approach to handling diverse datasets.

In addition to the above comparisons, “The Wilcoxon Signed Rank Test” is also used for statistical analysis of the proposed RE-SMOTE. The Wilcoxon Signed Rank Test (also known as Wilcoxon Signed Rank Sum Test) is non-parametric, and it is often used to determine the matching degree of the overall data distributions, especially for non-normal conditions. The statistical test results on AUC and F1 with DT-based, KNN-based, and SVM-based classifiers are given in Tables 6, 9, and 12, respectively.

R+ and R− represent the value of sign rank test for RE-SMOTE and compared models. The ratio of R+ and R− can be used to express the performance criteria, and it is expected for a big value. We can find from the result tables that R+ value is greater than R− for all cases.

For p-value, we have zero hypothesis: there is no difference between the performance of RE-SMOTE and other models. A smaller p-value means this zero hypothesis can be rejected. As can be seen from the results, the p-value is smaller for most cases (significantly less than alpha-value = 0.05). There is only one exceptional case in Table 13. p-value = 0.16178 for RE-SMOTE vs. SMOTE-WENN on F1. Therefore, for comprehensive consideration, there are significant differences between RE-SMOTE and other compared models.

In this section, the runtime for processing all 40 datasets has been recorded and presented in Table 13. Each dataset is processed using the methods described in this study, and the total time for each dataset is measured to provide a comprehensive overview of the computational performance. Based on the results, it can be observed that the processing time increases as the dataset size and imbalance ratio (IR) grow. Due to operations such as reverse k-nearest neighbor searches, the time consumption grows significantly as the dataset size increases.

Regarding the complexity, two core algorithms are analyzed. The time complexity of Algorithm 1 is O(d|D+||D−|), dominated by the nearest neighbor search between minority-class samples D+ and majority class samples D−.

For Algorithm 2, the time complexity of Algorithm 2 is O(d|S|2k), primarily driven by the nearest and reverse nearest neighbor searched within the synthesized dataset S. The primary computational burden arises from search operations, which become increasingly intensive as the size of the synthesized dataset grows.