Ensemble learning is a very important area of artificial intelligence and data mining, which aims to build an integrated model by combining individual learners to improve the overall performance [41]. In terms of integration approaches bagging and boosting (the adaptive boosting (AdaBoost) algorithm [42] is the most commonly-implemented type of boosting algorithm) are two representative models of ensemble learning. Bagging is one of the first ensemble learning algorithms and uses a parallel integration strategy to randomly select different subsets of the training data. Each subset is then trained based on the same individual learners, and the final prediction results are obtained using a minority-majority approach for the classification problem and a simple average approach for the regression problem [43,44]. The bagging algorithm can improve generalization by reducing the variance error [45]. The integration steps are as follows: Suppose we have a training set (1)D={(x1,y1),(x2,y2),…,(xm,ym)}

The bagging algorithm first resamples the data set, creates a new training subset, and then puts the sampled data into the original data set again. Although this approach may result in some training samples being selected multiple times, while others may not be selected at all, this does not affect the prediction performance of the model. After selecting the individual learning algorithm, put each training subset into the algorithm for calculation. (2)ht=L(Dt)

Combining multiple individual learners to build a learner with stronger predictive ability in the regression problem, the combination strategy is to simply average the prediction results of each weak learner. (3)H(x)=1T∑i=1Thi(x)

In Eqs. (1)–(3), D is the training dataset, (xi,yi)(i=1,…,m) is the sample in the training dataset, m is the total number of samples, xi is the features of the input data, yi is the label value of the sample, ht is an individual learner,Dt is the subset constructed after resampling, t is the number of samples, L is the individual learning algorithm, H(x) is a learner with stronger predictive ability after integration.

Individual learners refer to algorithmic models with simple structures. Generally speaking, in the bagging integration strategy, the individual learners are called base learners, and in the boosting integration strategy, they are called component learners [44]. For convenience, we collectively call them individual learners.

The main idea behind this algorithm is that the more similar things are, the more likely they are to be adjacent to each other, and it obtains the maximum possibility of the data type of the current point by looking for the data category with the most adjacent data points. This algorithm was chosen because of its simplicity and the ability to distinguish it from other algorithms through the idea of distance sampling [46]. The advantage of the algorithm is that it is simple and insensitive to outliers. The disadvantage of this algorithm is that it has high time complexity and spatial complexity, and its interpretation ability is not strong. However, under the condition of small samples, the computational pressure of the algorithm is greatly reduced. Fig. 3 shows the forecasting principle of the KNN algorithm.

Because the model used for comparison has the same construction steps as this framework, we describe its construction process together in this section, and Fig. 4 illustrates the overall design flow of this study.

In order to make the model comparison more convincing, we use K-nearest neighbor (KNN) [48], Support vector machine (SVM) [49], and Classification and regression trees (CART) [50] as individual learners for bagging and boosting integration, and in addition, we choose four more mature integrated models of Random forest (RF) [51], Extra tree [52], Gradient boosting [53] and Light gradient boosting machine (LightGBM) [54] for comparison. Based on these four models, our experiments first explored the influence of factors such as eliminating discrete values, reducing dimensions, and changing the proportion of data segmentation on the experiment, which provided the basis for data preprocessing in the framework construction process.

According to the exploration results of the influencing factors, in the aspect of data preprocessing, firstly, according to the corrosion rate, the experimental data set was eliminated by box-plot [55]. Considering the different divisions of the training set and the test set under the condition of small samples, the results may be quite different. Before data splitting, three random seeds (222, 444 and 666, respectively) are used for out-of-order processing, to ensure that there are multiple groups of experiments to compare under the same data set. The data was then split into training and test sets at a ratio of 9:1 [56], and after the data was uniformly normalized, Principal Component Analysis (PCA) [57] dimensionality reduction was performed to retain more than 90% of the information.

During the process of model building, a combination of 10-fold cross validation [58] and grid search was used to determine the optimal hyperparameter of each model. It was also debugged as a hyperparameter, and the grid search was carried out within a certain range. It was worth noting that we separately search the basic model and the integration method in the grid, and then integrated them according to the selected optimal hyperparameter. Firstly, the training data was divided into 10 subsets on average, of which 9 subsets were used for model training and the rest were used for model verification. This operation was repeated 10 times in a row, the average error was obtained, and then all the parameters in a certain range were traversed by the grid search method. Finally, the parameter combination with the smallest error in 10-fold cross-validation was obtained. The purpose of this process is to achieve the best predictability for each model and to minimize the influence of random seeds on the accuracy of the model. The generalization ability of each model was then tested on the test set and its error was calculated on the test set using mean square error (MSE), mean absolute percentage error (MAPE), and mean absolute error (MAE) metrics, where MSE is the main evaluation index. Finally, the Friedman ranking [59] was used to test the advantages and disadvantages of the model.

To verify the superiority of the model, we also compared the traditional empirical model with the more complex stack integration model and discussed whether the integrated basic learner can improve the performance.

MSE, MAE, and MAPE were used to determine the error value of the integrated algorithm on the test set. The differences between the predicted and true values of the bagging and boosting algorithms based on MSE are demonstrated in Fig. 5. The closer the result is to the diagonal, the smaller the error between the two groups of data. The results show that when the random seeds were 222 and 444, the extra trees, bagging + KNN, and AdaBoosting + SVR models showed the minimum MSE values. However, when the random seed was 666, this changed to bagging + CART, AdaBoosting + CART, and AdaBoosting + SVR. This indicates that the prediction performance of the model changes significantly depending on the training and test sets used.

To evaluate the overall performance of each model on multiple datasets, a Friedman ranking plot was drawn (Fig. 6). The blue dots in the graph indicate the average ranking of the algorithm, and the farther to the left the numerical value is, the better the model is. The results show that the prediction performance of the bagging + KNN algorithm is the best, followed by extra-trees and bagging + CART, and the prediction performance of random forest was the lowest. The horizontal lines in the figure indicate the allowable fluctuation range of each algorithm ranking. If the horizontal lines of a given pair of algorithms do not overlap each other, it indicates that there are significant differences between these algorithms. However, here, the algorithms in the graph all overlap with each other, indicating that there is no significant difference between them. The order of all algorithms from good to bad is bagging + KNN, extra trees, bagging + CART, adaboosting + KNN, adaboosting + CART, adaboosting + SVR, bagging + SVR, Gradient boosting, LightGBM, Random Forest.

To investigate whether the ensemble learning models performed better than the individual learners, the results of integrating the individual learners KNN, SVR, and CART with the bagging and AdaBoosting algorithms were compared in three randomized segmentation experiments (Fig. 11). These results show that the bagging algorithm reduces the MSE error values when the individual learners are KNN and the random seed is 222 or 444, while AdaBoosting increases the error value under all three random seeds; further, when the individual learners are SVR, except for the case with a random seed of 666, AdaBoosting significantly reduced the error, and in other cases, bagging and AdaBoosting had similar error values as the individual learners. However, when the individual learners are CART and the random seed is 444 or 666, both bagging and AdaBoosting reduce the prediction error values, while the opposite is observed for a random seed of 222.

A traditional model for oil and gas pipeline corrosion rate prediction, the de Waard model [5,6], is given by the following equations: (4)1Vcorr=1Vr+1Vm (5)log10Vr=4.93−1119t+273+0.58log10Pco2−0.34(pHact−pHco2) (6)pHco2=3.82+0.00384t−0.5log10Pco2 (7)Vm=2.45Vl0.8d0.2Pco2

In contrast to the de Waard model, the Norsok M506 model [9] uses different equations over different temperature intervals. In the temperature range from 20°C–150°C, the model is given by: (8)Vcorr=Ktfco20.62(τw19)0.146+0.0324log10fco2f(pH)t

Between 15°C–20°C, the model is given by: (9)Vcorr=Ktfco20.36(τw19)0.146+0.0324log10fco2f(pH)t

Between 5°C–15°C, the model is given by: (10)Vr=Ktfco20.36f(pH)t

In Eqs. (4)–(10), Vcorr is the corrosion rate in mm/a, Vr is the reaction rate in mm/a, Vm is the mass transfer rate in mm/a, t is the media temperature in °C, Pco2 is the CO₂ partial pressure in MPa, pHact is the actual pH, pHco2 is the pH of CO₂-saturated solvent, Vl is the liquid phase flow rate of the medium in m/s,d is the pipe diameter in m, fco2 is the fugacity of CO₂, Kt are temperature-dependent constants, τw is the wall shear force in Pa, and f(pH)t is the pH-dependence term.

As the parameters of the de Waard model include the pipe diameter, the effect of which cannot be measured under laboratory conditions, the Norsok M506 model was selected as the control model for this study, as it is a purely empirical model based on a large amount of experimental data. The remaining data after the exclusion of the discrete values were selected for the comparison of prediction results, and the MAE value was used as the performance metric. For the ensemble learning algorithms, the average values of MAE for each model under three random seeds were compared. Fig. 12 shows that the MSE error value of the bagging + KNN framework recommended in this study is only 0.0346, while the error value of the traditional Norsork M506 model is 0.1322. This framework is also slightly better than other ensemble models.

The Stacking [61] model allows any combination of different types of models. The integration strategy is to first use the K-fold cross-validation method on the original data to predict the model that needs to be integrated. After obtaining the predicted value of each integrated unit, the average value of the predicted results is used as input into a simpler prediction model (the purpose of this is to prevent overfitting), and the predicted value is the final result. The ensemble units of this ensemble model are usually selected as bagging model and boosting model. In this section, a total of 11 ensemble models (with the addition of the XGBoost [62] model) were used as base models to be integrated into the Stacking mode, in this experiment, to control the number of combinations, each model was only used once. Fig. 13 shows the prediction errors under different combinations. From this figure, it can be seen that with the increase of the number of combinations, the prediction error values become more concentrated, and almost every segment has similar error variation rules. Table 4 shows the minimum values of MSE for different combinations and the percentage of their predictions that are improved (relative to the models involved in the combination). When the random splitting seed is 666, under the premise that the prediction error of the basic model is larger, the integrated prediction error is further enlarged, which leads to the fact that the prediction performance of almost all models hardly improves under this data set partition condition. If only the random splitting seeds 222 and 444 are analyzed, as the increase of integration times, the probability that the prediction effect after integration is better than that of a single sub-model will gradually decrease. These results show that even if there is a combination of better prediction effects in stacking, regardless of the prediction performance of the model or the proportion of prediction performance improvement, the stacking integration mode is not very suitable for small sample prediction under this data.

In this study, the ensemble learning algorithms with the smallest MSE were found to be bagging + KNN, extra trees, and bagging + CART, which were all based on the bagging framework. This shows that the bagging algorithm is superior to boosting algorithm in small data set prediction. The results obtained here are consistent with Wang et al. [44], those who found that the bagging method was superior to boosting method in classification, and the former performed relatively better in the presence of noise. However, when using ensemble learning to analyze remote sensing images, Both Chan et al. and DeFries et al. [63,64] found that the AdaBoost algorithm had higher accuracy and that the bagging algorithm is more stable. The reason for the differences between the aforementioned studies and the present study is that in the latter, there are some instances where the discrete values (i.e., noise) are not completely removed, which is more detrimental to boosting methods [65]. The differences between the optimization strategies of the two algorithms also contributed to their different prediction performance. The bagging algorithm uses a parallel strategy to average the errors of some discrete values while boosting uses a serial strategy to increase the weight of the discrete value errors present in the training set during the optimization process. It is worth noting that among different models, the random forest algorithm has the lowest prediction rank, which is mainly due to the large prediction error of the last random seed. Bagging + KNN framework has achieved the best prediction results, but under the condition of small samples, data plays a vital role in the results, so the performance of other data sets is still worth exploring.

In this experiment, the prediction error of each model is reduced after the discrete values are eliminated, because when the amount of data with some special features is small, the distribution of these data relative to other data forms a small disjunctive area which may not be visible [66,67]; Therefore, it is not easy to determine whether the values in these areas are true values or noise [68], and the existence of such values leads to insufficient support of the prediction bounds, making it more difficult for the learning algorithm to achieve good generalization [26]. Therefore, excluding these values not only reduces the number of small intermittent areas but also has a positive impact on the model's generalization (the model's generalization is inherently weak under the condition of small samples). Admittedly, the model fitting in this study has some defects in the elimination of discrete value. Discrete rejection based on corrosion rate alone would have resulted in the incomplete rejection of discrete values as well as possible rejection of non-discrete values. The reason why this culling is not based on the feature values is that the data set has multiple dimensions, and discrete culling of each feature may lead to a small sample size, thus leading to a larger prediction errors. In the model applicability evaluation, when the random seed of the control partition is 666, the prediction error increases significantly, which is likely due to the discrete values in the test set being mixed with those that have not been removed. Therefore, when the study sample is small, more rigorous rules for determining the discrete values should be established. In addition, when building a prediction model based on small data set, when summarizing the model, the verification data should be kept as small as possible with respect to the training set.

The research shows that dimension reduction is helpful to improve the prediction accuracy of the model. In our experiment, when 90% of the information dimension are retained, the prediction effect is the best, while when other information dimensions are retained, the result fluctuates greatly. This shows that although PCA is a common method for dimensionality reduction of small sample data [69], there is no uniform standard for how much information should be retained in different scenarios, which should be selected according to the actual situation.

In the experiment of the influence of segmentation ratio on the prediction error, we found that with the decrease of the training set ratio, the prediction error becomes larger. This is because when the number of training sets decreases, over-fitting is more likely to occur, which will lead to poor generalization ability of the model. This is consistent with the current mainstream research [67], so whether it is a small sample condition or not, increasing the amount of training data will always help to predict the results.

Although many researchers believe that ensemble learning is superior to individual learners and helps solve small-sample problems [27], the results from Fig. 11 shows this is not necessarily the case. Windeatt [70] put forward that, to obtain a good integration, the conditions of accuracy and diversity must necessarily be satisfied. Therefore, when the dataset is small and contains some discrete values, ensemble learning methods should not be used blindly. In such cases, if the bagging algorithm is not able to achieve good prediction results, it is worth trying individual learners.

Fig. 12 shows that the predicted MAE value of the Norsok M506 model was 0.1322, which is significantly larger than that of the ensemble learning algorithm. This indicates that ensemble learning still performs better than traditional empirical models even under small sample conditions. In this case, the results from the Norsok M506 model are relatively conservative compared to other traditional models, which results in larger predicted corrosion rates and thus higher prediction errors [9].

The results of Fig. 13 show that the floating range of the prediction error becomes smaller as the number of combinations increases in the Stacking combination model. This means that if one wants to integrate with the Stacking model, it is not better to have more model combinations, rather a smaller number of combinations is more likely to make the combined prediction error decrease. In most cases of this part of the experiment, the prediction performance after using the stacking combination did not outperform the prediction performance of the underlying ensemble model, indicating that using the Stacking algorithm does not further improve the predictive power of the model. As stated by other researchers [71], improving prediction performance using Stacking models requires that the models involved in the integration are differentiated and have good performance, and these conditions cannot be met in the small sample condition. The results in this section show that although the prediction error values of the bagging + KNN framework are still largely under small sample conditions, it is more suitable for prediction under small sample conditions than the traditional model and the more complex Stacking model.