As the candidate kernels and RPs of the LSSVM model are respectively denoted as {Kerk}k=1K and {Regr}r=1R , the candidate learning parameters can be used to obtain the following matrix:

(1) Mpara=[[Ker1,Reg1]⋯[Ker1,Regr]⋯[Ker1,RegR]⋮⋮⋮⋮⋮[Kerk,Reg1]⋯[Kerk,Regr]⋯[Kerk,RegR]⋮⋮⋮⋮⋮[KerK,Reg1]⋯[KerK,Regr]⋯[KerK,RegR]]J

where [Kerk,Regr] is the jth element of learning parameter matrix Mpara , in which Mparaj=[Kerk,Regr] with j=1,⋯,J . Especially, J=K×R is the element number of matrix Mpara , that is, the number of LSSVM sub-sub-models.

Taken the jth pair [Kerk,Regr] of learning parameters for an example, the process of the sub-sub-model is built as follows. First, nonlinear mapping function φ(xn) is used to map {xn}n=1N into a higher dimensional feature space. Then, the optimization problem is solved by the LSSVM algorithm:

(2) {minWj,bj⁡OjLS−SVM=12wjTwj+12Reg∑n=1Nζjn2s.t:y^jn=wjTφ(xnj)+bj+ζjn

where wj is the weighted coefficient, bj indicates the bias, and ζjn denotes the prediction error of the nth sample. Thus, the Lagrange method can be employed to solve the optimization problem (2),

(3) Lj(wj,bj,ζj,βj)=12wjTwj+12∑n=1N(ζjn)2−∑n=1Nβjn[wjTφ(xjn)+bj+ζjn−y^jn]

where βj=[βj1,⋯,βjn,⋯,βjN] is the Lagrange operator vector, and ζj=[ζj1,⋯,ζjn,⋯,ζjN] is the prediction error vector. Taken the derivation of the parameters in Eq. (3), the optimum solution can be gotten by

(4) ∂Lj∂wj=0,∂Lj∂bj=0,∂Lj∂ξj=0,∂Lj∂βj=0,

(5) Ωkerj(x,xn)=<φ(x)⋅φ(xn)>=fkerj(x,xn,Kker),

where Ωkerk(⋅) is the kernel function with parameter Kerk .

Accordingly, the above problem becomes a linear equation system:

(6) [01⋯11fkerj(x1,x1,Kker),+1Regr⋯fkerj(x1,xN1,Kker)⋮⋮⋮⋮1fkerj(xN,x1,Kker)⋯fkerj(xN,xN,Kker)+1Regr]⋅[bjβj1⋮βjN]=[1y1⋮yN]

By solving the above system, βj and bj are obtained. Then, the LSSVM sub-sub-model can be denoted as

(7) y^subsubj=∑n=1Nβjn⋅Ωkerj(x,xn)+bj

For a concise expression, (7) can also be expressed as

(8) y^subsubj=fsubsubj(x;Mparaj)=fsubsubj(xj;Kerk,Regr)=fsubsubk,r(⋅)

Therefore, all these candidate sub-sub-models are denoted as {fsubsubj(⋅)}j=1J , and the prediction outputs are {y^subsubj}j=1J .

The prediction outputs of these candidate sub-sub-models can be rewritten as

(9) {y^subsubj}j=1J={ysubsub1,1,⋯,ysubsub1,r,⋯,ysubsub1,R,⋮ysubsubk,1,⋯,ysubsubk,r,⋯,ysubsubk,R,⋮ysubsubK,1,⋯,ysubsubK,r,⋯,ysubsubK,R}

where y^subsub1=fsubsub1,1(⋅) , y^subsubj=fsubsubk,r(⋅) , y^subsubJ=fsubsubK,R(⋅) .

In (9), the kth row contains all the sub-sub-models with the same Kerk and different {Regr}r=1R . Thus, {y^subsubj}j=1J can be rewritten as

(10) {y^subsubj}j=1J={{fsubsub1,r(⋅)}r=1R,⋮{fsubsubk,r(⋅)}r=1R,⋮{fsubsubK,r(⋅)}r=1R}

The candidate SEN-sub-models are built for each row of (10) by selecting and combining the candidate sub-sub-models based on different {Regr}r=1R . For example, the kth row {fsubsubk,r(⋅)}r=1R can construct the kth SEN-sub-model fSENsubk(⋅) through the same Kerk and {Regr}r=1R . So the prediction output (y^SENsubk)n of the nth modeling sample can be calculated by

(11) (y^SENsubk)n=fSENsubk(⋅)=∑rksel=1Rkselwrkselsel⋅(y^rkseln)sel=∑rksel=1Rkselwrkselsel⋅fsubsubk,rksel(xn,Kerk,Regrksel)

where fsubsubk,rksel(⋅) , Regrksel , and (y^rksell)sel are the selected rkselth sub-sub-model, its RP, and the prediction output of the nth modeling sample, respectively; Rksel indicates the ensemble size, and wrkselsel is the weighted coefficient of the selected sub-sub-models, which is calculated by

(12) wrkselsel=1σrksel2∑rksel=1Rksel1σrksel2

where σrksel is the standard variance of the prediction outputs {(y^rkseln)sel}n=1N form the selected rkselth sub-sub-model. As the ensemble size Rksel and the weighted algorithm are pre-set and determined, the ensemble sub-sub-models for constructing the kth SEN-sub-model fSENsubk(⋅) can be obtained by solving the following optimization problem:

(13) (y^SENsubk)n=∑rksel=1Rkselwrkselsel⋅(y^rkseln)sel=∑rksel=1Rkselwrkselsel⋅fsubsubk,rksel(zn,Kerk,Regrksel)s.t.{min{{RMSE((y^SENsubk,)Rksel)}Rksel=2R}RMSE((y^SENsubk,)Rksel)=1N∑n=1N(yn−∑rksel=1Rkselwrkselsel⋅fsubsubk,rksel(zn,Kerk,Regrksel))22≤Rksel≤R,∑rksel=1Rkselwrkselsel=1,0<wrkselsel<1{fsubsubk,rksel}rksel=1Rksel∈{fsubsubk,r(⋅)}r=1R{Regrksel}rksel=1Rksel∈{Regr}r=1R

where the root mean square error (RMSE) is used to evaluate the generalization performance of the SEN-sub-model with ensemble size Rksel .

In order to solve the above problem, the optimized ensemble sub-sub-models and their weighted coefficients are obtained through repeating the BBSEN optimization algorithm R−2 times [11], and this process is expressed by

(14) {{fsubsubk,r(x;Kerk,Regr)}r=1R,{yn}n=1N}=fBBSEN({fsubsubk,rksel(x,Kerk,Regrksel)}rksel=1Rksel,{wrkselsel}r=1Rksel)

where {fsubsubk,r(⋅)}k=1Kksel and {wsubsubk,r}k=1Kksel are the selected ensemble sub-sub-models and their weighted coefficients, respectively; and Rksel is the number of the selected sub-sub-models.

For simplification, all the RPs of the kth SEN-sub-model are denoted as (Regk)SENsub={Regrksel}rksel=1Rksel . After repeating the above modeling procedure K times, all the SEN-sub-models with KPs {Kerk}k=1K are obtained. And then these candidate SEN-sub-models and their prediction outputs can be denoted as {fSENsubk(⋅)}k=1K and {y^SENsubk}k=1K , respectively. Therefore, different RPs are adaptively selected in the construction process of these SEN-sub-models.

Based on the above sub-sections, the SEN-sub-models with the same KPs and different RPs are obtained. Then, Equation (10) can be transformed into the following form.

(15) {y^subsubj}j=1J={{fsubsub1,r(⋅)}r=1R,⋮{fsubsubk,r(⋅)}r=1R,⋮{fsubsubK,r(⋅)}r=1R}⇒⇒{{fsubsub1,r1sel(⋅)}r1sel=1R1sel,⋮{fsubsubk,rksel(⋅)}rksel=1Rksel,⋮{fsubsubK,rKsel(⋅)}rKsel=1RKsel,}={fSENsub1(⋅),⋮fSENsubk(⋅),⋮fSENsubK(⋅),}={fSENsubk(⋅)}k=1K={y^SENsubk}k=1K

In this context, BBSEN is further employed to obtain the prediction output y^SENksel of SEN-model fSENsubksel(⋅) by optimally selecting and combining SEN-sub-models based on KPs {Kerk}k=1K . Hence, the optimization description is presented as

(16) y^SENn=∑ksel=1KselwkselSENsel⋅(y^SENksel)n=wkselSENsel⋅fSENsubksel(⋅).t.{ RMSE(y^SENKsel)=1N∑n=1N(yn−∑ksel=1KselwkselSENsel⋅(y^SENksel)n)2=1N∑n=1N(yn−∑ksel=1KselwkselSENsel⋅fSENsubksel(zn,Kerksel,(Regksel)SENsub))22≤Ksel≤K,∑ksel=1KselwkselSENsel=1,0<wkselSENsel<1{fSENsubksel}ksel=1Ksel∈{fSENsubk(⋅)}k=1K{Kegksel}ksel=1Ksel∈{Kegr}r=1K

where (Regksel)SENsub is the RP set for constructing the kselth SEN-sub model with size RSENsubksel ; Ksel indicates the number of selected ensemble SEN-sub-models (ensemble size) for building the SEN-model; fSENsubksel(⋅) and Kegksel represent the kselth ensemble SEN-sub-model and its KP, respectively; y^SENKsel denotes the prediction outputs of the nth samples of the SEN-model with ensemble size Ksel ; and wkselSENsel is the weighted coefficient of the kselth ensemble SEN-sub-model, which is calculated by

(17) wkselSENsel=1σksel2∑ksel=1Ksel1σksel2

where σkksel is the standard variance of the prediction outputs {(y^SENksel)n}n=1N based on the selected kselth ensemble SEN-sub model. Consequentially, the above modeling process can be represented by

(18) {fSENsubk(x,Kerk,(Regk)SENsub)}k=1K{yn}n=1N}⟶BBSEN{{fSENsubksel(x,Kerksel,(Regksel)SENsub)}ksel=1Ksel{wkselSENsel}ksel=1Ksel

where {fSENsubksel(⋅)}ksel=1Ksel and {wSENsubksel}ksel=1Kksel are the selected ensemble SEN-sub-models and their weighted coefficients, respectively; and Kksel is the number of selected SEN-sub-models.

Accordingly, the SEN model fSENKksel(⋅) with ensemble size Kksel can be denoted as

(19) y^SENKsel=∑ksel=1KselwkselSENsel⋅fSENsubksel(⋅)=∑ksel=1KselwkselSENsel⋅(∑rksel=1Rkselwrkselsel⋅fsubsubk,rksel(z,Kerksel,Regrksel))ksel

In Eq. (19), the SEN-model fSENKsel(⋅) selects both the kernel and RPs adaptively from the candidates. Moreover, the ensemble size of SEN-sub-model is also decided adaptively in the SEN modeling process.

Supposed that Kksel∈[2,K) is hold, and then (K−2) SEN models can be constructed. However, the complexity of the SEN model significantly increases with the ensemble size. In order to solve this problem, the model complexity in this paper is measured by the number of single LSSVM models. Thus, the complexity of the best sub-sub-model, the kth SEN-sub-model fSENsubk(⋅) , and the SEN model fSENKsel(⋅) are 1, Rksel , and ∑ksel=1KselRSENsubksel , respectively. To make a tradeoff between prediction accuracy and model complexity, the metric index for selecting the suitable soft sensor model is defined by

(20) ξSENKsel=λRMSE(fSENKsel(⋅))∑Ksel=2K−1RMSE(fSENKsel(⋅))+(1−λ)∑ksel=1KselRSENsubksel∑Ksel=2K−1∑ksel=1KselRSENsubksel

where λ is the coefficient between 0 and 1.

According to the above metric index, the SEN-model with the lowest RMSE is selected as the final DXN soft measuring model. Thus, the final prediction output is obtained by

(21) y^=fsel(y^SEN2,⋯,y^SENKsel,⋯,y^SENK−1)s.t.min{ξSEN2,⋯,ξSENKsel,⋯,ξSENK−1}

where fsel(⋅) represents the final AMLSEN-LSSVM model. In this way, the learning parameters and ensemble sizes of different SEN layers are selected adaptively.

In order to present preliminary results, two UCI benchmark datasets, Boston housing and concrete compressive strength, are used to validate the proposed method, which are listed as follows.

For Boston housing data, the inputs include: (1) the per capita crime rate of the town (CRIM);(2) the proportion of residential land zoned over 25,000 sq. ft. (ZN); (3) the proportion of non-retail business acres per town (INDUS); (4) the Charles River dummy variables (CHAS); (5) the nitric oxide concentrations (NOX); (6) the average number of rooms per dwelling (RM); (7) the proportion of owner-occupied units built before 1940 (AGE); (8) the weighted distances to five employment centers of Boston (DIS); (9) the index of radial highway accessibility (RAD); (10) the full property tax rate per $10,000 (TAX); (11) the pupil–teacher ratio of the town (B); (12) the lower status of the population (LSTAT); and (13) the median value of owner-occupied homes per $1000 (MEDV). The output is the housing values of the suburbs of Boston with data size 506.

For concrete compressive strength data, the inputs include: (1) cement; (2) blast furnace slag; (3) fly ash; (4) water; (5) superplasticizer; (6) coarse aggregate; (7) fine aggregate of the various ingredients of concrete placement per cubic meter; and (8) conserved days. The output is concrete compressive strength with data size 1030.

According to the candidate learning parameters, 14 × 17 = 238 sub-sub-models based on the LSSVM are constructed. Correspondingly, the BBSEN method is employed to build 14 SEN-sub-models. The statistical results are shown in Tab. 2.

Tab. 2 illustrates that: (1) the best SEN-sub-model of Boston housing dataset has KP 100 and RPs {25, 50, 100, 200, 400, 800, 1600, 3200, 6400, 12800, 25600, 51200, 102400} with RMSE 3.1146 and an ensemble size 14. The best sub-sub-model picks up KP 100 and RP 200 with RMSE 3.1953, and the best EnAll-sub-sub-model possesses the same KP with RMSE 3.4288;(2) the best SEN-sub-model of concrete compressive strength data has the KP 100 and RPs {25, 50, 100, 200, 400, 800, 1600, 3200, 6400, 12800, 25600, 51200, 102400} with RMSE 7.368 and an ensemble size 14. The best sub-sub-model picks up KP 100 and RP 800 with RMSE 7.7284, and the best EnAll-sub-sub-model possesses the same KP with RMSE 9.8368;(3) different KPs have different effects on the prediction performance of the SEN-sub-models, so that parts of the best sub-sub-models have lower RMSEs than those of the SEN-sub-models. Thus, setting a suitable RP is very important to demonstrate the relationship among RMSEs, ensemble sizes, and different types of models with the single KP, which are shown in Figs. 2 and 3.

Figs. 2 and 3 show that suitable learning parameters are crucial. From this perspective, the ensemble size with the best prediction performance of the Housing and Concrete data are 14 and 6 in terms of the same KP 100, respectively. However, the ensemble size of the SEN-sub-model does not increase with the KPs after 4000.

For the 14 SEN-sub-models, the BBSEN method is used again to obtain the SEN-model with a different ensemble size, and the detailed statistical results of the SEN-models are illustrated in Tab. 3.

Tab. 3 shows that:(1) in the Boston housing dataset, the SEN-model with an ensemble size 9, KPs {0.1, 1, 100, 1000, 2000, 4000, 6000, 8000, 10000}, has the best prediction performance (RMSE 3.3781) among all the SEN-models, which is higher than that of the best SEN-sub-mode, but lower than that of the EnAll-SEN-sub-model;(2) in the concrete compressive strength dataset, the SEN-model with an ensemble size 9, KPs {0.1, 1, 100, 1000, 2000, 4000, 6000, 8000, 10000}, has the best prediction performance (RMSE 8.2221) among all the SEN-models. However, it is larger than that of the best SEN-sub-mode, but lower than that of the EnAll-SEN-sub-model; (3) the best SEN-sub-models in the above datasets has the best prediction performance. Moreover, the prediction performance of different SEN-models is not further improved with the increase of the ensemble size. The relationship between RMSEs and the ensemble sizes of SEN models is shown in Figs. 4 and 5.

The above results show that it is essential to make a tradeoff between prediction performance and model complexity. The proposed indices of different SEN-models with λ = 0.5 are shown in Tab. 3, and the metric index curves of the SEN-model with different λ values are shown in Fig. 6.

Seen form Fig. 6, the optimal value of the metric index shifts to the left with increasing coefficient ( λ ) value. Thus, it can be determined by the practical requirement. The prediction curves of the best SEN-sub-model (SENsub) and the best SEN-model (SenSENsub) on the minimum prediction errors as well as the ensemble all SEN-sub-model (EnSENsub) are shown in Figs. 7 and 8, respectively.

The above results show that the proposed method can model the two UCI benchmark datasets effectively, which can make an adaptive selection of the learning parameters.

The proposed method is compared with the PLS, KPLS, GASEN-BPNN, and GASEN-LSSVM approaches. Concretely, the number of latent variables (LVs) and kernel LVs (KLVs) are decided by the leave-one-out cross-validation, and the number of hidden nodes of the GASEN-BPNN is set to two times plus one of the original inputs’ number. Thus, the structure of BPNN is 13-27-1 for the housing data, and 8-16-1 for concrete data. Meanwhile, the learning parameters of GASEN-LSSVM are used as ones of the best sub-sub-model in the proposed method. For GASEN-BPNN and GASEN-LSSVM, the modelling process is repeated 10 times to overcome their randomness.

Tab. 4 shows that the proposed method has the best prediction performance without any disturbance.

Although the GASEN-based prediction results are disturbed within a certain range due to the random initialization of the input weights and the bias of BPNN and population GA, the PLS/KPLS methods can activate the extracted latent variables to construct a linear or nonlinear model with a single KP for stabilization.