According to the probability theory and information theory, the mutual information (MI) of two random variables is a measurement of the mutual dependence between them, which is used to evaluate the amount of information contributed by the appearance of one random variable to the appearance of another random variable. Different from the correlation coefficient, mutual information is not limited to real-valued random variables, and generally determines the similarity of the joint distribution p(x,y) and the product of the decomposed marginal distribution p(x)p(y). In fact, MI is a measure of mutual dependence between two sets of events.

Formally, the MI of two discrete random variables X and Y can be defined as:

(1) I(X;Y)=∑y∈Y∑x∈Xp(x,y)log⁡(p(x,y)p(x)p(y)),where p(x,y) is the joint probability distribution function of X and Y, and p(x),p(y) are the marginal probability distribution functions of X and Y, respectively.

Intuitively, MI measures the information shared by X and Y: it measures the degree to which one of these two variables is known to reduce the uncertainty of the other. For example, if X and Y are independent of each other, we can conclude that X does not provide any information to Y, and vice versa, so their MI(X;Y)=0. At the other extreme, if X is a deterministic function of Y, and Y is also a deterministic function of X, then all the information passed is shared by X and Y: the value of X determines the value of Y, and vice versa. Therefore, in this case, the MI and Y (or X) alone contain the same uncertainty, which is called the entropy of Y (or X). Moreover, this MI is the same as the entropy of X and the entropy of Y. Indeed, a very special case of this situation is when X and Y are actually the same random variable.

In this study, there is a hypothesis that samples that are frequently grouped into a category, usually have greater MI with this category. Then we introduce the general feature selection method based on MI. The steps of the MI feature selection procedure: 1) divide the dataset; 2) sort these features according to their MI values; 3) select the top n features and adopt prescribed machine learning model to train; 4) evaluate the error rate of feature subset on the testing dataset. Through the MI-based feature selection, these features related to the samples from the process can be sorted by correlation, but how many features need to be selected depends on prior knowledge.

The multi-mode process probabilistic fault classifier is used to estimate the class-posterior probability p(y|x) (i.e., to predict a class label yte for a test input point xte), which can be further utilized in the fault detection of nonlinear multi-mode processes. The class-posterior probability p(y|x) is formulated as:

(4) p(y|x;θy)≡∑Nten=1θy,nK(x,xnte),where θy=(θy,1,…,θy,Nte)⊤ is the Nte-dimensional parameter vector and K(x,x′) is a kernel function, which is used to extend x to a higher-dimensional state space, thus making the simple linear LSPC classifier applicable in the original low-dimensional probabilistic classifier p(y|x). Then, the commonly used Gaussian kernel is defined as:

(5) K(x,x′)=exp⁡(−∥x−x′∥22σ2),where σ denotes the Gaussian kernel width. The goal is to minimize the following squared error performance index Jy, by determining the LSPC parameter θy:

(6) Jy(θy)≡12∫(p(y|x;θy)−p(y|x))2pte(x)dx=12∫p(y|x;θy)2pte(x)dx−∫p(y|x;θy)p(y|x)pte(x)dx+C=12θy⊤Qθy−qy⊤θy+C,where C is a constant, and the elements of the Nte×Nte matrix Q and the Nte-dimensional vector qy=(qy,1,…,qy,Nte)⊤are defined as:

(7) Qn,n′=∫K(x,xnte)K(x,xn′te)pte(x)dx,

(8) qy,n≡∫K(x,xnte)p(y|x)pte(x)dx.

Then, we can formally approximate Q and qy, using the aforementioned importance weight sampling method, i.e., the importance weight Eq. (2). As a matter of fact, Q and qy can be expressed as follows:

(9) Qn,n′=∫K(x,xnte)K(x,xn′te)ptr(x)w(x)dx,

(10) qy,n=∫K(x,xnte)p(y|x)ptr(x)w(x)dx=p(y)∫K(x,xnte)ptr(x|y)w(x)dx,where ptr(x|y) denotes the training input density for a certain class y. Then, using the training samples {(xntr,yntr)}n=1Ntr, Q and qy can be estimated in accordance with Eqs. (9) and (10):

(11) Q^n,n′≡1Ntr∑Ntrn′′=1K(xn′′tr,xnte)K(xn′′te,x ′te)w(xn′′tr)ν,

(12) q^y,n≡1Ntr∑n′:yn′tr=yK(xn′tr,xnte)w(xn′tr)ν,where the class-prior probability p(y) is estimated by Ntr(y)/Ntr, Ntr(y) represents the number of training samples with label y, and the flattening parameter 0≤ν≤1. As argued in [21], there exists the problem of bias-variance trade-off (i.e., the importance weights influence the bias and variance of the IWLSPC model). If ν→1, the bias gets smaller, but the variance tends to be larger; yet if ν→0, then the bias is larger, but the variance is smaller. The problem can be formally described as:

(13) θ^y≡argminθy[12θy⊤Q^θy−q^y⊤θy+λ2θy⊤θy],where (λ/2)θy⊤θy is a regularization term with the parameter λ≥0 avoiding over-fitting. Then, the IWLSPC solution is given analytically as:

(14) θ^y=(Q^+λINte)−1q^y,where INte denotes the Nte-dimensional identity matrix. Considering that the class-posterior probability is non-negative by its definition, then the solution p^(y|x) can be modified as:

(15) p^(y|x)≡1Zmax(0,∑Nten=1θ^y,nK(x,xnte)),

Finally, using the learned class-posterior probability p^(y|x), we can predict the class label yte of a new test sample xte by

(16) y^te≡argmaxyp^(y|xte),

If there does not exist covariate shift during a steady-state operation of the nonlinear process, i.e., pte(x)=ptr(x), the IWLSPC is actually equivalent to the traditional LSPC method.

In the field of process system engineering, Tennessee Eastman (TE) process is a widely used Benchmark for process control [27,28] and fault monitoring [29–31]. The TE process is an open simulation platform in chemical engineering and process industry, and it’s developed by Eastman Company from the United States. With the support to the dynamic simulation of a chemical reaction process, various features in the complex industrial process system can be well simulated with TE process. The data generated by the TE has complex characteristics (e.g., time-varying, strong coupling, and high nonlinearity). Therefore, it is widely adopted in fields such as system optimization, process monitoring and fault diagnosis, and it also can be used to verify the control and fault diagnosis for industrial chemical processes.

The TE process consists of five units: 1) stripper reboiler, 2) condenser, 3) flash separator, 4) two-phase exothermic reactor and 5) circulating compressor (as shown in Fig. 1). The process contains a total of 52 variables, in which 22 are process variables, 18 are component variables and the other 12 are manipulated variables.

The TE process contains four gaseous raw materials (A, C, D and E), two liquid products (G and H), and by-products (F) and inert gas (B). The irreversible, exothermic chemical reaction in TE process is as follows:

(17) A(g)+C(g)+D(g)→G(l)A(g)+C(g)+E(g)→H(l)A(g)+D(g)→F(l)3D(g)→2F(g)

The gas components (A, C, D and E) and the inert component (B) are fed into the reactor, thus forming the liquid products (G and H), where the reaction rate obeys the Arrhenius function of the reaction kinetics. Coming out of the reactor, the product steam is condensed to liquid and transferred to a liquid separator, from which a string of steam would be resent to the reactor through the compressor. Some recirculation shunt should be dismissed to avoid accumulating the by-products and inert components in the reaction. The condensed products from the separator (Steam 10) are sent to the stripper. Steam 4 is applied to remove the remaining reagents, in Steam 10, mixed with the recirculating steam from Steam 5. The products (G and H) are sent to the downstream process. Most of the by-products are evacuated as gas in the vapor-liquid separator. According to the different component mass ratios of G and H in the product, the TE process supports 7 operation modes/working conditions. The simulation platform can set 21 fault modes, e.g., the fault modes of step type, random change type, slow drift type, valve sticky type, valve stuck type and unknown types.

The TE process supports a variety of operation modes and a total of 28 prescribed fault types are given in detail. The simulated multi-mode data can be obtained from the TE process. A number of typical fault conditions in TE simulation dataset are selected, i.e., the TE process operation data without fault, and fault status #4, #10, #13, #16, #18 and #28. Then, the implementation process of the algorithm is described. We continuously extract a group pf time-series samples in different conditions (including different conditions with 6 faults and the normal condition). As shown in Table 1, two different operation modes are set up in this experiment.

The training dataset contains the samples obtained in mode 1, in which the data in normal condition and in fault condition is 2400, respectively. Meanwhile the testing dataset contains samples obtained in mode 2, in which the data in normal condition and in fault condition is 200, respectively. The data generated in mode 1 is used to train the binary fault classification model, and the data generated in mode 2 is used to verify the TE process fault classification (i.e., classification accuracy). Since the training and testing datasets are obtained in different operation modes in TE process, it can be considered that the probability distribution of input samples in training datasets is different from that of the testing dataset. There exists the covariate shift phenomenon between the training dataset in operation mode 1 and the testing dataset in operation mode 2. To reduce the amount of data and increase the efficiency of the method, it is necessary to select some variables from all process variables, which have strong correlations with the fault variables. We introduce the mutual information for such a task. The MI is equivalent to dimensionality-reduction operation on the original data.

In the experiment, the MI-based variable selection works as follows: 1) re-arrange the variables in the training and testing dataset in the order of MI value; 2) choose 25 variables with largest MI value in the training and testing dataset, respectively; 3) observe the selected 25 variables, and select all the common variables among them; 4) calculate the MI value between the selected common variables, respectively, and remove these variables with low MI value (there is weak correlation or negative correlation, this operation is to prevent the occurrence of negative transfer); 5) delete other variables, the remaining variables are exactly the process variables we need, and these variables selected can be used to train the model of fault classification.

The MI-based pre-processing can not only reduce the computational workload, but also make the selected data more effective. On the other hand, in addition to data redundancy, not all the data are valid for fault classification and fault detection model. Some data have stronger correlation with the fault, and that the correlation between process variable and fault variable is not always the same. Besides, there is some data that cannot indicate whether there exist some faults in the process system, and the data is invalid for fault detection. Therefore, with the MI-based feature selection, we can extract more useful information for fault classification model and form the reorganized input dataset for training process. In this way, the data pre-processing can greatly eliminate the invalid samples and variables, and retain the most relevant variables for fault detection and classification.

The training and testing dataset processed by MI feature selection is input to IWLSPC for fault detection and classification. Specifically, we perform the testing with binary fault classification and multiple fault classification. In addition, the IWLSPC is compared with other existing methods in detail.

Firstly, we apply the proposed method to the binary fault detection and classification of TE process. In fact, if the binary fault situation refers to the two categories of having some specific faults or no fault, then binary fault classification is equivalent to the judgement of the process fault, i.e., exist or not exist. Therefore, to improve the comparability of different binary fault classification, we classify all the fault categories with the case without fault, that is, the designed algorithms should be able to distinguish whether there exist some faults or not in the TE process under covariate shift. The detailed performance comparison of binary fault classification is shown in Table 2. Taking no fault and specific fault as the two labels of binary fault classification, the results presented are the accuracy of fault classification with the testing dataset. In addition, the results marked as  ∗ in the table are the results of data-driven fault classification model trained by the validation dataset with particularly few samples (only 200 samples), the results are used to demonstrate the effectiveness of importance-weighted transfer learning under covariate shift.

First, we sorted the correlation of the characteristics related to the fault according to their MI values, and then selected the most relevant process variables. The pre-processing can not only reduce the amount of data, but also significantly improve the effectiveness of the data training. We compared the performance of extracting feature variables with/without MI. As shown in Table 2, by introducing the MI to the extraction of variables, the IWLSPC can better detect whether a fault exists or not.

Because the training and testing datasets we use are obtained in different operation modes, the process dynamic characteristics and fault-related information contained in the training and testing datasets are inconsistent. Once the training process completes, the traditional machine learning based classifications cannot adapt to the change of operation modes in the testing dataset according to the sample distribution. The importance weighting used in this study considers the probability distribution of the test input samples during the training process, and it carries out the weighted re-organization of the training input samples based on the importance weights, which can improve the accuracy of fault classification on the testing dataset. As shown in Table 2, for most types of faults, the performance of the MI-based IWLSPC is better than that of the traditional LSPC, and the total average performance of 28 fault types is also better than the MI-based LSPC. The result shows that the IWLSPC can better detect whether there exists a fault under covariate shift. It achieves superior performance to detect whether there exist some specific faults or there do not exist faults for the current state.

Furthermore, we compared the effectiveness of commonly used machine learning based classification algorithms and applied them to the binary fault classification for the TE process. As shown in Table 2, we can see that in most cases, the IWLSPC achieves excellent fault classification performance. Although SVM has some advantages in some cases, IWLSPC achieves better fault classification performances in most of the fault types. However, the reliability of MI + IWLSPC can be guaranteed, which is exactly why MI-based IWLSPC outperforms for the total average fault classification accuracy. It is proven that LSPC and IWLSPC can not only be used to binary fault classification, but also can be applied to classify multiple faults simultaneously.

In order to demonstrate the effectiveness of importance-weighted transfer learning under covariate shift. We have also conducted experiments with extremely small samples, in which we directly train the fault classification model on the small dataset. Specifically, we trained the afore-mentioned fault classification models on the validation set with only 200 samples. This kind of small sample training is very practical for nonlinear multi-mode industrial processes, because some of the actual industrial processes usually do not have sufficient historical data under working conditions. The results marked with ^* in Table 2 show that the performance of directly training on the validation dataset is poor, because there is not sufficient data for the training process. However, with the importance-weighted transfer learning method, the data-driven fault classification model achieves satisfactory performance even on a small sample verification dataset where the covariate shift exists. It confirms the effectiveness of the importance-weighted transfer learning method.

The probabilistic-based classification method for industrial process fault classification works well. In this study, we compared the effectiveness of the BP neural network, traditional LSPC and IWLSPC that is suitable for dealing with covariate shift. It is noted that we used MI to extract feature variables in both methods, thus mainly focusing on the difference among BP neural network, LSPC and IWLSPC, i.e., the effectiveness of the importance-weighted method. The detailed comparison is shown in Table 3.

In Table 3, the effectiveness of the IWLSPC algorithm for multi-fault classification of the TE process under covariate shift is detailed. According to Table 3, we can conclude that the IWLSPC achieves better performance for the simultaneous classification for multiple faults in the TE process. Especially in some cases, the accuracy of the IWLSPC is much higher than that of the traditional LSPC, which confirms the effectiveness of IWLSPC. In addition, it is noted that although the BP neural network is extremely effective in the classification of certain types of faults, the classification performance of this method under different conditions has serious fluctuations, resulting in a very poor average performance.