The diagnosis of faults in rotating machinery, especially in their initial stages, has gained considerable attention due to its critical importance in averting potential catastrophic incidents and allowing for adequate maintenance time [1–3]. Vibration-based analysis has been widely applied because of its inherent advantages in revealing the characteristic features of mechanical faults [4–6].

From the perspective of mechanical fault mechanisms, when critical rotating components such as rolling bearings, rotors, and gearboxes exhibit localized defects, the transmission path from the damaged meshing position to the fixed accelerometer undergoes changes in its evolution process [7,8]. Simultaneously, strong vibration responses from other mechanical components and significant background noise can render these fault impacts extremely weak. Therefore, appropriate signal processing techniques are essential prerequisites for identifying fault features [9].

Most signal processing techniques, particularly methods such as Discrete Wavelet Transform (DWT) and Wavelet Packet Transform (WPT) [10], hold great promise for identifying faults in rotating machinery. When diagnosing gear faults, DWT’s inherent limitations lead to low precision in diagnosing transient frequencies [11–13]. To address these limitations, Wavelet Packet Transform (WPT) is commonly utilized to reveal high-frequency regions of transient components [14]. However, both WPT and DWT are plagued by sub-sampling, which reduces time-based resolution. In a study by Hong et al. [15], the combination of Hilbert spectrum with Maximum Overlap Discrete WPT was used to analyze vibration signals from gear faults. Wang et al. [16] proposed a denoising method based on Dual-Tree Complex Wavelet Transform (DTCWT) for fault diagnosis in rotating machinery. Subsequently, DTCWT has been applied to analyze vibration signals from gearboxes [17,18] and biomedical signals [19]. Cai et al. [20] introduced a new method for sparse signal decomposition using Time-Frequency Hermite Transform (TQWT). However, traditional wavelet filters designed for linear systems perform poorly on nonlinear systems, especially in detecting transient components with high-frequency characteristics, due to low resolution and poor shift invariance.

Flexible Analytical Wavelet Transform (FAWT) is a relatively recent concept initially explored by Bayram [21]. FAWT offers several advantages over traditional binary wavelet filters, including flexible selection of time-frequency windows, effective shift invariance, the capability to decompose complex signals based on oscillatory behavior, and the ability to capture weak fault features. FAWT allows for arbitrary sampling rates in both low-pass and high-pass channels, providing a flexible partitioning scheme where scaling and translation factors can be easily adjusted. Additionally, by fine-tuning the width of the frequency transition band, FAWT can achieve optimal oscillatory bases for detecting various oscillatory pulses. Therefore, the application of the FAWT method in fault diagnosis for rotating machinery proves advantageous in improving the detection performance of subtle fault features [22–24].

Although FAWT holds great potential for signal decomposition, signals obtained from rotating machinery often exhibit nonlinear and non-stationary characteristics, posing challenges for the extraction of fault-related features. Entropy serves as a widely used metric for quantifying the randomness and dynamic changes in real dynamic systems, and it finds extensive application in fault diagnosis for rotating machinery [25]. Specifically, Permutation Entropy (PE), as a nonlinear parameter measuring the randomness and dynamic changes in time series, has been proven highly effective in detecting the dynamic characteristics of vibration signal time series. PE boasts advantages such as simplicity in calculation, rapid computation, robustness to nonlinear monotonic transformations, and stability, making it efficient for detecting and amplifying dynamic changes in vibration signals. When employing PE for feature extraction, defining events and comparing the complexity of time-domain data are crucial for its effective application [26,27].

In recent years, quantum theory has experienced rapid development, emerging as a revolutionary and enigmatic theoretical framework. Diverging significantly from traditional modes of information representation, quantum theory, with its principles of superposition, coherence, and entanglement, accurately captures many objective regularities [28]. In this paper, quantum theory is introduced into permutation entropy to form Nonlinear Quantum Permutation Entropy, serving as a fault feature parameter for rotating machinery. Leveraging the advantages of the novel expression offered by quantum theory and the dynamic characteristics of permutation entropy, Nonlinear Quantum Permutation Entropy (QPE) enables more sensitive and accurate characterization of subtle faults in rotating machinery.

In conclusion, this paper presents a novel method for extracting weak features in rotating machinery. The approach first utilizes the FAWT to decompose vibration signals collected from rotating machinery, obtaining several sub-band signals. Nonlinear Quantum Permutation Entropy of each sub-band signal is then extracted as fault characterization feature vectors. As the Multiclass Extreme Learning Machine (ELM) is a fast, simple, and efficient artificial neural network algorithm known for its quick training speed and strong generalization capability, it is employed in this study for classification. This intuitively demonstrates the effectiveness of the proposed method. Finally, the superiority of the proposed method is validated through comparative experiments.

The organization of the remaining sections of this paper is as follows. Section 2 provides a brief overview of the theoretical background of FAWT. In Section 3, the Nonlinear Quantum Permutation Entropy algorithm is elucidated. Building on this, Section 4 proposes a fault feature extraction method based on FAWT and Nonlinear Quantum Permutation Entropy. Section 5 introduces the experimental setup and comparative study. Additionally, discussions follow each application case. Finally, Section 6 summarizes the conclusions.

FAWT achieves signal decomposition by using an Iterated Filter Bank (FB). The filter bank comprises a low-pass filter H(w) and two high-pass channels G(w) and G(-w). Here, G(w) analyzes the “positive frequency,” while G(-w) analyzes the “negative frequency” [29]. Due to this significant positive and negative frequency separation characteristic, FAWT facilitates arbitrary selection of sampling rates in the high-pass channels. Consequently, by employing the Hilbert transform, one can flexibly control redundancy, scaling factors, and Q factors. The frequency responses of the low-pass and high-pass channels in FAWT can be defined by Eq. (1), where H(w) represents the frequency response of the scaling function, and G(w) represents the frequency response of the analytic wavelet function [30,31].

(1)H(w)={pq,|w|<wppqθ(w−wpws−wp),wp≤w≤wspqθ(π−w+wpws−wp),−ws≤w≤−wp0,|w|>wsG(w)={2rsθ(π−w−w0w1−w0),w0≤w≤w12rs,w1<w<w22rsθ(w−w2w3−w2),w2≤w≤w30,w∈[0,w0]∪[w3,2π]

where, p and q regulate the sampling rate of the low-pass channel, while r and s are parameters controlling the sampling rate of the high-pass channel. Additionally, w_s and w_p represent the stopband and passband frequencies of the low-pass filter, respectively. The definitions of other parameters are as follows: (2)wp=1−βpπ+εp,ws=πq,w0=1−βrπ+εr,w1=pqrπ,w2=πr−εr,w3=πr+εr,ε=132(p−q+βqp+q)π,θ(w)=12(1+cos⁡w)2−cos⁡w for w∈[0,π]where, the constants β and ε are non-negative constants satisfying β<1. The transition band θ(w) is constructed using Daubechies wavelets with 2 vanishing moments, forming an orthogonal wavelet filter in FAWT.

The reconstruction of the filter bank in FAWT must adhere to the following two conditions, as expressed in Eqs. (3) and (4): (3)|θ(π−w)|2+|θ(w)|2=1 (4)(1−pq)≤β≤rs

The redundancy and the Q factor is expressed by Eqs. (5) and (6), respectively. (5)R=r/s1−p/q (6)Q=2−ββ

For the analysis of gearbox vibration signals, FAWT provides adjustable parameters to control the quality factor, scaling factor, and redundancy. Based on the advantages of Genetic Algorithm optimization such as strong global search capability, high parallelism, wide applicability, and robustness, we have chosen Genetic Algorithm as the method for optimizing the parameters of FAWT. The Genetic Algorithm effectively explores the parameter space to find the optimal parameter combination, thereby enhancing the performance of FAWT and improving the accuracy and reliability of fault diagnosis. Its strong global search capability and robustness enable Genetic Algorithm to handle complex parameter optimization problems and achieve good results in various application scenarios.

Quantum theory, as an emerging science, has experienced rapid development across various disciplines. Permutation entropy, as an algorithm quantifying the state of time series signals, plays a crucial role. The focal point of this study is how to incorporate quantum theory into permutation entropy, offering broader avenues for development. Quantum theory, as a powerful tool for information processing, continuously propels rapid advancements in related fields. Leveraging the novel concepts and distinctive state representations used to describe the microscopic world, quantum theory has also found effective applications in the processing of vibration signals [32,33].

Quantum bits, or qubits, are the fundamental units describing the quantum world in quantum theory. The states they represent are a form of superposition. The mathematical expression for this is: (7)|φ⟩=a|0⟩+b|1⟩where, |0⟩ and |1⟩ are the quantum basis states for a quantum bit, with coefficients a and b representing the probability amplitudes of the quantum state. These coefficients can be real or complex numbers, and the square of the modulus of the probability amplitude defined as the quantum probability. |a|2 and |b|2 represent the probabilities of the quantum basis states |0⟩ and |1⟩ occurring, respectively. The quantum probability amplitudes adhere to the normalization condition: (8)|a|2+|b|2=1

From Eqs. (7) and (8), it is evident that a quantum bit can describe various states composed of different combinations of two basic states with different probabilities.

Assuming a one-dimensional vibration signal X={x(i),i=1,2,…,N} composed of N sampling points, each element x(i) of the signal X is normalized using the following equation: (9)y(i)=x(i)−min(X)max(X)−min(X)y(i)∈[ 0,1 ]

Here, min(X) and max(X) represent the minimum and maximum values of the signal X, respectively.

The quantum representation of vibration signals is a crucial aspect for further data processing. According to reference [34], a mathematical expression for the nonlinear quantum representation of vibration signals is proposed, mapping the vibration signal from the time domain space to the quantum space for the analysis of its states. After normalizing the vibration signal, its nonlinear quantum expression is given by: (10)|y(i)⟩=cos⁡(y(i)×π/2)|0⟩+sin⁡(y(i)×π/2)|1⟩where, cos⁡(y(i)×π/2) and sin⁡(y(i)×π/2) represent the probability amplitudes of the two basis states, |0⟩ and |1⟩, respectively. Additionally, cos2⁡(y(i)×π/2) and sin2⁡(y(i)×π/2) represent the probabilities of the two basis states, |0⟩ and |1⟩. Due to the normalization condition cos2(y(i)×π/2)+sin2(y(i)×π/2)=1, which satisfies the criteria for quantum bits, this can be applied for the quantization of vibration signals. Since the probabilities cos2⁡(y(i)×π/2) and sin2⁡(y(i)×π/2) undergo nonlinear variations, Eq. (9) is termed the nonlinear quantization of vibration signals.

If each sampling point of the vibration signal undergoes nonlinear quantization using quantum bits, then a vibration signal composed of adjacent k sampling points can be described by k quantum bits. Here, the state of the i-th quantum bit is represented as |ϕi⟩=ai|0⟩+bi|1⟩. In a multi-qubit system, the ground state, composed of multiple symbols, is often referred to as a state vector. Therefore, the state vector of the vibration signal can be expressed as the tensor product of k quantum bits: (11)|Yk⟩=|y(1)⟩⊗|y(2)⟩⊗⋯⊗|y(k)⟩=a1a2⋯ak|00⋯0⟩+a1a2⋯ak−1bk|00⋯01⟩+⋯+b1b2⋯bk|11⋯1⟩=∑i=12kwi|ib⟩where, for the quantum system (vibration signal) |Y_k⟩, the i-th state vector is denoted as |i_b⟩. The state vector is expressed in binary form, where w_i is the probability amplitude of the state vector |i_b⟩ and |w_i|² is the probability of the state vector |i_b⟩. According to the normalization condition in quantum theory, the probabilities should satisfy: (12)∑i=02i−1|wi|2=1

Permutation entropy, as a nonlinear parameter for quantifying the randomness and dynamic variations of time series signals, has found widespread applications in various fields. Inspired by the resemblance between permutation entropy and the representation of information states in quantum theory, which involves various quantum basis states or state vectors and their associated probabilities, the integration of quantum theory with permutation entropy has been proposed to describe the operational state of vibration signals. This combination is utilized to extract and characterize feature parameters related to the system’s operational state [26].

During the operation of a gearbox, its vibration signals are associated with its operational state. Different vibration time series correspond to different probabilities or probability amplitudes of various basis states or state vectors in their quantum representation. The entropy values of these basis states or state vectors are calculated using the permutation entropy method, and these are utilized as features to characterize the gearbox’s operation.

Combining the concepts of quantum theory and permutation entropy, the basic principles of nonlinear quantum permutation entropy are outlined as follows:

(1) Normalization of Vibration Signals. Let the time series of the vibration signal be denoted as X = {x(i), i = 1, 2, …, N}. Applying Eq. (9) to normalize the time series of the vibration signal, resulting in the normalized time series Y = {y(i), i = 1, 2, …, N}.

(2) Phase space reconstruction. Conducting phase space reconstruction on the time series Y, a matrix Y₀ is obtained. (13)Y0=[Y0(1)Y0(2)⋯Y0(j)⋯Y0(K)]=[y(1)y(1+τ)⋯y(1+(m−1)τ)y(2)y(2+τ)⋯y(2+(m−1)τ)⋯⋯⋯⋯y(j)y(j+τ)⋯y(j+(m−1)τ)⋯⋯⋯⋯y(K)y(K+τ)⋯y(K+(m−1)τ)]where, j = 1, 2, 3, …, K. m is the embedding dimension, τ is the time delay, and K = N − (m − 1) τ. Each row Y₀ (j) in the reconstruction matrix represents a reconstruction component, and there are total of K reconstruction components in the reconstruction matrix.

(3) Nonlinear quantization of reconstruction components. Each reconstruction component in the reconstruction matrix undergoes nonlinear quantization. A multi-qubit system is employed to form quantum bit state vectors composed of m qubits. The total number of state vectors is n = 2^m, namely: (14)|Y0(j)⟩=wj,1|00⋯0⟩+wj,2|00⋯01⟩+⋯+wk,n|11⋯1⟩=∑k=1nwj,k|ib⟩where, |ib⟩ represents the i-th state vector of the quantum system, w_j,k is the probability amplitude of the state vector |i_b⟩, and |w_j,k|² is the probability of the state vector |ib⟩. These probabilities satisfy the normalization condition: (15)∑k=1n|wj,k|2=1

(4) Each row in the matrix is regarded as a component. By sorting the reconstruction components |Y0(j)⟩ in ascending order based on the probability amplitudes of each state vector, we obtain: |Y0[j+(j1−1)τ]⟩≤|Y0[j+(j2−1)τ]⟩≤…≤|Y0[j+(jm−1)τ]⟩, where, j1,j2,…,jm represent the index numbers of the columns in which each element is located. Therefore, for each row in the matrix |Y0⟩, a set of symbol sequences S(g)=(j1,j2,…,jm), namely ordinal pattern, can be obtained., where g=1,2,…,l. Then, calculate the probability distribution pk of the ordinal patterns, where k = 1, 2, 3, …, n.

(5) Calculate nonlinear quantum permutation entropy by estimating the Shannon entropy of the ordinal probability distribution of each state vector. (16)Hq∗(X)=−∑i=1npkln⁡pk

(6) Normalization of nonlinear quantum permutation entropy. The maximum value ln(m!) of nonlinear quantum permutation entropy H_q(X) is obtained when P_k = 1/m!. For convenience, H_q(m) is typically normalized, namely: (17)Hq(X)=Hq∗(X)ln⁡(m!)

The nonlinear quantum permutation entropy H_q(X) represents the probability distribution of the state vectors in the ascending order after quantizing the time series X. A larger H_q(X) indicates a more regular probability distribution of the state vectors’ arrangement, while a smaller H_q(X) suggests greater differences in the probability of state vectors’ arrangement. For a gear transmission system, a larger H_q(X) implies a more stable operational state, closer to the normal condition, while a smaller H_q(X) indicates an unstable gearbox operational state, deviating from the normal condition and exhibiting anomalies. The variation of H_q(X) can reflect and amplify subtle changes in the time series. The workflow of the nonlinear quantum permutation entropy algorithm is illustrated in Fig. 1.

In a gearbox, the occurrence of faults in gears can lead to changes in the frequency components and amplitudes of the corresponding vibration signals. Moreover, the presence of noise interference in the vibration signals complicates the analysis. Taking a planetary gearbox as an example, the nonlinear quantum permutation entropy is analyzed for three types of simulated vibration signals: Normal operation, local fault in the sun gear, and local fault in the planetary gear. This analysis aims to demonstrate the feasibility of using nonlinear quantum permutation entropy as a characteristic feature for gearbox condition monitoring.

According to the reference [35], the vibration simulation signal for a single-stage planetary gearbox with normal gear conditions is given by: (18)x(t)=[1−cos⁡(2πfct)]⋅cos⁡(2πfmt+θ)where, f_c represents the rotation frequency of the planetary carrier, f_m is the meshing frequency of the planetary gearbox, and θ is the initial phase of the meshing vibration.

The vibration simulation signal for local fault in the sun gear is: (19)x(t)=[1−cos⁡(2πfs(r)t)][1+0.5cos⁡(2πfst)]⋅cos⁡[2πfmt+0.5sin⁡(2πfst)+θ]where, fs(r) is the absolute rotation frequency of the sun gear, and f_s is the characteristic frequency of the local fault in the sun gear.

The vibration simulation signal for local fault in the planetary gear is: (20)x(t)=[1−cos⁡(2πfct)][1+0.5cos⁡(2πfpt)]⋅cos⁡[2πfmt+0.5sin⁡(2πfpt)+θ]where, f_p is the characteristic frequency of the local fault in the planetary gear.

According to the reference [35], the parameters for the simulation signals are set as shown in Table 1.

Setting θ = 0 and all initial phases to zero, Gaussian white noise with a signal-to-noise ratio of 10 dB is added to the time-domain signals for the three states. The sampling frequency is set to 5120 Hz, and the simulation time is 10 s. The vibration time-domain simulation signals for the three states are shown in Fig. 2.

During the phase space reconstruction, for the vibration simulation signals corresponding to the three states, the delay time is determined to be τ = 2 using the mutual information method, and the embedding dimension is determined to be m = 6 using the false nearest neighbor method. Nonlinear quantum permutation entropy is calculated every 1 s, and a total of 20 simulation signal samples are computed. The results are shown in Fig. 3. From Fig. 3, it can be observed that the QPE values of the normal gears are higher than those of the faulty gears. Additionally, the vibration signals for the three states in the planetary gearbox exhibit significant differences in nonlinear quantum permutation entropy, providing preliminary evidence for the effectiveness of nonlinear quantum permutation entropy as a feature for monitoring the operational state of a single-stage planetary gearbox. Moreover, the relatively small fluctuations in the nonlinear quantum permutation entropy values for the vibration signals in each state indicate a certain level of noise resistance.

Previous studies have demonstrated the superiority of nonlinear quantum entropy as a feature parameter for faults. However, the time-domain representation of vibration signals in actual rotating machinery often conceals a significant amount of useful information, making fault feature extraction challenging. In order to effectively reveal potential fault information, it is necessary to decompose the vibration signals first and expose the fault information sufficiently in the decomposed sub-signals. Therefore, this study proposes a novel method for extracting weak fault features based on Flexible Analytical Wavelet Transform (FAWT) and Nonlinear Quantum Permutation Entropy. FAWT overcomes many inherent limitations of Wavelet Transform (WT) by adaptively selecting suitable wavelet bases and sensitive sub-bands using arbitrary scaling and translation factors, thereby maximizing the revelation of hidden fault features. Then, by extracting the nonlinear quantum permutation entropy of each sub-signal, a feature matrix is obtained. Finally, the feature matrix is divided into training and testing sets, which are input into ELM for classification and identification.

However, due to the extreme sensitivity of FAWT results to parameter settings, this study employs a Genetic Algorithm (GAs) to select the optimal control parameters, using the maximization of the feature kurtosis spectrum entropy as its fitness function. To obtain appropriate FAWT parameters, a Genetic Algorithm (GAs) is employed to optimize the wavelet basis function factors p, q, r, s, and β for FAWT. The feature kurtosis spectrum entropy indicator is used to evaluate the pulse and periodic behavior of each sub-band signal. By optimizing the FAWT parameters to maximize kurtosis spectrum entropy, the optimal FAWT basis functions can be obtained. The determined basis function factors can better decompose vibration signals, thereby having a greater potential to reveal weak fault features.

The kurtosis indicator is one of the commonly used features in the field of fault diagnosis, particularly sensitive to transient impulses. For an input signal x(n), its kurtosis is defined as follows: (21)Kurt[x]=E[(x(n)−u)4]σ4where, μ and σ are the mean and standard deviation of x(n), respectively.

Given that vibration signals in gearboxes often exhibit characteristics of periodic fault impulses, selecting the kurtosis spectrum entropy as the adaptability function for FAWT parameter selection is considered. Let S (ω) be the envelope spectrum of the input signal x_n(t), and E_s be the entropy of the envelope spectrum. Assume J represents the number of layers in the FAWT decomposition. Then, the envelope spectrum is segmented into J frequency intervals across the frequency axis. The portion of the spectrum sampled within the i-th interval is referred to as the probability distribution P_i. The kurtosis spectrum entropy CKSE defined using this probability distribution satisfies the following conditions: (22)CKSE=Kurt[x]/(−∑i=1JPilog2⁡(Pi)),∑i=1JPi=1

Using the maximum value of the kurtosis spectrum entropy as the fitness function in the genetic algorithm to optimize the parameters of FAWT aims to maximize the optimization of FAWT decomposition sub-bands. This ensures that the sub-band signals contain the most fault feature information.

Hence, this study employs the constructed optimal FAWT basis to decompose the input signal into various scales. Subsequently, it captures the nonlinear quantum permutation entropy of wavelet sub-bands to unveil potential fault characteristics. Based on the advantages of faster training speed and simpler parameter adjustment, Extreme Learning Machine (ELM) is selected for the final classification and recognition. In summary, the process of extracting weak fault features in rotating machinery can be summarized as follows: (1) Collect vibration signals from gears or bearings in rotating machinery. (2) Use the FAWT method to decompose the vibration signals, maximizing the kurtosis spectrum entropy as the fitness function in the decomposition. Utilize a Genetic Algorithm to select suitable parameters for the FAWT basis. (3) Calculate the nonlinear quantum permutation entropy of each decomposed sub-band signal, forming a multidimensional feature matrix. (4) Input the feature matrix into ELM, using part of it for model training and the other part for classification and recognition. (5) Output the final fault type and severity. The overall flowchart is shown in Fig. 4.

This study presents an innovative approach for extracting fault features by integrating nonlinear quantum permutation entropy with Flexible Analytical Wavelet Transform (FAWT). The developed method is utilized to capture subtle characteristics in rolling bearings and gearboxes, effectively recognizing fault signatures in rotating machinery across diverse operational scenarios. The key contributions of this investigation are outlined as follows:

(1) The application of FAWT offers a more adaptable time-frequency coverage. By employing arbitrary scaling and translation factors, FAWT’s oscillatory properties become adjustable, providing additional time-frequency information for detecting fault components. Utilizing a Genetic Algorithm and adhering to the maximization principle of the kurtosis spectrum entropy, an adaptive and dynamically designed optimal basis for the input signal is achieved. This facilitates the revelation of subtle fault features in the decomposed wavelet sub-bands.

(2) Quantum theory, as a profoundly transformative framework, holds enormous potential when introduced into the analysis of vibration signals. Its integration into vibration signal analysis is poised to offer novel perspectives to the field. The utilization of nonlinear quantum permutation entropy in the process of nonlinear quantization takes into account the actual values of the time series and ensures highly accurate calculations of the probability distribution of the arrangement of various states. Therefore, leveraging nonlinear quantum permutation entropy allows for a more precise reflection of the system’s operational state. Comparisons with conventional entropy values such as sample entropy indicate that nonlinear quantum permutation entropy proves to be more effective for feature extraction from vibration signals in complex structured rotating machinery.

(3) The feature extraction algorithm proposed, based on FAWT and nonlinear quantum permutation entropy, is employed on both rolling bearing and experimental gear data. Experimental findings indicate that the proposed method effectively discerns various fault types and severity levels with accuracy. Comparative analysis with existing techniques highlights the effectiveness and practicality of the proposed model.