Sensors play a crucial role in the design of exoskeletons and wearable robots as they greatly influence reliability and cost. Wearable robots most commonly use IMU sensors which record 3-axis acceleration, angular velocity, and gyroscope signals [4,15]. IMUs are preferred as they provide reliable data and are cost-effective. EMG sensors are employed for human activity recognition (HAR) as they can measure electrical signals generated from muscle contractions during physical activities [3]. While EMG data can significantly enhance their performance wearable robots are relatively expensive and inconvenient to wear. Other types of sensors, such as load cells, pressure sensors, (GONIO), are also used. Furthermore, the fusion of signals from multiple sources has also been widely applied to leverage their respective advantages [8]. The selection of sensor types and data processing methods can greatly impact the locomotor intent classifier performance, and finding ways to achieve better results with fewer sensors has been the primary research focal point.

Motivated by their successful application in various fields, DL methods have gained significant popularity in locomotor intent classification. CNNs, known for their effective feature learning in CV, have been applied to locomotor intent classification by similarly processing sensor data [27]. While CNNs offer advantages such as parameter sharing and sparse connections, they require higher computational resources. LSTM networks, known for their ability to process time series data, haves shown promising results in classifying human locomotion [21]. To enhance efficiency, studies have explored CNN-RNN hybrid networks. Wang et al. [28] proposed a CNN-LSTM hybrid approach, achieving a validation accuracy of 95.90% in recognizing six types of activities (walking, lying, sitting, standing, and stair ascent, and stair descent) using a smartphone IMU. Zhu et al. [23] proposed an attention-based CNN-LSTM model for knee/ankle joint angle movement prediction, using AM for feature extraction. In contrast to [23], the AM in LREAL was introduced after the LSTM layer to dynamically redistribute attention across different features at different time steps, aiming for complete spatiotemporal feature selection.

The interpretability of DNNs is often criticized due to the difficulty in explaining inferences as concise interactions among parameters and the network [29]. Unlike human inferences based on experiences and knowledge, most DNNs are purely data-driven. This lack of prior knowledge can result in low interpretability and potential errors in the models [30]. To address these issues, knowledge representation has been utilized to describe the richness of the world in computer systems, allowing artificial intelligence to understand and utilize it for reasoning and inference. Rule-based representation, a common formalism of knowledge representation, has been widely employed in expert systems and achieved excellent results [31]. Considering the aforementioned challenges and inspired by the concept of rules embedded in neural networks [25], we propose a rule-modulating block that incorporates locomotion rules into a knowledge representation and combines it with DNNs. This approach aims to enhance interpretability and improve the model performance by leveraging both data-driven and rule-based learning.

We used a the ENABL3S public benchmark dataset to validate the proposed method. The dataset consists of data from IMU sensors (placed on the thigh and shank of the subjects), GONIO sensors (placed on knee and ankle of the subjects) and EMG sensors, collected from 10 able-bodied subjects (seven males and three females) with an average age of (25.5 ± 2) y, height of (174 ± 12) cm, and weight of (70 ± 14) kg. The subjects move according to the required mode [sit (S), stand (ST), ground-level walking (LW), ramp ascending/descending (RA/RD), and stair ascent/descent (SA/SD)]. The ramps have a 10° slope slopes of 10° and the stairs consist of four steps. The odd-numbered moving sequence is: S → ST → LW → SA → LW → RD → LW → ST → S; and the even-numbered moving sequence is: S → ST → LW → RA → LW → SD → LW → ST → S. There are seven kinds of steady locomotion modes in ENABL3S with recorded true labels. Each data sample contains the signals from five IMU sensors recording six data channels (3 axis acceleration and angular velocity) and four GONIO sensors. By using a key fob, the start and end of the steady locomotion modes are recorded and the data is labeled according to the true moving mode. Although only labels of steady locomotion modes are available in ENABL3S, the information on the transition modes can be obtained by extracting the data between the steady modes as ENABL3S marks the time of the beginning and ends of the steady states.

The IMU data underwent low-pass filtering at 20 Hz to eliminate high-frequency noise. In the ENABL3S dataset, the IMU and GONIO data are sampled at 500 Hz (i.e., sampling once every 2 ms). Every 250 samples (2 ms per sample) were segmented into a 500 ms analysis window with a sliding window of 50 ms. We used the locomotion mode at the end of each gait event to obtain the label of each analysis window. If the beginning and end of the gait event maintained the same locomotion mode, analysis windows for the gait event were directly labeled as “steady locomotion intent” (Fig. 1). Gait events with different modes can experience two different situations. If the analysis window was located (i) fully within a transition gait event, it was labeled according to the starting and ending mode and (ii) partially within a transition gait event of transitions, it was labeled based on the majority of its location (>50%; Table 1). We utilized the acceleration and angular velocity data of only two IMU sensors on the upper and lower right leg in 3 axes (X, Y and Z axes) and GONIO data, so the initial input of the network can be denoted as x∈R500∗12. The above data was divided as follow: 90% for training and 10% for testing. Additionally, 20% of the training set was also used for validation to prevent the network from over-fitting.

The proposed LREAL network is composed of two levels (Fig. 2). The firstlevel, i.e., the steady/transition classifier, is responsible for initially classifying the input sequence and dividing it into the two main states. This classifier treats both steady and transition states equally. Its purpose is to serve as a pre-classification step for the subsequent level.

The second level consists of two distinct classifiers and a rule-modulating block. The input sequence divided by the first-level classifier is directed to the appropriate classifier based on the assigned state label. The steady-state classifier receives the fragments from the steady-state portion of the sequence as its input. The output of this classifier is stored and passed onto the rule-modulating block as auxiliary information, which helps in preparing for the recognition of transitions.

As mentioned previously, transitions occur between steady states, i.e., there always exists a preceding steady state that significantly influences the range of post-transition behavior. There also exist specific rules governing the motion transformations, such as considering the previous locomotion mode when encountering a transition [32]. Incorporating these rules and prior information from the classifier for steady locomotion into the transition-state classifier does not affect practical applications or add computational cost. Therefore, the rule-modulating block is designed to introduce motion transformation rules to the network. It receives transmission information from the steady-state classifier to obtain the preceding condition for each window for transitions. This information is combined with the motion transformation rules to assist the transition-state classifier in recognition. This process embeds the provided motion conversion rules with auxiliary information from the preceding steady state, resulting in an output vector that represents refined information.

Finally, the transition-state classifier uses the sequence of recognized transitions from the first-level classifier and refined information from the rule-modulating block to generate the final output. Note that the classification of steady states solely relies on the data, while the transition recognition combines the data with rule-embedded information. This integration of data and rule-based information enhances the of transition recognition accuracy.

As different samples within a specific analysis window may contribute differently to the recognition of various activities, the proposed AT-LSTM model employs an AM to capture the relationship between the samples and locomotion classes (Fig. 3). The attention layer first derives a vector representation for the features of each sample and maps to a probability value indicating the likelihood of the analysis window belonging to a certain locomotion class. Formally, we constructed the attention layer as follows.

We first formulated the attention importance score, aik, for each feature in the kth sample by multiplying the input vector, hik, with a weight matrix, Ws, and normalizing the result to a probability which represents the degree of relevance between the ith feature and the locomotion mode of the kth sample. It can be expressed as (1)aik=exp⁡(sik)∑j=1nexp(sjk)where sik=hikWs.

To concentrate on the feature instead of focusing on a specific sample, each feature of a sample is assigned a separate importance score instead of a fixed score. Then we can denote the newly-constructed feature vectors of ith feature, eik, as (2)eik=hikaikwhich indicates the degree of relevance between the ith feature of kth sample and the locomotion class. By flattening eik ,we then combine the different features of the sample within the feature vector, vki, as (3)vki=eik

In conclusion, the attention layer helps the network to better predict the locomotion classes by focusing on the key sample of the analysis window.

After obtaining vki, the probability vector, p=[p1,p2,…,pc,], for analysis window classification can be formulated as (4)p=softmax(wpvki+bp)where w_p and b_p are the weight matrix and bias vector, respectively.

The input of the attention-based LSTM network (Fig. 4) is an analysis window of dimension 250 × 12. The three LSTM layers have 128, 64, and 32 units, respectively, and the LSTM layers will keep the structure of N-to-N, which means the output of the final LSTM layer remains a sequence. The attention layer takes the sequence as the input (Fig. 3). Finally, we add a dense layer with 100 units and a softmax layer that obtains the final classification result. The size of the softmax layer is designed based on tasks. Specifically, the layer size is set to (i) 2 when the classifier is designed to distinguish the steady and transition states, and (ii) 7/12 when recognizing the steady and transition states in detail.

The rule-modulating block was designed based on the locomotion transformation information from the ENABL3S dataset. It combines this information with that from the steady-state classifier and passes it to the transition-state classifier for recognition. The rule was designed based on the previous locomotion mode before the transition, as we know that each steady state can only have limited options for subsequent transition states (e.g., for the S steady state, only the S-ST transition is possible; and for the ST steady state, only the ST-S and ST-LW transitions are possible). All rules were obtained based on the actual locomotion mode in ENABL3S and encoded as within the M7×12 matrix in the network (Table 2). The rows of the matrix represent the previous steady locomotion state and the columns represent the subsequent transition. If the transformation from the previous locomotion to the subsequent transition is valid(invalid), its value in the matrix is set to 1(0).

After obtaining the conversion rules, the rule-modulating block combines it with the probability vector, p=[p1,p2,…,p7,], of the previous analysis window from the steady-state classifier as (5)xj=∑i=17pi×Mi,jj∈[1,12]

The obtained encoded information vector X1∗12, which represents the probability of the next transition, is passed on to the transition-state classifier as input. To prevent the encoded information from being diluted, it is directly input into the attention layer along with the hidden layer output vector from the previous LSTM layer to construct the final transition-state classifier (Fig. 5).

We utilized an initial set of hyperparameters and investigated their impact on the model to narrow down the range for the subsequent grid search (Table 3). A grid search with a 5-fold cross-validation was then performed on the hyperparameters to maximize validation accuracy (Table 4; a detailed discussion on the influence of the hyperparameters can be found in Section 5.3.6). The configuration shown in Table 4 was selected as the optimal choice for subsequent experiments involving the classifier. To deal with the imbalanced distribution of the different classes in the ENABL3S dataset (Table 1), we adjusted the weight of each class for training inversely proportional to class frequency in the dataset as

(6)Weighting=nsamples_with_classnclasses∗nsamples_with_class

To validate the effectiveness of the proposed method, we devised two distinct experimental scenarios. The first scenario aimed to evaluate the classification performance of each LREAL component individually. Thus, the performance of the three classifiers was tested separately in classifying pure data for steady/transitions states. In the second scenario, we introduced mixed data and examined the performance of LREAL when considering the mutual influence of various components. Furthermore, in both scenarios, the networks were trained and tested on user-dependent and user-independent base.

We used the accuracy (ACC) score to measure the overall performance of our proposed LREAL network; it can be expressed as

(7)ACC=NcorrectNtotal×100%where N_correct is the number of correct classifications and N_total is the total number of data points. We also evaluated the network performance in two bases: (1) Data from one out of ten subjects was used to train and test a user-dependent network; (2) data from nine out of ten subjects was used as the training set, while the remaining subject was used as test set to construct a user-independent network. The processes were repeated 10 times until all subjects were used.

Furthermore, when evaluating the overall performance, the number of sensors utilized serves as a crucial metric. In terms of practical applications, achieving comparable results with fewer sensors signifies superior model performance.