As fifth generation technology standard (5G) technology develops, the possibility of being exposed to the risk of cyber-attacks that exploits vulnerabilities in the 5G environment is increasing. The existing personal recognition method used for granting permission is a password-based method, which causes security problems. Therefore, personal recognition studies using bio-signals are being conducted as a method to access control to devices. Among bio-signal, surface electromyogram (sEMG) can solve the existing personal recognition problem that was unable to the modification of registered information owing to the characteristic changes in its signal according to the performed operation. Furthermore, as an advantage, sEMG can be conveniently measured from arms and legs. This paper proposes a personal recognition method using sEMG, based on a multi-stream convolutional neural network (CNN). The proposed method decomposes sEMG signals into intrinsic mode functions (IMF) using empirical mode decomposition (EMD) and transforms each IMF into a spectrogram. Personal recognition is performed by analyzing time–frequency features from the spectrogram transformed into multi-stream CNN. The database (DB) adopted in this paper is the Ninapro DB, which is a benchmark EMG DB. The experimental results indicate that the personal recognition performance of the multi-stream CNN using the IMF spectrogram improved by 1.91%, compared with the single-stream CNN using the spectrogram of raw sEMG.
Personal recognitionelectromyogram signalmulti-stream networkempirical mode decompositionIntroduction
5G technology has been studied to realize a society of enhanced mobile broadband and low latency communication interconnected with networks of other industries as well as mobile networks [1]. As 5G technology develops, internet of things (IoT) devices incorporating information and communication technology (ICT) technology can be used in fields such as self-driving cars and healthcare, as shown in Fig. 1 [2]. As such, 5G technology connects all aspects of life to communication networks, so information security is important. The three major elements of information security are confidentiality, integrity, and availability [3]. Among them, confidentiality means disclosing information to an authorized user, and integrity means enabling information modification to an authorized user. In order to proceed with confidentiality and integrity, a personal recognition process that distinguishes authorized users is required. The existing personal recognition method used for granting permission is a password-based method. However, it causes security problems such as malicious manipulation and leakage of personal information [4]. To complement for this, personal recognition study using bio-signals is being conducted as a method to access control to devices incorporating IoT technology [5,6].
5G technology application field
The bio-signals are generated while biological activity, and examples of bio-signals include EMG, electrocardiogram (ECG), and electroencephalogram (EEG) signals. Studies on personal recognition based on bio-signals are mainly conducted using ECG and EEG. The registered ECG information, which is generated by heart rates, cannot be modified just as fingerprint, iris, and face data; once the registered personal information is leaked, this can lead to financial losses. The registered EEG information, which varies by the activity of the cerebrum, can be modified; however, its signals are easily distorted by hair and scalp, and people feel discomfort using the measuring instrument. EMG signals can be modified by the performed action; hence, they can address the challenges faced by existing personal recognition methods. Furthermore, EMG signals can be measured more conveniently than EEG because the sensors for data acquisition can be attached to arms and legs.
Among bio-signals, EMG signals are generated while skeletal muscles contract and can be measured for various muscles depending on the measurement position [7]. Furthermore, EMG signals indicate complex information reflecting neuromuscular control, as well as the physiology of muscle tissues [8]. There are two methods for measuring EMG signals: invasive and non-invasive methods. The invasive method has the advantage of low noise because the needle electrode is inserted into the muscle. However, it is difficult to apply this method to personal recognition owing to the pain felt when the needle electrode is inserted into the muscle. By contrast, the non-invasive method attaches the electrode to the skin and can measure the EMG signals more conveniently than the invasive method [9]. The EMG measured via the non-invasive method is called sEMG. Fig. 2 presents the measurement procedure for sEMG signals.
sEMG signal measurement using the surface electrode method
Existing studies on personal recognition using sEMG had extracted features from sEMG [10–15] or increased feature data by overlapping windows [16–18]. However, overlapping windows within the same data can trigger overfitting, which degrades the recognition performance owing to data generalization [19]. Empirical mode decomposition can increase feature data without data overlapping by decomposing data into physically meaningful components. Existing studies adopted sEMG-based EMD to remove noise from the sEMG, rather than increasing feature data [9,15,20,21].
This paper proposes a personal recognition method using multi-stream CNN-based sEMG. The proposed method removes noises from the sEMG using notch filter (NF) and band-pass filter (BPF), and then decomposes the data to intrinsic mode functions 1–4 using EMD. Each decomposed IMF is transformed into two-dimensional spectrograms, input to the CNN designed with four streams, and employed for personal recognition. The obtained experimental results indicate that the personal recognition method using the spectrograms of multi-stream CNN-based IMF improved the performance by 1.91% more than the method using raw sEMG, and by 1.13% more than the existing personal recognition study using CNN. This verified that the personal recognition performance can be improved by adopting the information provided by IMF after decomposing sEMG to EMD. The remainder of this paper is organized as follows. Section 2 describes the sEMG personal recognition method using the multi-stream CNN-based EMD proposed in this paper. Section 3 explains the results of the experiment using the proposed method. Finally, Section 4 concludes the paper.
Proposed Personal Recognition Method Using Multi-Stream CNN and sEMG
The proposed personal recognition method using the proposed multi-stream CNN-based sEMG is illustrated in Fig. 3. The noises in the signals are removed via sEMG preprocessing using a digital filter. The preprocessed sEMG is decomposed into IMFs 1–4 using EMD, and two-dimensional spectrograms are generated using IMFs 1–4 decomposed from sEMG. The proposed multi-stream CNN used generated spectrograms for learning, and personal recognition is performed with this learned information.
Flowchart of proposed personal recognition using sEMG based on multi-stream CNN
sEMG Signal Preprocessing
sEMG has the advantage of being substantially convenient as a non-invasive method; however, skin impedance emerges owing to attachment of electrodes to the muscles for data measurement. Furthermore, sEMG signals are corrupted by various factors, such as power line interference, white Gaussian noise, and baseline wandering [8,9]. To eliminate power line interferences, this paper employs NF in the 60 Hz band, and sEMG is preprocessed using BPF in the 5–500 Hz band, which contains several gesture information to reduce the effects of other factors. Fig. 4 presents signal changes before and after sEMG preprocessing. The blue-dotted and red-solid lines represent raw and denoised sEMG signals, respectively. The denoised sEMG signals are decomposed to IMF using EMD.
Comparison before and after sEMG signal preprocessing
sEMG Signal Decomposition Using EMD
Similar to other bio-signals, sEMG exhibits nonlinear characteristics. Hence, it is inappropriate to use an algorithm based on linearity [22]. EMD, which can be used to decompose sEMG, is a suitable method for efficiently processing nonlinear signals, such as sEMG [23]. EMD is mathematically expressed as a sum of IMF and residual for given signals, as expressed in Eq. (1). A low-order IMF represents a high-frequency component, while a high-order IMF represents a low-frequency component [9].
x(t)=∑i=1nIMFi+residual
EMD is a data adaptive technique that does not require prior parameter setting and repeatedly performs the sifting process. Through the sifting process as described below, decomposes sEMG into IMF and residuals [24].
Calculate the local minimum and maximum from signals ri(t) (if i=1, ri(t) =x(t)).
Calculate the bottom and top envelopes using the local extreme values.
Calculate the mean m(t) of the bottom and top envelopes.
Calculate the difference c(t) between ri(t) and m(t).
Check if the calculated c(t) satisfies the IMF condition.
If the IMF condition is not satisfied, repeat from Step 1, using ri(t) =c(t).
If the IMF condition is satisfied, then IMFi = c(t),ri+1ri−c(t).
The decomposed IMF is defined as a function that satisfies the following two conditions, and the residual expresses a monotonic function. Fig. 5 presents the IMF signals decomposed using EMD. In Fig. 5, a higher order of the IMF indicates a lower frequency component, and it can be observed that IMFs 1–4 contain important sEMG information [12,25].
Raw sEMG decomposition results using EMD
The number of extreme values and number of zero crossings are exhibit a difference of one or less.
The mean of the top and bottom envelopes must be equal to zero.
Multi-Stream-Based Personal Recognition Using sEMG Signals
To harness the combined advantages of the time and frequency domains of sEMG, this paper performs multi-stream CNN-based personal recognition by converting sEMG to two-dimensional spectrograms using the formula expressed in Eq. (2), where r(t), wf(t), R, and w denote the IMF signal, window function, window length, and angular frequency, respectively. The results obtained from converting IMF signals into spectrograms are presented in Fig. 6, and the time–frequency features can be analyzed simultaneously.
S=∫1Nr(t)wf(t−R)e−iwtdt
Spectrogram generation using IMF signals
Existing studies on personal recognition with bio-signals primarily adopted handcraft features. However, the handcraft feature extraction method may not extract optimal features because it employs a predefined function, and its performance may appear high solely under a specific condition. To address these problems, deep learning, which performs training via forward and back propagations, is being considered. Deep learning can extract the optimal features of data because it extracts features adaptively to data without predefining the function [26].
The multi-stream-based CNN structure defined in this paper is illustrated in Fig. 7. Each stream consists of eight convolution layers and three pooling layers. Convolution layers calculate features using feature maps, which comprise 8, 16, 32, and 64 size. Pooling layers adopt maxpooling to prevent down-scale weighting by which the features decrease and set with a 2 × 2 filter and stride 2 [19]. The multi-stream CNN designed with four streams extracts features using the spectrograms of IMFs 1–4 and merges the output of the eighth convolution layer in the fully connected layer. The training of the multi-stream CNN is performed with a learning rate of 0.001, an Adam optimizer, a batch size of 128, and 100 epochs.
Multi-stream CNN structure for personal recognition using spectrograms
Experimental Results and Discussion
The personal recognition experiment with the proposed multi-stream CNN-based sEMG used Ninapro DB2, a benchmarking EMG DB. Ninapro DB2 represents the sEMG measured when subjects with intact muscles make hand and wrist gestures. Each gesture was maintained for five second and repeated six times, with a three second rest between gestures. sEMG was measured in twelve channels at the 2,000 Hz sampling rate. The measured muscle was a right forearm, on which eight channels of electrodes were placed in equal intervals, and a channel was placed each on the flexor digitorum, extensor digitorum, biceps, and triceps [27].
The personal recognition experiment with the multi-stream CNN-based sEMG adopted the sEMG of seventeen gestures performed by twenty subjects. In this experiment, subjects were recognized via 1:n comparison, and the data comprised four training and two test data. The experiment was conducted using seventeen gestures simultaneously as input data, rather than performing the personal recognition experiment seventeen times for one gesture. For comparison, the single-stream CNN was experimented using raw sEMG, IMF1, IMF2, IMF3, and IMF4 separately, whereas the multi-stream CNN used IMFs 1–4. The structure of the single-stream CNN is presented in Fig. 8. The preprocessing method, learning rate, and epoch were set as the same as those of the multi-stream CNN experiment.
Single-stream CNN structure for personal recognition
Tab. 1 presents the results of the personal recognition experiment using the proposed multi-stream-based sEMG. As presented in this Tab. 1, the single-stream CNN using raw sEMG exhibited higher performance than the CNN using IMF1, IMF2, IMF3, and IMF4 separately. However, the personal recognition accuracy of the multi-stream CNN using IMFs 1–4 was 98.48%, which was 1.91% higher than that of the raw sEMG personal recognition method. This verifies that the personal recognition method using the proposed multi-stream-based sEMG improved performance better than the existing method using raw sEMG.
Results of personal recognition experiments based on multi-stream CNN using spectrograms
Model
Signal
Accuracy (%)
Single-stream
Raw sEMG
96.57
IMF1
96.37
IMF2
95.59
IMF3
94.17
IMF4
94.56
Multi-stream (proposed)
IMFs 1–4
98.48
Fig. 9 presents the results of the confusion matrix of personal recognition obtained with the multi-stream CNN-based sEMG. As illustrated in this figure, the subject with the lowest personal recognition performance was subject no. 9, followed by subject no. 6. Subject no. 9 was misrecognized as no. 14 six times. In the experiment with single-stream CNN using raw sEMG, subject no. 9 was misrecognized nine times to no. 14. Fig. 10 presents the sEMG signals for the Ninapro DB2 no. 9 gestures of subjects no. 9 and 14. As illustrated in this figure, the sEMG waveforms of subjects no. 9 and 14 were significantly similar. Consequently, subject no. 9 was most frequently misrecognized as no. 14, both in the multi- and single-stream CNN experiments.
Personal recognition confusion matrix based on multi-stream CNN using spectrograms
Ninth gesture and sEMG signals (subjects no. 9 and 14) of Ninapro DB
Tab. 2 presents the results obtained from the comparative experiment between the proposed personal recognition method using the multi-stream-based sEMG and existing studies. The existing personal recognition method using sEMG employed the directly acquired sEMG without benchmarking data and did not describe the data numbers used in the training and test. Therefore, the comparative experiment was conducted after setting the same data composition using Ninapro DB2, which was employed in this paper. The performance comparison with existing personal recognition studies was conducted using the accuracy and precision metrics. According to the comparison result, the approach by Shin et al. [10] exhibited the lowest performance because they only adopted time-domain features such as zero-crossing (ZC) and variance (VAR). The approach proposed by Kim et al. [11] also exhibited low performance because they adopted a relatively shallow neural network. Lu et al. [14] performed personal recognition using continuous wavelet transform (CWT) and CNN; however, it was difficult to determine meaningful features via wavelet analysis because nonlinear time series, such as sEMG, contains several periodic components [28]. Consequently, they exhibited a performance lower by 1.13% than that of the proposed multi-stream-based personal recognition method.
Comparison with previous works using Ninapro DB2
Database
Author
Accuracy (%)
Precision (%)
Ninapro DB2
Shin et al. [10]
89.26
89.43
Kim et al. [11]
96.52
96.63
Lu et al. [14]
97.35
97.47
Proposed
98.48
98.51
Conclusion
This paper proposed a personal recognition method using sEMG based on multi-stream to access control to devices of 5G. The proposed method decomposed sEMG to IMFs using EMD after preprocessing the sEMG data by NF and BPF. Each IMF converted time–frequency features into spectrograms that can be analyzed simultaneously. In addition, each spectrogram was adopted as the input to the multi-stream CNN for personal recognition. The performance of the proposed method was compared with the single-stream CNN using Ninapro DB2. The results obtained from the experiments indicate that the proposed method using the spectrograms of IMFs 1–4 and multi-stream CNN improved the performance by 1.91% more than the personal recognition method using raw sEMG and by 1.13% more than the existing personal recognition study. These results verify that the personal recognition method proposed in this paper can increase feature data and improve personal recognition performance without overlapping sEMG signals. In a future study, information without activated muscles will be eliminated from the sEMG, and the number of subjects for the personal recognition experiment will be increased.
This research was results of a study on the “HPC Support” Project, supported by the ‘Ministry of Science and ICT’ and NIPA.
Funding Statement: This research was supported by Basic Science Research Program through the National Research Foundation of Korea (NRF) funded by the Ministry of Education (No. 2017R1A6A1A03015496) and the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (No. NRF-2021R1A2C1014033).
Conflicts of Interest: The authors declare that they have no conflicts of interest to report regarding the present study.
Data Availability: The Ninapro DB2 used to support the findings of the study have been deposited in Ninaweb (http://ninaweb.hevs.ch/).
ReferencesS.Nowaczewski and W.Mazurczyk, “Securing feature internet and 5G using customer edge switching using DNSCrypt and DNSSEC,” Journal of Wireless Mobile Networks, Ubiquitous Computing, and Dependable Applications, vol. 11, no. 3, pp. 87–106, 2020.H.Kim, “5G core network security issues and attack classification from network protocol perspective,” Journal of Internet Services and Information Security, vol. 10, no. 2, pp. 1–15, 2020.S.Fonyi, “Overview of 5G security and vulnerabilities,” The Cyber Defense Review, vol. 5, no. 1, pp. 117–134, 2020.D. A.Hahn, A.Munir and S. P.Mohanty, “Security and privacy issues in contemporary consumer electronics [energy and security],” IEEE Consumer Electronics Magazine, vol. 8, no. 1, pp. 95–99, 2018.P.Huang, L.Guo, M.Li and Y.Fang, “Practical privacy-preserving ECG-based authentication for IoT-based healthcare,” IEEE Internet of Things Journal, vol. 6, no. 5, pp. 9200–9210, 2019.N.Karimian, P. A.Wortman and F.Tehranipoor, “Evolving authentication design considerations for the internet of biometric things (IoBT),” in Proc. CODES, New York, NY, USA, pp. 1–10, 2016.M.Karuna and S. R.Guntur, “EMG signal analysis using intrinsic mode functions to discriminate upper limb movement,” in Proc. AISP, Amaravati, India, pp. 1–3, 2020.X.Song, S.Guo, B.Gao and W.Wang, “Motion recognition of the bilateral upper-limb rehabilitation using sEMG based on ensemble EMD,” in Proc. ICMA, Tianjin, China, pp. 1637–1642, 2014.Z.Sun, X.Xi, C.Yuan, Y.Yang and X.Hua, “Surface electromyography signal denoising via EEMD and improved wavelet thresholds,” Mathematical Biosciences and Engineering, vol. 17, no. 6, pp. 6945–6962, 2020.S. H.Shin, J. H.Jung, M. G.Kang and Y. T.Kim, “A study on EMG signal acquisition modules and artificial neural networks for personal authentication,” in Proc. CSCI, Las Vegas, NV, USA, pp. 1458–1459, 2018.S. H.Kim, J. H.Ryu, B. H.Lee and D. H.Kim, “Human identification using EMG signal based artificial neural network,” Journal of the Institute of Electronics and Information Engineers, vol. 53, no. 4, pp. 142–148, 2016.M. R.Lee, J. H.Ryu and I. C.Youn, “Biometric personal identification based on gait analysis using surface EMG signals,” in Proc. ICCIA, Beijing, China, pp. 318–321, 2017.H.Yamaba, T.Kurogi, K.Aburada, S. I.Kubota, T.Katayamaet al., “On applying support vector machines to a user authentication method using surface electromyogram signals,” Artificial Life and Robotics, vol. 23, no. 1, pp. 87–93, 2018.L.Lu, J.Mao, W.Wang, G.Ding and Z.Zhang, “A study of personal recognition method based on EMG signal,” IEEE Transactions on Biomedical Circuits and Systems, vol. 14, no. 4, pp. 681–691, 2020.M. U.Khan, Z. A.Choudry, S.Aziz, S. Z. H.Naqvi, A.Ayminet al., “Biometric authentication based on EMG signals of speech,” in Proc. ICECCE, Istanbul, Turkey, pp. 1–5, 2020.A. M. H.Wong, M.Furukawa, H.Ando and T.Maeda, “Dynamic hand gesture authentication using electromyography (EMG),” in Proc. SII, Honolulu, HI, USA, pp. 300–304, 2020.Q.Li, P.Dong and J.Zheng, “Enhancing the security of pattern unlock with surface EMG-based biometrics,” Applied Sciences, vol. 10, no. 2, pp. 541, 2020.S.Morikawa, S.Ito, M.Ito and M.Fukumi, “Personal authentication by lips EMG using dry electrode and CNN,” in Proc. IOTAIS, Bali, Indonesia, pp. 180–183, 2018.M. G.Kim, “A study on user recognition system based on ensemble convolutional neural networks using synthetic electrocardiogram generation,” Ph.D. dissertation, Chosun University, Republic of Korea, 2019.Y.Zhang, P.Xu, P.Li, K.Duan, Y.Wenet al., “Noise-assisted multivariate empirical mode decomposition for multichannel EMG signals,” Biomedical Engineering Online, vol. 16, pp. 107, 2017.H.Ashraf, A.Waris, S. O.Gilani, M. U.Tariq and H.Alquhayz, “Threshold parameters selection for empirical mode decomposition-based EMG signal denoising,” Intelligent Automation & Soft Computing, vol. 27, no. 3, pp. 799–815, 2021.D. H.Kisa, M. A.Ozdemir, O.Guren and A.Akan, “EMG based hand gesture classification using empirical mode decomposition time-series and deep learning,” in Proc. TIPTEKNO, Antalya, Turkey, pp. 1–4, 2020.S.Aziz, S. Z. H.Naqvi, M. U.Khan and T.Aslam, “Electricity theft detection using empirical mode decomposition and k-nearest neighbors,” in Proc. ICETST, Karachi, Pakistan, pp. 1–5, 2020.N. E.Huang, X.Chen, M. T.Lo and Z.Wu, “On Hilbert spectral representation: A true time-frequency representation for nonlinear and nonstationary data,” Advances in Adaptive Data Analysis, vol. 3, no. 2, pp. 63–93, 2011.C.Sapsanis, G.Georgoulas, A.Tzes and D.Lymberopoulos, “Improving EMG based classification of basic hand movements using EMD,” in Proc. EMBC, Osaka, Japan, pp. 5754–5757, 2013.M.Taherisadr, P.Asnani, S.Salster and O.Dehzangi, “ECG-based driver inattention identification during naturalistic driving using mel-frequency spectrum 2-D transform and convolutional neural networks,” Smart Health, vol. 9–10, pp. 50–61, 2018.M.Atzori, A.Gijsberts, C.Castellini, B.Caputo, A. G. M.Hageret al., “Electromyography data for non-invasive naturally-controlled robotic hand prostheses,” Scientific Data, vol. 1, pp. 140053, 2014.J. W.Lee, H. W.Lee and C. S.Yoo, “Selection of mother wavelet for bivariate wavelet analysis,” Journal of Korea Water Resources Association, vol. 52, no. 11, pp. 905–916, 2019.