One of the challenging problems in dynamic thresholding in order to detect the signal and to locate the “genuine” signal frame is to compute an offset voltage for each Electric Potential Sensor (EPS) and adjust the threshold values periodically before detecting the target hand motion. As most hand motions and gestures are being done within a short period of time, we set the update cycle time unit to be a second. We implemented two EPS (Sensor A, B, sensor type PS25401) [14,23], and each sensor started with unidentical initial offset due to the various electric charging and discharging states near the sensors according to diverse environmental conditions in the moment of time.

Through our empirical past simulations, the initial voltage vinit∈R (unit: V) ranges from −0.2≤vinit≤0.02 with μ(∀vinit)=−0.08 and σ(∀vinit)=0.03 where μ denotes average and σ is standard deviation, when implementing EPS to measure the capacitance changes when the subject is proximally located. The statistics were acquired from four distinct hand gesture types (Tab. 1), each conducting 600 trials; n(∀vinit)=600, and the distribution visualization of vinit is shown in Fig. 2a. Let S={vn|1≤n≤T⋅1000,n∈N} be a set of time series raw sensor (voltage) data vn, where the sampling rate fs=1T=0.001sec which n(S)=1000 when T=0.001sec.

While a sensor is operational, which would constantly generate S, two thresholds εu, εℓ are being computed dynamically in every T=0.5sec, having εu=v(n,T)+τ and εℓ=v(n,T)−τ^ where εu, εℓ each indicates upper and lower threshold, v(n,T) is signal value on index n and time T. Through 12,000 trials of Monte Carlo simulation, we empirically estimated the optimal τ,τ^= 0.02 when the distance between the EPS and the subject's hand is 10∼30 cm, as 0.02 is sufficient enough to encompass the signals in order to detect the actual hand gesture signal. To prove this, we measured the Correct Detection Rate (CDR) which is suggested in Section 3. Note that the signal decreases exponentially as the distance increases linearly, since density of the electromagnetic signal are inverse proportional to the squared distance; E(V/m)=Q4πε0R2aR and F(N/C)=Q1Q24πε0R2 where Q indicates an electric charge, R is a distance, ε0 is a dielectric constant and aR is an unit vector. S1 and S2 produces similar signal patterns while each set was produced from single analogous hand motion, however the starting signal offset and their gradient Δv=vn−vn−1 in both identical index could be disparate as amount of the internal charges of the subject could vary; ε(S1,u,ℓ)(T1)≠ε(S2,u,ℓ)(T2) and ΔvS1≠ΔvS2 where T1≠T2 and (T1,T2)=0.5⋅c with constant c∈N.

Likewise, due to heterogeneous property of S, dynamic ε(S,u,ℓ)(T) tracks the discrete variant vn(T) and its upper and lower bound, which maintains the robustness regardless of time-variant offset value. As for the time-varying signal pattern which is the electric field disturbances due to hand motion, charging and discharging the electrics triggers the voltages to display waveform. Figs. 2b and 2c shows the typical output signal that charges and discharges the sensor plate which soars up (otherwise falls down), reaching its peak (bottom), then descends to bottom point (vice versa) and finally returns to an initial level when there was a hand gesture near the sensor plate.

Due to these features, dynamic ε(S,u,ℓ)(T) are considered to be effective in order to detect our targets which limits the possible starting point of a next hand motion, and we empirically show its detection rate in Section 3.

Each individual hand motion generates unidentical signal phases depending on various conditions such as hand movement speed, distance and direction. In order to detect and extract the corresponding motion frames, we considered not only the duration, but also the time frame that could be divided into the stages of motion. The detection is composed of 4 steps where the left and right term enclosed with braces each indicate the two contrasting cases (case 1, 2). The detection steps are indicated as follows; Step1.{(vn−1(T)≤ε(S,u)(T))&&(vn(T)≥ε(S,u)(T))}||{(vn−1(T)≥ε(S,ℓ)(T))&&(vn(T)≤ε(S,ℓ)(T))} Step2.{(vn−1(T)≥ε(S,u)(T))&&(vn(T)≤ε(S,u)(T))}||{(vn−1(T)≤ε(S,ℓ)(T))&&(vn(T)≥ε(S,ℓ)(T))} Step3.{(vn−1(T)≥ε(S,ℓ)(T))&&(vn(T)≤ε(S,ℓ)(T))}||{(vn−1(T)≤ε(S,u)(T))&&(vn(T)≥ε(S,u)(T))} Step4.{(vn−1(T)≤ε(S,ℓ)(T))&&(vn(T)≥ε(S,ℓ)(T))}||{(vn−1(T)≥ε(S,u)(T))&&(vn(T)≤ε(S,u)(T))}

Note that when vn(T)∈S satisfies the first step, it halts updating the ε(S,u,ℓ)(T) and continues to update after step 4. Once the set of input S fully satisfies the above conditions, we acknowledge that the motion has been detected. An initial motion frame size is computed based on the vn′(T′) of Step 1 to vn′′(T′′) of the Step 4; [vn′(T′),vn′′(T′′)]. The initial frame Finit={vi|n′≤i≤n′′,i∈N}.

However, this would trigger to lose the intrinsic features as this frame merely includes the partial actual motion signals as there may be more features on vn(T^) where T^⟨T′&&T^⟩T′′. Thus, the final frame F^ is determined by extending additional 10% of the |n′′−n′| to the Finit in each preliminary left and right directions; F^={vj|(n′−|n′′−n′|⋅0.1)≤j≤(n′′+|n′′−n′|⋅0.1),j∈N}. Fig. 3 shows the motion detection and frames extraction visualization result, which were automatically performed in real-time through dynamic threshold ε(S,u,ℓ)(T) as the sensors were consistently generating the dataset.

In Fig. 3, a signal of hand gesture moved from left to right (LR) is shown. As we make another hand gesture, it detects the motion and locates the following frame continuously. Fig. 3 contains two motion signals; each signal obtained from two different sensors (A-red and B-blue), where the dotted lines intersected with the signals show the starting point of the frame (magenta dot) and ending point of the frame (black dot). Each red and blue dot indicates the intersection point where ε(S,u,ℓ)(T)= vi. Through these steps we could successfully distinguish the hand motions and compute the significant frame that encompasses the authentic motion signals.

When hand motions are identified by the sensor, the time period of the extracted frame is diverse even if the motions were the same types, due to the speed or distance range of the motion. Likewise, the amplitude of the motion signal also tends to change on every new motion since the subject's potential electrostatic state varies through numerous conditions such as textile of the cloth, location, or nearby machines, etc [4,9,11,12]. Thus, it is imperative to conduct normalization in order to be properly trained in deep learning models as normalizing input data is known to be a productive measure to enhance the performance. In our signal, the time (X-axis) and voltage (Y-axis) are the two axes that are to be normalized. For Y-axis, standardization was applied and let F^(Y)={v^j|1≤j≤d,j∈N,v^j=vj−μ(F^)σ(F^)} where d=(n′′−|n′′−n′|⋅0.1)−(n′+|n′′−n′|⋅0.1), F^(Y)∼N(μ(F^)=0,σ(F^)=1). For X-axis, we normalized n(F^(Y)) into 1000, deleting d−1000 data in k-periodical sequence, where k=[dd−1000]. This leads to d⋅(k−1)k+m=1000, computing F^(X,Y)={v^λ|1≤λ≤1000,λ∈N}, where 0≤m<(d−1000). The normalized signal is shown in Fig. 4, where the dotted line indicates the extracted frame of clockwise (CW) hand gesture, meanwhile solid line is the result after the normalization. Note that even if the identical subject performs the same motion type in a homogeneous environment such as time and location, the phase of the signal is distinctive due to the constantly varying charging state.

After applying the normalized motion frame, the signal frame must be transformed from 1-dimension voltage signal to 2-dimension image in order to train the CNN model. Based on F^(X,Y), where n(F^(X,Y))=1000, we primarily reduce n(F^(X,Y)) into 900, deleting the v^1∼50 and v^950∼1000 in order to compute the 30 × 30 × 1 image (900=30⋅30). Fig. 5 shows the schematization of transforming the motion frame into the 30 × 30 × 1 image format.

In this section, we define the structure of our two CNN-based classifiers; MCNN and BPR-CNN that were implemented to effectively categorize the types of input hand motion signal images.

Through the empirical experiments, we evaluate the performance of our designed methodology. Utilizing the EPS sensor [14,28], four hand motion types indicated in Tab. 1 were extracted from each of six subjects, 100 gestures for each motion, collecting a total of 2400 motion samples. Among the dataset, we randomly split the 2160 samples for training and 240 for test data. The extracted raw signal was processed through consecutive signal processing methods starting from the LPF and SMA, followed by automatically detecting hand motions and setting signal frame by dynamic threshold, and normalizing the signal. Next, we transform the signal into an image and a pre-trained classifier determines its label. Note that η=4, since our objective is to classify the four motion types (Tab. 1). All this process (Fig. 1) is operated in real-time and test dataset were generated and classified (Less than second when computing through CPU i7–7500U RAM 8GB). The performance has been measured through our metrics of Correct Detection Rate (CDR), Motion Frame Matching Rate (MFMR) and Detection & Extraction Success Rate (DESR). CDR shows the degree of correspondence between the signal and the actual motion, and the MFMR quantitatively assesses the matching rate of the computed motion frame. Finally, DESR is obtained by CDR multiplied by MFMR to indicate their combined accuracy level. The training epoch was set with 20 and learning rate of 0.01, with relu activation function in each convolution layer.

Following Tab. 2 shows the result of our three designated metrics, which validates that the proposed method of our study works with high accuracy of around 98% on average.

Following Tab. 3 displays the experiment results of the four classifier algorithms. Their performances were evaluated with classification accuracy based on identical test data of four specific hand motions in Tab. 1. We denote the average of four motion accuracy as Classification Correction Rate (CCR), which is computed in Tab. 3. From each motion in Tab. 1, three distinct output signals are produced; sensor A value, sensor B value and subtracted value (A-B). Performance of the two CNN classifiers (MCNN and BPR-CNN) were also compared with other algorithms such as HMM and SVM. Our experiment results show that the suggested motion detection and frame extraction based on the two threshold works with high CDR and MFMR, and also the classification accuracy of BPR-CNN classifier outperformed other competitive models.