As a streamlined variant of LSTM, GRU features just two gate mechanisms: an update gate and a reset gate [32]. The internal configuration of GRU, shown in Fig. 1, together with the standard forward propagation equations, is detailed below: (2)zt=σ(Wz[ht−1,xt]+bz) (3)rt=σ(Wr[ht−1,xt]+br) (4)h~t=tanh⁡(Wh~[rt∘ht−1,xt]+bh~) (5)ht=zt∘h~t+(1−zt)∘ht−1

The following are the explanations for the related symbols:

The symbol ∘ stands for the Hadamard product, which refers to element-wise multiplication.

In (2), Wz denotes the weight matrix associated with the update gate. In fact, Wz is formed by concatenating two matrices: Wz,h, which corresponds to the input vector ht−1, Wz,x, which corresponds to the input vector. Therefore, Eq. (2) can be written as: (6)[Wz][ht−1xt]=[Wz,hWz,x][ht−1xt]=Wz,hht−1+Wz,xxt

Obviously, the weight matrix in other Eqs. (3)–(5) can be also rewritten in the same form as (6). For the convenience of subsequent analysis, we set the biases br, bz, and bh~ to 0 and get new expressions as follows: (7)zt=σ(Wz,h⋅ht−1+Wz,x⋅xt) (8)rt=σ(Wr,h⋅ht−1+Wr,x⋅xt) (9)h~t=tanh⁡(Wh~⋅(rt∘ht−1)+Wh~⋅xt) (10)ht=(1−zt)∘ht−1+zt∘h~t

If {xn,Tn}n=1N⊂RN×RN as the given set of the training samples, where the xn represent the n-th input sample and the Tn is the label, respectively. Let ytn=σ(wout⋅htn)∈R be the actual output for each input Xn, and yt0=wout ⋅ht. Thereby, the error function is defined by the following formula: (11)E~(W)=12N∑n=1N(ytn−Tn)2=12N∑n=1N(σ(wout ⋅htn)−Tn)2=1N∑n=1Nσn(wout ⋅htn)where σn(r)=12(σ(r)−Tn)2,r∈R,1≤n≤N.

The standard approach to achieve L1 regularization entails incorporating a penalty term within the error function, expressed as: (12)E(W)=E~(W)+λ‖wout ‖1

This can be written as: (13)E(W)=∑n=1Nσn(wout ⋅htn)+λ‖wout ‖1where λ>0 is the penalty parameter, while ||⋅||1 indicates L1-norm.

However, there is no derivative of the L1 regularization term at the origin [33,34]. To tackle the non-differentiable problem of the L1 regularization term, a smoothing approximation function is introduced. Smoothing approximation is essentially the use of continuous differentiable functions instead of absolute value functions. In this paper, a quadratic form smoothing approximation function is used, which means: (14)h(x)={|x|,|x|≥α|x|22α+α2,|x|<α

The smoothing coefficient α is a constant greater than zero. Fig. 2 illustrates the effect of α on the degree of approximation. It is easy to see that when the smoothing coefficient α tends to zero, the approximation function increasingly resembles the absolute function. Therefore, in practical applications, the smaller the smoothing coefficient, the closer the actual effect of the regularization term is to the L1 regularization method.

By incorporating a smooth approximation function into the error propagation mechanism of L1 regularized GRU, the issue of non-differentiability at the origin is overcome, providing a basis for a rigorous analysis of the error function’s monotonicity. Specifically, the error function expression of the smoothed SL1-GRU model is derived as follows by replacing the L1 regularization term with the smoothed approximation function L1(Wout): (15)E=∑n=1Nσn(Wout ⋅htn)+λL1(Wout ),λ>0

The element L1(Wouti,j) is positioned in the i-th row and j-th column of the matrix L1(Wout). Specifically, L1(Wouti,j) is defined as follows: (16)L1(Wouti,j)={|Wouti,j|,if |Wouti,j|≥α|Wouti,j|22α+α2,if |Wouti,j|<αhere, α is a given bounded constant.

The optimization algorithm, Stochastic Gradient Descent (SGD), is frequently used to train GRU. To achieve the fastest reduction of the error function E, the direction of weight changes should be the same as the negative gradient of E in the weight matrix. The learning rate, symbolized by η, is a scalar hyperparameter that determines the step increment for each iteration in the optimization algorithm. ∇WE represents the partial derivative of the error function E with respect to the weight W. If Wk and Wk+1 denote the weight matrices for the k-th and (k+1)-th iterations, respectively, and ΔWk represents the change in the weight matrix from Wk to Wk+1. The weight update rule for the SGD algorithm is defined as: (17)Wk+1=Wk+ΔWk=Wk−η∇WE

This equation indicates that during each iteration, SL1-GRU updates the weights by deducting the result of multiplying the learning rate η by the gradient of the error function with respect to the weights from the current weight matrix Wk, causing the weights to change in a direction that reduces the error function. By iteratively applying this rule, the weights are adjusted to minimize the error.

Define δh,tk as the partial derivative of E over htk, and it is given by: (18)δh,tk=∂E∂htk

Similarly, define: (19)δz,tk=∂E∂ztk∘ztk∘(1−ztk)=δh,tk∘(h~tk−ht−1k)∘ztk∘(1−ztk) (20)δr,tk=∂E∂rtk∘rtk∘(1−rtk)=ht−1k∘[(δh,tk∘ztk∘(1−(h~tk)2t)Whk]∘rtk∘(1−rtk) (21)δh~,tk=∂E∂h~tk∘(1−(h~tk)2)=δh,tk∘ztk∘(1−(h~tk)2)

For each weight matrix, the partial derivatives of E are as follows: (22)∇Wz,hkE=∂E∂Wz,hk=∑i=1tδz,ikhi−1k, (23)∇Wz,xkE=∂E∂Wz,xk=∑i=1tδz,ikxik, (24)∇Wr,hkE=∂E∂Wr,hk=∑i=1tδr,ikhi−1k, (25)∇Wr,xkE=∂E∂Wrx=∑i=1tδr,ikxik, (26)∇Wh~,rkE=∂E∂Whk=∑i=1tδik(rik∘hi−1k), (27)∇Wh~,xkE=∂E∂Wxk=∑i=1tδikxik,

For the output weight matrix Wout, the partial derivative of E specifically is: (28)∂E∂Woutk=∑n=1Nσn′(Woutk⋅htk,n)ΔWoutk⋅htk,n+λ∂L1(Wout)∂Woutwhere λ>0.

According to (17) and (22) to (28), the weights are updated iteratively by: (29)Wz,hk+1=Wz,hk+ΔWz,hk=Wz,hk−η∂E∂Wz,hk (30)Wz,xk+1=Wz,xk+ΔWz,xk=Wz,xk−η∂E∂Wz,xk (31)Wr,hk+1=Wr,hk+ΔWr,hk=Wr,hk−η∂E∂Wr,hk (32)Wr,xk+1=Wr,xk+ΔWr,xk=Wr,xk−η∂E∂Wr,xk (33)Wh~,rk+1=Wh~,rk+ΔWh~,rk=Wh~,rk−η∂E∂Wh~,rk (34)Wh~,xk+1=Wh~,xk+ΔWh~,xk=Wh~,xk−η∂E∂Wh~,xk (35)Woutk+1=Woutk+ΔWoutk=Woutk−η∂E∂Woutk

Based on the above analysis, the SL1GRU algorithm flow is presented in Algorithm 1.

To demonstrate the generalization capabilities of SL1-GRU, we approximate a one-dimensional function f(x) and a two-dimensional function q(x,y) in this section. The mathematical expressions of these functions are as follows:

Nonlinear oscillatory function: (40)f(x)=8+2e1−x2cos⁡(2πx),x∈[−0.5,3.5]

The peaks function, commonly used in numerical experiments, defined as: (41)q(x,y)=3(1−x)2e−x2−(y+1)2−10(x5−x3−y5)e−x2−y2−13e−(x+1)2−y2,x,y∈[−2.5,2.5]

For the nonlinear oscillatory function (40), 100 points are uniformly distributed in the interval [−0.5,3.5] and denoted as xi for i=1,2,…,100, serving as inputs. The corresponding outputs are given by f(xi)+ϵi, where ϵi∼N(0,0.01). For the peaks function (41), a two-dimensional grid is generated with x,y uniformly sampled within [−2.5,2.5], resulting in 100 sample points. The outputs are perturbed by Gaussian noise ϵi,j∼N(0,0.01), yielding q(xi,yj)+ϵi,j. The network weights of six models (RNN, LSTM, GRU, L1-GRU, L2-GRU, SL1-GRU) are initialized randomly in [−0.5,0.5], with the learning rate η set to 0.001. Regularization coefficients for L1-GRU, L2-GRU and SL1-GRU are λ=0.0005, while the smoothing parameter for SL1-GRU is α=0.01.

Fig. 3a shows the approximation performance of RNN, LSTM, GRU, L1-GRU, L2-GRU, and SL1-GRU for the nonlinear target function f(x) in [0.5,3.5]. Regularized models (L1-GRU, L2-GRU, SL1-GRU) align more closely with the actual curve, with SL1-GRU achieving the best accuracy in oscillatory regions, highlighting its robustness in capturing nonlinear dynamics. Fig. 3b illustrates the sparsity evolution over training iterations. L1-GRU and SL1-GRU achieve significantly higher sparsity, stabilizing around 0.8 after 2000 iterations, while GRU and LSTM show lower sparsity, reflecting greater parameter complexity. These results demonstrate the effectiveness of regularization in promoting model sparsity.

Similarly, we approximate the two-dimensional function using the same approaches, with the approximation results of SL1-GRU presented in Fig. 4. The results highlight SL1-GRU’s ability to effectively capture global trends and local variations.

This part presents an evaluation and comparison of the classification efficacy for RNN, LSTM, GRU, L1-GRU, L2-GRU, and SL1-GRU. Table 1 is a summary of the dataset utilized in the simulation experiment. The network weights are randomly initialized in [−0.5,0.5]. Each network is set up with a hidden layer of 32 nodes. The dataset’s features determine the input layer’s node count, while the number of output layer nodes equals the count of classes.

As shown in Fig. 5, we use grid search to explore the hyperparameter space by testing combinations of learning rate η, regularization factor λ, and smoothing coefficient α within predefined ranges. Each combination of these hyperparameters is evaluated using k-fold cross-validation to ensure robust and reliable performance metrics. The evaluation criterion is based on the test accuracy achieved by SL1-GRU, aiming to identify the parameter set that maximizes accuracy while maintaining generalization. It is determined that {α=0.01,λ=0.00005,η=0.005} constitutes the optimal parameter combination for the wine dataset, achieving the highest test accuracy for SL1-GRU. This approach is similarly applied to other datasets, and the results, summarized in Table 2, highlight the effectiveness of grid search in identifying optimal hyperparameters.

Table 3 compares the training accuracy, test accuracy, sparsity, and training time of different models on the same dataset. These experimental results represent the average values obtained over 10 trials. Sparsity, defined as the ratio of elements in the neural network’s weight matrix that are less than 1×10−5 to the total number of elements in the weight matrix, is used as an indicator of network sparsity. Mathematically, it can be expressed as: (42)Sparsity=Num0Numnwhere the number of elements in the weight matrix that are less than 1×10−5 is denoted by Num0, and Numn represents the overall element count of the matrix. It can be observed in Table 3 that although the training accuracy of SL1-GRU may not be the highest, its test accuracy is consistently the best across all datasets, highlighting its excellent generalization ability. Moreover, both L1-GRU and SL1-GRU exhibit significantly higher sparsity compared to other models. Except for one dataset, SL1-GRU achieves the highest sparsity, demonstrating that the proposed method effectively enhances network sparsity. Additionally, benefiting from its superior sparsity, SL1-GRU requires the shortest training time, indicating that it significantly improves computational efficiency.

From Fig. 6, it can be observed that the loss function curve of SL1-GRU monotonically decreases and gradually stabilizes at zero as the number of iterations increases, which verifies Theorem 1. Meanwhile, in Fig. 6b, the gradient curve of SL1-GRU decreases the fastest, and as the number of iterations approaches infinity, its gradient also tends to zero, consistent with Theorem 2. Fig. 6c shows that the weight curves of L1-GRU and SL1-GRU do not grow indefinitely, indicating that both regularization methods effectively suppress weight growth. Among them, SL1-GRU is more effective in constraining network weights, stabilizing them around a constant value of approximately 140, aligning with Theorem 3.

The performance of SL1-GRU in regression tasks is also considered. The dataset utilized in this part is detailed in Table 4. For RNN, LSTM, GRU, L1-GRU, L2-GRU, and SL1-GRU, the hidden layer is designed with 32 nodes. The nodes in both the input and output layers are configured based on the dataset’s features and labels, respectively. The learning rate is established at η=1×10−3, the regularization factor at λ=3×10−4, and the smoothing coefficient at α=0.01. The initial weight range is [−0.5,0.5] as in the previous part.

In the evaluation of regression models, the standard metric used is Mean Squared Error (MSE), which is calculated using the following formula: (43)MSE=1n∑i=1n(predi−truei)2,where pred1,pred2,…,predn indicate the predicted values, and the set of actual values is denoted by true1,true2,…,truen.

Table 5 shows that the Test MSE of SL1-GRU is consistently the smallest, indicating that it performs the best on the test set and has the strongest generalization ability. From the perspective of sparsity, the network weights of SL1-GRU remain the sparsest, which suggests that it eliminates unimportant parameters to enhance the computational efficiency of the model while maintaining its excellent performance.