Artificial neural networks (ANNs) have been extensively studied as flexible function approximators and parametric models across a wide range of scientific and engineering tasks. General surveys and application-driven reviews document the breadth of ANN deployments and motivate their use in identification and control, particularly when explicit first-principles modeling is difficult or when data-driven adaptation is required [1–3]. Foundational treatments established the core modeling paradigms and training principles, including linear and nonlinear network structures and their algorithmic implementations [4–6]. In this classical view, many practical learning rules can be interpreted as iterative optimization procedures acting on a quadratic or near-quadratic objective, a perspective that remains central to modern online learning formulations.

A large portion of the ANN training literature is rooted in incremental (first-order) updates. Recent theoretical analyses of learning dynamics in deep linear networks provide explicit characterizations of transient behavior and the interaction between initialization, regularization, and optimization geometry under gradient-based training [7,8]. Within the quadratic-loss setting, basic gradient-descent rules and their stochastic variants remain canonical examples of first-order adaptation mechanisms and highlight how data statistics, conditioning, and step-size choices shape stability and speed of learning [9,10]. In parallel, modern energy-based learning continues to emphasize the role of objective shaping and conditioning in learnability [11]. These works collectively support the view that optimization geometry—not only model expressiveness—plays a decisive role in whether training proceeds smoothly or becomes trapped in slow transient regimes.

Beyond standard multilayer architectures, several specialized ANN families have been developed for robustness, interpretability, and control-oriented deployment. Radial basis function networks and their robust variants provide a well-established pathway to stable approximation under uncertainty and noise [12,13]. In adaptive and self-learning control, ANN-based schemes have been reported for real-time compensation and online tuning, where training must remain stable under streaming data and limited numerical precision [14]. Related approaches in intelligent control also include fuzzy and cerebellar-model architectures that stress adaptation under nonlinearities and disturbances [15,16]. Complementary lines of research continue to refine computationally efficient first-order training, including recent advances in stochastic gradient descent variants [17].

A particularly demanding class of identification problems arises in nonlinear and chaotic dynamics, where sensitivity to initial conditions and measurement noise can degrade learning reliability. Recent studies illustrate both the feasibility of parameter identification in chaotic systems and the numerical challenges of learning from chaotic trajectories [18–20]. These challenges are amplified in embedded or resource-constrained implementations, where finite-precision arithmetic and strict real-time requirements can exacerbate ill-conditioning and lead to slow or stagnant learning. Recent reviews on hardware realizations of neural methods, including FPGA-oriented implementations and embedded control applications, highlight the practical importance of training rules that remain stable and well-conditioned under limited precision. Consistent with these trends, widely used embedded platforms and rapid-prototyping toolchains have enabled end-to-end experimental validation of online learning strategies on microcontrollers [21,22].

Despite the breadth of architectures and applications, an enduring challenge in first-order online training is the susceptibility to slow plateaus and weight growth when the regressor is ill-conditioned or when optimization directions become nearly flat. Classical quadratic (L2/Tikhonov) regularization is a standard remedy to improve conditioning, yet fixed regularization may introduce steady-state bias when the target objective is the unregularized least-squares criterion. This motivates strategies that improve early-stage conditioning while preserving asymptotic fidelity to the original objective, which is the central perspective adopted in this work.

Single-layer ANNs with a linear activation function are equivalent to linear regression models and are widely used to represent linear-in-parameters structures in system identification, adaptive filtering, and control. In these applications, the model output can be written as a linear combination of known regressors and unknown parameters, so that training reduces to minimizing a least-squares functional. Although a closed-form solution exists for batch least squares, embedded and real-time settings often require iterative, lightweight, and online algorithms. For this reason, first-order methods based on gradient descent remain attractive due to their low computational complexity and ease of implementation [2,4,17].

In practice, however, standard gradient descent may perform poorly when the regressor data are ill-conditioned or nearly rank-deficient. These situations are frequent in online identification problems with delayed signals, correlated regressors, or limited excitation. The resulting cost surface can contain nearly flat directions, which leads to slow progress, long plateaus, and very large intermediate weights. On finite-precision hardware, such dynamics can manifest as training stagnation and numerical instabilities that are commonly described as neural-paralysis-like plateau behavior in the neural-network literature [23]. In the linear setting considered here, the effect is not caused by saturation of nonlinear activation functions, but rather by loss of curvature and poor conditioning of the quadratic objective.

A standard remedy is to augment the least-squares loss with a quadratic (Tikhonov) regularizer, which enforces strong convexity and penalizes large weights. Fixed regularization, however, introduces a bias: the minimizer of the regularized problem does not generally coincide with the minimizer of the original least-squares cost. This trade-off is particularly undesirable in identification tasks, where asymptotic accuracy is essential.

This work proposes GD-TDR (Gradient Descent Algorithm with Regularizer—Time Decay), a first-order scheme that interpolates between these two extremes. The algorithm employs a quadratic regularizer whose coefficient decays exponentially over time. As a result, the early iterations benefit from improved curvature and bounded weights, while the regularization vanishes asymptotically and the algorithm recovers the minimizer of the unregularized least-squares functional.

The main contributions of this work are summarized as follows:

a unified analytical framework that explicitly connects classical gradient descent, gradient descent with fixed quadratic (L2) regularization, and the proposed Gradient Descent with Time-Decaying Regularization (GD-TDR) through a single decay parameter, thereby clarifying their structural similarities and fundamental differences;

The paper is organized as follows. Section 2 introduces the linear ANN model and the least-squares training objective. Section 3 compares gradient-descent training schemes and presents GD-TDR. Section 4 states and proves the convergence theorem and provides the GD-TDR pseudocode. Section 5 shows how common identification models can be written in linear ANN form. Sections 6 and 7 report the numerical and embedded validation, respectively. Section 8 concludes the paper and outlines future work.

Consider a single-layer ANN with linear activation (purelin) and no bias. Its output is (1)y^:=w⊺x,where the input vector is x:=[x1 x2 ⋯ xn]⊺ and the weight vector is w:=[w1 w2 ⋯ wn]⊺. Fig. 1 illustrates the architecture.

Given a finite data set {(xk,yt,k)}k=1N, define the prediction errors ek:=y^k−yt,k and the quadratic cost (2)J(w):=1N∑k=1Nek2.

Let X∈ℳn×N(R) be the regressor matrix and yt∈RN the target vector, (3)X:=[x1 x2 ⋯ xN],yt:=[yt,1 yt,2 ⋯ yt,N]⊺.

Then (4)J(w)=1N‖X⊺w−yt‖22.

The gradient and Hessian of J with respect to w are (5)∇J(w)=2NX(X⊺w−yt),∇2J(w)=2NXX⊺.

The Hessian is positive semidefinite. It is positive definite (and hence J is strongly convex) if and only if XX⊺ is nonsingular.

Using (2), the GD update with step size η>0 is (6)w(j+1)=w(j)−η∇J(w(j))=w(j)−2ηNXe(j),where e(j):=X⊺w(j)−yt. When J is strongly convex and η is chosen appropriately, GD converges linearly to the unique minimizer. If XX⊺ is singular or ill-conditioned, J is not strongly convex and GD may exhibit slow progress and large intermediate weights.

A standard approach to improve conditioning is to add a quadratic regularizer (7)Jγ(w):=J(w)+γ2w⊺Pw,where γ>0 and P∈ℳn×n(R) is symmetric positive definite. The Hessian becomes (8)∇2Jγ(w)=2NXX⊺+γP≻0,so Jγ is strongly convex for any γ>0. The GD-QR update reads (9)w(j+1)=w(j)−η(2NXe(j)+γPw(j)).

Fixed regularization controls weight growth and improves curvature, but the minimizer of Jγ generally differs from the minimizer of J. In identification problems, this bias can be detrimental.

GD-TDR replaces the constant regularization weight γ in (9) by a time-decaying sequence (10)γ(j):=γ0λj,γ0>0,0<λ<1.

The weight update becomes (11)w(j+1)=w(j)−η(2NXe(j)+γ(j)Pw(j)).

Two limiting cases highlight the relationship among the three schemes:

Classical GD: setting γ0=0 yields (6).

Therefore, GD-TDR provides a continuous mechanism to improve conditioning early in training while asymptotically removing the regularization bias. The theoretical properties of this scheme are stated in Theorem 1.

Consider a discrete transfer function (27)Y(z)U(z):=anz−s+an−1z−s+1+⋯+a1z−1+a0bmz−m+bm−1z−m+1+⋯+b1z−1+1,with s≤m. After inverse z-transform (zero initial conditions), one obtains an ARX-like representation (28)y[n]=∑k=0saku[n−k]−∑r=1mbry[n−r].

Defining (29)w:=[a0a1⋯asb1⋯bm]⊺,x[n]:=[u[n]u[n−1]⋯u[n−s]−y[n−1]⋯−y[n−m]]⊺.

The model becomes y[n]=w⊺x[n], which is exactly the output of a single-layer linear ANN. Fig. 2 sketches this identification setup.

For a discrete state-space (DSS) system (30)x[n+1]=Ax[n]+Bu[n],with appropriate dimensions, the right-hand side is linear in the unknown entries of A and B. By stacking the parameters into a single vector (or matrix) and defining a regressor vector that contains state and input components, the DSS update can also be written in the form x[n+1]=Wϕ[n], where W collects the unknown parameters. This is compatible with GD-TDR, which can be applied entrywise.

A practical issue is causality: to update parameters at time n, the target x[n+1] is needed. A simple remedy is to insert a one-step delay in the learning loop, which preserves the identification objective while keeping the update implementable in real time. Fig. 3 illustrates the ANN view of the DSS model, while Fig. 4 shows a delay-based identification for DSS parameter identification.

Remark 2: Although the focus is on linear-in-parameters models, the same idea can be used to identify local linearizations of nonlinear systems around operating points, provided the regressors are constructed accordingly.

The numerical validation considers online parameter identification of a chaotic system whose dynamics are equivalent to those of a Chua-type oscillator. Chaotic trajectories provide a demanding excitation pattern for adaptive algorithms and are known to expose slow transients and stagnation effects in gradient-based learning [19,20].

The Chua oscillator is described by (31){x˙=α(y−x−φ(x)),y˙=x−y+z,z˙=−βy,where (32)φ(x):=m1x+12(m0−m1)(|x+1|−|x−1|).

For the parameter values α=15.6, β=25, m0=−1.1429, and m1=−0.7143, the corresponding attractor is shown in Fig. 5.

To pose an identification problem that is linear in the unknown parameters, the dynamics are rewritten as (33)[x˙y˙z˙]⏟yt=[w11w12w13w14w21w22w23w24w31w32w33w34]⏟W[xyzd]⏟ϕ,d:=|x+1|−|x−1|.

In this form, W is an unknown parameter matrix to be estimated online from the measured signals. The validation compares GD-TDR against classical GD. In addition, Section 3 provides an explicit comparison with fixed quadratic regularization (GD-QR), which is obtained as the special case λ=1.

Fig. 6 shows the main program used in the numerical simulations.

In the reported tests, the learning factor was set to η=0.02. For GD-TDR, the regularization parameters were initialized at γ0=0.99 and decayed according to (10) with λ=0.9998. The simulations used the ode8 solver with fixed step size and sampling time 0.001s.

Fig. 7 shows the convergence of the estimated parameters towards the target values, while Fig. 8 reports the norm of the estimation error. The final identified parameters for GD-TDR, GD and GD-QR are listed in (34) to (36), respectively.

(34)WGD-TDR=[−4.45715.6000.0003.3431.000−1.0001.0000.0000.000−25.0000.0000.000],NE=0.0001. (35)WGD=[−4.43715.5800.0013.3270.997−0.9971.0000.0020.0202−24.9870.0000.0151],NE=0.0013. (36)WGD-QR=[−3.312414.32320.09692.59130.8175−0.87120.99080.1169−1.0477−23.4959−0.32020.5065],NE=0.0419.

GD and GD-TDR algorithms converge to high-accuracy estimates; however, GD-TDR exhibits markedly faster convergence and substantially shorter stagnation transients. In the considered experiment, GD requires approximately 800s to reach steady convergence, whereas GD-TDR attains comparable accuracy in about 200s. The error norm (NE) trajectories further indicate that GD spends a significant portion of the runtime in plateau-like regions before converging, a behavior consistent with ill-conditioning and the presence of flat directions in the quadratic objective function.

GD-QR mitigates oscillations during the convergence process but yields the poorest parameter convergence (NE=0.0419). This behavior is expected, since the QR-based transformation effectively shifts the original optimal point to an alternative one in order to increase the convexity of the cost functional, thereby smoothing the convergence dynamics at the expense of final parameter accuracy.

From an algorithmic viewpoint, these improvements can be interpreted through Theorem 1: the decaying regularization increases the curvature of Jj in the early iterations, preventing the gradient from vanishing far from the minimizer and discouraging weight growth in nearly-flat directions. As γ(j) decreases, the method smoothly transitions toward the unregularized least-squares objective, thereby avoiding the steady-state bias that would occur if a fixed γ were used (GD-QR).