The Multi-Layer Perceptron (MLP) model is a type of feedforward Artificial Neural Network (ANN) that serves as a foundation architecture for deep learning or Deep Neural Networks (DNNs) [11]. It operates as a supervised learning approach. The MLP consists of three layers: the input layer, the output layer, and one or more hidden layers [12]. It is a fully connected network, meaning each neuron in one layer is connected to all neurons in the subsequent layer.

In an MLP, the input layer receives the input data and performs feature normalization. The hidden layers, which can vary in number, process the input signals. The output layer makes decisions or predictions based on the processed information [13]. Fig. 3a depicts a single-neuron perceptron model, where the activation function φ (Eq. (1)) is a non-linear function used to map the summation function (xw+b) to the output value y.

(1)y=φ (xw+b)

In Eq. (1), the terms x, w, b, and y represent the input vector, weighting vector, bias, and output value, respectively [14]. Fig. 3b illustrates the structure of the Multi-Layer Perceptron (MLP) model.

Convolutional Neural Networks (CNNs) are a powerful class of deep learning models widely applied in various tasks, including object detection, speech recognition, computer vision, image classification, and bioinformatics [15]. They have also demonstrated success in time series prediction tasks [16]. CNNs are feedforward neural networks that leverage convolutional structures to extract features from data [17]. CNN has a two-stage architecture that combines a classifier and a feature extractor to provide automatic feature extraction and end-to-end training with the least amount of pre-processing necessary [18]. Unlike traditional methods, CNNs automatically learn and recognize features from the data without the need for manual feature extraction by humans [19]. The design of CNNs is inspired by visual perception [17]. The major components of CNNs include the convolutional layer, pooling layer, fully connected layer, and activation function [20,21]. Fig. 4 presents the pipeline of the convolutional neural network, highlighting how each layer contributes to the efficient processing and successful progression of input data through the network.

Convolutional Layer: The convolutional layer is a pivotal component of CNN. Through multiple convolutional layers, the convolution operation extracts distinct features from the input. In image classification, lower layers tend to capture basic features such as texture, lines, and edges, while higher layers extract more abstract features. The convolutional layer comprises learnable convolution kernels, which are weight matrices typically of equal length, width, and an odd number (e.g., 3 × 3, 5 × 5, or 7 × 7). These kernels are convolved with the input feature maps, sliding over the regions of the feature map and executing convolution operations [22]. Fig. 5 illustrates the schematic diagram of the convolution process.

Pooling Layer: Typically following the convolutional layer, the pooling layer reduces the number of connections in the network by performing down-sampling and dimensionality reduction on the input data [23]. Its primary purpose is to alleviate the computational burden and address overfitting issues [24]. Moreover, the pooling layer enables CNN to recognize objects even when their shapes are distorted or viewed from different angles, by incorporating various dimensions of an image through pooling [25]. The pooling operation produces output feature maps that are more robust against distortion and errors in individual neurons [26]. There are various pooling methods, including Max Pooling, Average Pooling, Spatial Pyramid Pooling, Mixed Pooling, Multi-Scale Order-Less, and Stochastic Pooling [27–30]. Fig. 6 depicts an example of Max Pooling, where a window slides across the input, and the contents of the window are processed by a pooling function [31].

Fully Connected (FC) Layer: The FC layer is typically located at the end of a CNN architecture. In this layer, every neuron is connected to all neurons in the preceding layer, adhering to the principles of a conventional multi-layer perceptron neural network. The FC layer receives input from the last pooling or convolutional layer, which is a vector created by flattening the feature maps. The FC layer serves as the classifier in the CNN, enabling the network to make predictions [10].

Activation Functions: Activation functions are fundamental components in convolutional neural networks (CNNs), indispensable for introducing non-linearity into the network. This non-linearity is crucial for CNN’s ability to model complex patterns and relationships within the data, allowing it to perform tasks beyond simple linear classification or regression. Without non-linear activation functions, a CNN would be limited to linear operations, significantly constraining its capacity to accurately represent the intricate, non-linear behaviors typical of many real-world phenomena [32].

Fig. 7 typically illustrates how these activation functions modulate input signals to produce output, highlighting the non-linear transformations applied to the input data across different regions of the function curve. In this figure, xi represents the input feature, while wij denotes the weight associated with the connection between the input feature xi and neuron j. The figure shows that neuron j receives n features simultaneously. The output from neuron j is labeled by yj, and its internal state, or bias, is indicated by bj. The activation function, depicted as f(.), could be any one of several types such as the Rectified Linear Unit (ReLU), hyperbolic tangent (Tanh), Sigmoid function, or others [33,34].

These various activation functions are shown in Fig. 8, with emphasis on their distinct characteristics and profiles. These activation functions are essential for convolutional neural networks (CNNs) to be more effective in a variety of applications by allowing them to recognize intricate patterns and provide accurate predictions. Sigmoid and Tanh functions are frequently referred to as saturating nonlinearities due to the way they act when inputs are very large or small. As per the reference, the Sigmoid function approaches values of 0 or 1, whereas the Tanh function leans towards −1 or 1 [17]. Different alternative nonlinearities have been suggested for reducing problems associated with these saturating effects, including Rectified Linear Unit (ReLU) [35], Leaky ReLU [36], Parametric Rectified Linear Units (PReLU) [37], Randomized Leaky ReLU (RReLU) [38], S-shaped ReLU (SReLU) [39], and Exponential Linear Units (ELUs) [40], Gaussian Error Linear Units (GELUs) [41].

ReLU (Rectified Linear Unit) is one of the most often used activation functions in modern CNNs because of how well it solves the vanishing gradient issue during training. The definition of ReLU in mathematics is as Eq. (2), where the input to the neuron is represented by x [34]. (2)f(x)=max (0, x)={xi,if xi≥00,if xi<0

This feature helps CNN learn complicated features more efficiently by effectively “turning off” any negative input values while maintaining positive values. It also keeps neurons from being saturated during training.

As an alternative, the definition of the Sigmoid function is represented by Eq. (3), where x stands for the input of the neuron. (3)f(x)=1e−x

Although the sigmoid distinctive S-shape and capacity to condense real numbers into a range between 0 and 1 make it useful for binary classification, its propensity to saturate can hinder training by causing the vanishing gradient problem in deep neural networks.

Convolutional Neural Networks (CNNs) are extensively used in various fields, including natural language processing, image segmentation, image analysis, video analysis, and more. Several CNN variations have been developed, such as AlexNet [42], VGG (Visual Geometry Group) [43], Inception [44,45], ResNet (Residual Networks) [46,47], WideResNet [48], FractalNet [49], SqueezeNet [50], InceptionResNet [51], Xception (Extreme Inception) [52], MobileNet [53,54], DenseNet (Dense Convolutional Network) [55], SENet (Squeeze-and-Excitation Network) [56], Efficientnet [57,58] among others. These variants are applied in different application areas based on their learning capabilities and performance.

Recurrent Neural Networks (RNNs) are a class of deep learning models that possess internal memory, enabling them to capture sequential dependencies. Unlike traditional neural networks that treat inputs as independent entities, RNNs consider the temporal order of inputs, making them suitable for tasks involving sequential information [59]. By employing a loop, RNNs apply the same operation to each element in a series, with the current computation depending on both the current input and the previous computations [60].

The ability of RNNs to utilize contextual information is particularly valuable in tasks such as natural language processing, video classification, and speech recognition. For example, in language modeling, understanding the preceding words in a sentence is crucial for predicting the next word. RNNs excel at capturing such dependencies due to their recurrent nature [61–63].

However, a limitation of simple RNN is their short-term memory, which restricts their ability to retain information over long sequences [64]. To overcome this, more advanced RNN variants have been developed, including Long Short-Term Memory (LSTM) [65], bidirectional LSTM [66], Gated Recurrent Unit (GRU) [67], bidirectional GRU [68], Bayesian RNN [69], and others.

Fig. 9 depicts a simple recurrent neural network, where the internal memory (ht) is computed using Eq. (4) [70].

(4)ht=g (Wxt+Uht+b)

In this equation, g() represents the activation function (typically Tanh), U and W are adjustable weight matrices for the hidden state (h), b is the bias, and x denotes the input vector.

RNNs have proven to be powerful models for processing sequential data, leveraging their ability to capture dependencies over time. The various types of RNN models, such as LSTM, bidirectional LSTM, GRU, and bidirectional GRU, have been developed to address specific challenges in different applications.

Temporal Convolutional Networks (TCN) represent a significant advancement in neural network architectures, specifically tailored for handling sequential data, particularly time series. Originating as an extension of the one-dimensional Convolutional Neural Network (CNN), TCN was first introduced by [84] in 2017 for the task of action segmentation in video data, and its application was further generalized to other types of sequential data by [85] in 2018. TCN retains the powerful feature extraction capabilities inherent to CNN while being highly efficient in processing and analyzing time series data.

The purpose of training a TCN is to forecast the next l values of the input time series. Assume that we have a sequence of inputs x0,x1,…,xl. We would like to predict, at each time step, some corresponding output y0,y1,…,yl, whose values are equal to the inputs shifted forward l time steps. The primary limitation is that it can only use the inputs that have already been observed: x0,x1,…,xt, when forecasting the output yt for a given time step t [86]. TCN is characterized by two fundamental properties: (1) The convolutions within the network are causal, ensuring that the output at any given time step depends solely on the current and past inputs, without any influence from future inputs. (2) Similar to Recurrent Neural Networks (RNNs), TCN can process sequences of arbitrary length and produce output sequences of identical length. The three primary components of a typical TCN are residual connections, dilated convolution, and causal convolution [85,87,88]. Fig. 14 illustrates the schematic architecture of a TCN model.

Causal Convolution:

The TCN architecture is built upon two foundational principles. To adhere to the first principle, the initial layer of a TCN is a one-dimensional fully convolutional network, wherein each hidden layer maintains the same length as the input layer, achieved through zero-padding. This padding ensures that each successive layer remains the same length as the preceding one. To satisfy the second principle, TCN employs causal convolutions. A causal convolution is a specialized one-dimensional convolutional network where only elements from time t and earlier are convolved to produce the output at time t. Fig. 15 demonstrates the structure of a causal convolutional network.

Dilated Convolution:

TCN aims to effectively capture long-range dependencies in sequential data. A simple causal convolution can only consider a history that scales linearly with the depth of the network. This limitation would necessitate the use of large filters or an exceptionally deep network structure, which could hinder performance, particularly for tasks requiring a longer history.

The depth of the network could lead to issues such as vanishing gradients, ultimately degrading network performance or causing it to plateau. To address these challenges, TCN employs dilated convolutions [90], which exponentially expand the receptive field, allowing the network to process large time series efficiently without a proportional increase in computational complexity. The architecture of a dilated convolutional network is depicted in Fig. 16.

By inserting gaps between the weights of the convolutional kernel, dilated convolutions effectively increase the network’s receptive field while maintaining computational efficiency. The mathematical formulation of a dilated convolution is given by Eq. (11). (11)F(s)=(x∗fd)(s)=∑i=0k−1f(i)⋅xs−d⋅iwhere d is the dilation rate, k is the size of the filter, and s−d⋅i accounts for the direction of the past. Dilation is the same as adding a fixed step in between each pair of neighboring filter taps. When d=1, dilated convolution becomes a regular convolution. As d increases, the output at the higher layers reflects a broader range of inputs, improving performance on long-range dependencies in time series.

Residual Connections:

To construct a more expressive TCN model, it is essential to use small filter sizes and stack multiple layers. However, stacking dilated and causal convolutional layers increases the depth of the network, potentially leading to problems such as gradient decay or vanishing gradients during training. To mitigate these issues, TCN incorporates residual connections into the output layer. Residual connections facilitate the flow of data across layers by adding a shortcut path, allowing the network to learn residual functions, which are modifications to the identity mapping, rather than learning a full transformation. This approach has been shown to be highly effective in very deep networks.

A residual block [46] has a branch that lead to a set of transformations F, whose outputs are appended to block’s input x, as shown in Eq. (12). (12)o=Activation (x+F(x))

This method enables the network to focus on learning residual functions rather than the entire mapping. The TCN residual block typically consists of two layers of dilated causal convolutions followed by a non-linear activation function, such as Rectified Linear Unit (ReLU). The convolutional filters within the TCN are normalized using weight normalization [91], and dropout [92] is applied to each dilated convolution layer for regularization, where an entire channel is zeroed out at each training step. In contrast to a conventional ResNet, where the input is directly added to the output of the residual function, TCN adjusts for differing input-output widths by performing an additional 1×1 convolution to ensure that the element-wise addition ⊕ operates on tensors of matching dimensions.

Kolmogorov-Arnold Networks (KANs) represent a promising alternative to traditional Multi-Layer Perceptrons (MLPs) by leveraging the Kolmogorov-Arnold theorem, a sophisticated mathematical framework that enhances the capacity of neural networks to process complex data structures. KANs were first introduced in 2024 by [93], with the goal of incorporating advanced mathematical theories into deep learning architectures to improve their performance on intricate tasks. While MLPs are inspired by the universal approximation theorem, KANs are motivated by the Kolmogorov-Arnold representation theorem [94], which states that any multivariate continuous function f over a bounded domain can be expressed as a finite composition of simpler one-dimensional continuous functions: (13)f(x1, …, xn)=∑q=12n+1Φq(∑p=1nϕq,p (xp))where ϕq,p is a mapping [0, 1]→R and Φq is a mapping R→R.

KAN maintain a fully connected structure like MLP, but with a key distinction: while MLP assign fixed activation functions to nodes (neurons), KAN assign learnable activation functions to edges (weights). Consequently, KAN do not employ traditional linear weight matrices; instead, each weight parameter is replaced by a learnable one-dimensional function parameterized as a spline. Unlike MLP, which apply non-linear activation functions at each node, KAN nodes only sum the incoming data, relying on the rich, learnable spline functions to introduce non-linearity. Although this approach might initially seem computationally expensive, KAN often result in significantly smaller computation graphs compared to MLP. Fig. 17 illustrates the structure of a KAN.

The Kolmogorov-Arnold Network (KAN) can be expressed specifically as follows: (14)KAN(x)=(ΦL−1∘ΦL−2∘⋅⋅⋅∘Φ1∘Φ1)(x)

The transformation of each layer, Φl, operates on the input xl to generate xl+1, the input for the following layer, as follows: (15)xl+1=Φl(xl)=(ϕl,1,1(⋅)ϕl,1,2(⋅)ϕl,2,1(⋅)ϕl,2,2(⋅)…ϕl,1,nl(⋅)…ϕl,2,nl(⋅)⋮⋮ϕl,nl+1,1(⋅)ϕl,nl+1,2(⋅)⋱⋮…ϕl,nl+1,nl(⋅))xlwhere each activation function ∅l,j,i is a spline, offering a rich, flexible response surface to inputs from the model: (16)Spline(x)=∑iciBi(x),ci  are trainable coefficients

Several variants of KANs have emerged to tackle specific challenges in various applications: ➢

Convolutional KAN (CKAN) [95]: CKAN is a pioneering alternative to standard CNN, which have significantly advanced the field of computer vision. Convolutional KAN integrate the non-linear activation functions of KAN into the convolutional layers, leading to a substantial reduction in the number of parameters and offering a novel approach to optimizing neural network architectures.

The concept of an autoencoder originated as a neural network designed to reconstruct its input data. Its fundamental objective is to learn a meaningful representation of the data in an unsupervised manner, which can have various applications, including clustering [104].

An autoencoder is a neural network that aims to replicate its input at its output. It consists of an internal hidden layer that defines a code representing the input data. The autoencoder network is comprised of two main components: an encoder function, denoted as z=f(x), and a decoder function that generates a reconstruction, denoted as r=g(z) [108]. The function f(x) transforms a data point x from the data space to the feature space, while the function g(z) transforms z from the feature space back to the data space to reconstruct the original data point x. In modern autoencoders, these functions z=f(x) and r=g(z) are considered as stochastic functions, represented as pencoder(z|x) and pdencoder(r|z), respectively, where r denotes the reconstruction of x [109]. Fig. 18 illustrates an autoencoder model.

Autoencoder models find utility in various unsupervised learning tasks, such as generative modeling [110], dimensionality reduction [111], feature extraction [112], anomaly or outlier detection [113], and denoising [114].

In general, autoencoder models can be categorized into two major groups: Regularized Autoencoders, which are valuable for learning representations for subsequent classification tasks, and Variational Autoencoders [115], which can function as generative models. Examples of regularized autoencoder models include Sparse Autoencoder (SAE) [116], Contractive Autoencoder (CAE) [117], and Denoising Autoencoder (DAE) [118].

Variational Autoencoder (VAE) is a generative model that employs probabilistic distributions, such as the mean and variance of a Gaussian distribution, for data generation [104]. VAE provide a principled framework for learning deep latent-variable models and their associated inference models. The VAE consists of two coupled but independently parameterized models: the encoder or recognition model and the decoder or generative model. During “expectation maximization” learning iterations, the generative model receives an approximate posterior estimation of its latent random variables from the recognition model, which it uses to update its parameters. Conversely, the generative model acts as a scaffold for the recognition model, enabling it to learn meaningful representations of the data, such as potential class labels. In terms of Bayes’ rule, the recognition model is roughly the inverse of the generative model [119].

A notable neural network architecture for generative modeling, capable of producing realistic and novel samples on demand, is the Generative Adversarial Network (GAN), initially proposed by Goodfellow in 2014 [105]. A GAN consists of two key components: a generative model and a discriminative model. The generative model aims to generate data that resemble real ones, while the discriminative model aims to differentiate between real and synthetic data. Both models are typically implemented using multilayer perceptrons [120]. Fig. 19 depicts the framework of a GAN, where a two-player adversarial game is played between a generator (G) and a discriminator (D). The generator’s updating gradients are determined by the discriminator through an adaptive objective [121].

As previously mentioned, GANs operate based on principles derived from neural networks, utilizing a training set as input to generate new data that resembles the training set. In the case of GANs trained on image data, they can generate new images exhibiting human-like characteristics.

The following outlines the step-by-step operation of a GAN [122]:

The generator, created by a discriminative network, generates content based on the real data distribution.

Generative Adversarial Networks (GANs) have introduced numerous applications across various domains, including image blending [123], 3D object generation [124], face aging [125], medicine [126,127], steganography [128], image manipulation [129], text transfer [130], language and speech synthesis [131], traffic control [132], and video generation [133].

Furthermore, several models have been developed based on the Generative Adversarial Network (GAN) framework to address specific tasks. These models include Laplacian GAN (Lap-GAN) [134], Coupled GAN (Co-GAN) [120], Markovian GAN [135], Unrolled GAN [136], Wasserstein GAN (WGAN) [137], and Boundary Equilibrium GAN (BEGAN) [138], CycleGAN [139], DiscoGAN [140], Relativistic GAN [141], StyleGAN [142], Evolutionary GAN (E-GAN) [121], Bayesian Conditional GAN [143], Graph Embedding GAN (GE-GAN) [132].

The Deep Belief Network (DBN) is a type of deep generative model utilized primarily in unsupervised learning to uncover patterns within large datasets. Consisting of multiple layers of hidden units, DBNs are adept at identifying intricate patterns and extracting features from data. Unlike discriminative models, DBNs exhibit a higher resistance to overfitting, making them well-suited for feature extraction from unlabeled data [144].

The stack of Restricted Boltzmann Machines (RBMs), which operate in an unsupervised learning framework, is a fundamental part of DBN. Every RBM in a DBN is made up of a hidden layer that contains latent representations and a visible layer that represents observable data features [145]. RBMs are trained layer by layer: first, each RBM is trained independently, and then all of the RBMs are fine-tuned together as a whole within the DBN.

During the forward pass, the activations represent the probability of an output given a weighted input. In the backward pass, the activations estimate the probability of inputs given the weighted outputs. Through iterative training of RBMs within a DBN, these processes converge to form joint probability distributions of activations and inputs, allowing the network to effectively capture the underlying data structure [146,147]. Fig. 20 illustrates the schematic structure of a Deep Belief Network (DBN).

Deep learning models require large amounts of data to perform well. The performance of these models typically improves as the volume of data increases. However, in many cases, sufficient data may not be available, making it difficult to train deep learning models effectively [10].

Three possible approaches may be used to appropriately handle the insufficient data problem. The first method is Transfer Learning (TL), which is used to DL models by reusing pre-trained model pieces in new models. We thoroughly reviewed the transfer learning strategy in Section 7.

Data augmentation is the second method of gathering additional data. The goal of data augmentation is to improve the trained models’ capacity for generalization. Generalization is necessary for networks to overcome small datasets or datasets with unequal class distributions, and it is especially crucial for real-world data [259]. There are several strategies for augmenting data, and each one is contingent upon the characteristics of the datasets [260]. A few of these techniques are geometric transformations [261], Mixup augmentation [262], Random oversampling [263], Feature space augmentation [264], generative data augmentation [265], and many more.

The third approach considers using simulated data to increase the training set’s volume. If you have a good understanding of the physical process, you can sometimes build simulators from it. Consequently, the outcome will include simulating as much data as necessary [10,266].

In real-world situations, particularly in those that deep learning models address, the issue of class imbalance is common. If the majority of instances in the data set belong to one class and the remaining instances belong to the other class, then there is a class imbalance in a binary classification scenario. In multi-class, multi-label, multi-instance learning as well as in regression difficulties and other situations, class imbalances are present and are actually reinforced [267].

It has been determined that there are three main approaches to addressing imbalanced data: data-level techniques, algorithm-level techniques, and hybrid techniques. The focus of data-level techniques is to add or remove samples from training sets in order to balance the data distributions. These techniques balance the data distributions by adding new samples to the minority class (oversampling) or removing samples from the majority class (undersampling) [268,269]. A variety of oversampling techniques, including Synthetic Minority Over-sampling Technique (SMOTE) [270], Borderline-SMOTE [271], Adaptive Synthetic (ADASYN) [272], SVM (Support Vector Machine)-SMOTE [273], Majority Weighted Minority Oversampling Technique (MWMOTE) [274], Sampling With the Majority (SWIM) [275], Reverse-SMOTE (R-SMOTE) [276], Constrained Oversampling (CO) [277], SMOTE Based on Furthest Neighbor Algorithm (SOMTEFUNA) [278], and many more can be used to solve imbalanced data problems. Also, there are several techniques for undersampling, including EasyEnsemble [279], BalanceCascade [279], Inverse Random Undersampling [280], MLP-based Undersampling Technique (MLPUS) [281], and others.

Algorithm-level approaches modify existing learning algorithms to mitigate the bias towards the majority class. These techniques require specialized knowledge of both the application domain and the learning algorithm to diagnose why a classifier fails under imbalanced class distributions [268]. Two of the most commonly used methods in this context are Cost-Sensitive Learning [282,283] and One-Class Learning [284].

The third approach consists of hybrid methods, which combine algorithm-level techniques with data-level methods in the appropriate way. Hybridization is required to address issues with algorithm and data-level approaches and improve classification accuracy [285].

Overfitting occurs when a deep learning model learns the systematic and noise components of the training data to the point that it adversely affects the model’s performance on new data. In fact, overfitting occurs as a result of noise, the small size of the training set, and the complexity of the classifiers. Overfitted models tend to memorize all the data, including the inevitable noise in the training set, rather than understanding the underlying patterns in the data [24]. Overfitting is addressed with methods including dropout [92], weight decay [286], batch normalization [287,288], regularization [289], data augmentation, and others, although determining the ideal balance is still difficult.

In deep neural networks, the computation of gradients is propagated layer by layer, leading to a phenomenon known as the vanishing or exploding gradient problem. As gradients are backpropagated through the network, they can exponentially diminish or grow, respectively, causing significant issues in training. When gradients vanish, the weights of the network are adjusted so minimally that the model’s learning process becomes exceedingly slow, potentially stalling altogether. Conversely, exploding gradients can cause weights to be updated excessively, leading to instability and divergence during training. This problem is particularly pronounced with non-linear activation functions such as sigmoid and tanh, which compress the output into a narrow range, further exacerbating the issue by limiting the gradient’s magnitude. Consequently, the model struggles to learn effectively, especially in deep networks where gradients must pass through many layers [8].

To mitigate the vanishing and exploding gradient problem, several strategies have been developed. One effective approach is to use the Rectified Linear Unit (ReLU) activation function, which does not saturate and therefore helps to maintain the gradient flow throughout the network [290]. Proper weight initialization techniques, such as Xavier initialization [291] can also reduce the likelihood of gradient issues by ensuring that initial weights are set in a way that prevents gradients from becoming too small or too large [292]. Another solution is batch normalization, which normalizes the inputs of each layer to maintain a stable distribution of activations throughout training. By doing so, batch normalization helps to alleviate the vanishing gradient problem and can accelerate convergence by reducing internal covariate shifts. Overall, addressing the vanishing and exploding gradient problem is crucial for training deep neural networks effectively, enabling them to learn complex patterns without succumbing to instability or inefficiency [288].

Catastrophic forgetting is a critical challenge in the pursuit of artificial general intelligence within neural networks. It occurs when a model, after being trained on a new task, loses its ability to perform previously learned tasks. This phenomenon is particularly problematic in scenarios where a model is expected to learn sequentially across multiple tasks without forgetting earlier ones, such as in lifelong learning or continual learning applications. The root cause of catastrophic forgetting lies like neural networks, which update their weights based on new training data. When trained on a new task, the model adjusts its parameters to optimize performance on that task, often at the expense of previously acquired knowledge. As a result, the model may exhibit excellent performance on the most recent task but perform poorly on earlier ones, effectively “forgetting” them [293].

Several strategies have been proposed to address catastrophic forgetting. One such approach is Elastic Weight Consolidation (EWC) [294], which penalizes changes to the weights that are important for previous tasks, thereby preserving learned knowledge while allowing the model to adapt to new tasks. Incremental Moment Matching (IMM) [295] is another technique that merges models trained on different tasks into a single model, balancing the performance across all tasks. The iCaRL (incremental Classifier and Representation Learning) [296] method combines classification with representation learning, enabling the model to learn new classes without forgetting previously learned ones. Additionally, the Hard Attention to the Task (HAT) [293] approach employs task-specific masks that prevent interference between tasks, reducing the likelihood of forgetting.

Underspecification is an emerging challenge in the deployment of machine learning (ML) models, particularly deep learning (DL) models, in real-world applications. It refers to the phenomenon where an ML pipeline can produce a multitude of models that all perform well on the validation set but exhibit unpredictable behavior in deployment. This issue arises because the pipeline’s design does not fully specify which model characteristics are critical for generalization in real-world scenarios. The underspecification problem is often linked to the high degrees of freedom inherent in ML pipelines. Factors such as random seed initialization, hyperparameter selection, and the stochastic nature of training can lead to the creation of models with similar validation performance but divergent behaviors in production. These differences can manifest as inconsistent predictions when the model is exposed to new data or deployed in environments different from the training conditions [297].

Addressing underspecification requires rigorous testing and validation beyond standard metrics. Stress tests, as proposed by D’Amour et al. [297], are designed to evaluate a model’s robustness under various real-world conditions, identifying potential failure points that may not be apparent during standard validation. These tests simulate different deployment scenarios, such as varying input distributions or environmental changes, to assess how the model’s predictions might vary. Moreover, some researches have been conducted to analyze and mitigate underspecification across different ML tasks [298,299].

The IMDB dataset is a widely used dataset for sentiment analysis tasks. It consists of movie reviews along with their corresponding binary sentiment polarity labels. The dataset contains a total of 50,000 reviews, evenly split into 25,000 training samples and 25,000 testing samples. There is an equal distribution of positive and negative labels, with 25,000 instances of each sentiment. To reduce the correlation between reviews for a given movie, only 30 reviews are included in the dataset [300]. Positive reviews often contain words like “great,” “well,” and “love,” while negative reviews frequently use words like “bad” and “can’t.” However, certain words such as “one,” “character,” and “well” appear frequently in both positive and negative reviews, although their usage may differ in terms of frequency between the two sentiment classes [72].

In our analysis, we employed eight different deep learning models including CNN, RNN, LSTM, Bidirectional LSTM, GRU, Bidirectional GRU, TCN, and Transformer for sentiment classification using the IMDB dataset. Fig. 27 presents a structural overview of the deep learning model intended for analyzing the performance of eight different models on the IMDB dataset.

In this architecture, text data is first passed through an embedding layer, which transforms the high-dimensional, sparse input into dense, lower-dimensional vectors of real numbers. This allows the model to capture semantic relationships within the data. In the second layer, one of eight models: CNN, RNN, LSTM, Bi-LSTM, GRU, Bi-GRU, TCN, or Transformer is employed for feature extraction and data training. This layer is crucial for capturing patterns and dependencies in the data. Following this, a dropout layer is included to address the issue of overfitting by randomly deactivating a portion of the neurons during training, which helps improve the model’s generalization. Subsequently, the multi-dimensional vector turns into a one-dimensional vector using a flatten layer, enabling it to work with fully connected layers. Finally, the output is passed through a fully connected (Dense) layer, which uses a Softmax function for classification, converting the model’s predictions into probabilities for each class.

Building a neural network with high accuracy necessitates careful attention to hyperparameter selection, as these adjustments significantly influence the network’s performance. For example, setting the number of training iterations too high can lead to overfitting, where the model performs well on the training data but poorly on unseen data. Another critical hyperparameter is the learning rate, which affects the rate of convergence during training. If the learning rate is too high, the network may converge too quickly, potentially overshooting the global minimum of the loss function. Conversely, if the learning rate is too low, the convergence process may become excessively slow, prolonging training. Therefore, finding the optimal balance of hyperparameters is essential for maximizing the network’s performance and ensuring effective learning.

In the experiment phase, consistent parameters were applied across all models to ensure a standardized comparison. The parameters were set as follows: epochs = 30, batch size = 64, dropout = 0.2, with the loss function set to “Binary Crossentropy,” and the optimizer function set to Stochastic Gradient Descent (SGD) with a learning rate of 0.2. For the CNN model, 100 filters were used with a kernel size of 3, along with the Rectified Linear Unit (ReLU) activation function. The RNN, LSTM, Bi-LSTM, GRU, and Bi-GRU models each employed 64 units. The TCN model was configured with 16 filters, a kernel size of 5, and dilation rates of [1, 2, 4, 8]. The Transformer model was set up with 2 attention heads, a hidden layer size of 64 in the feed-forward network, and the ReLU activation function. These parameter settings and architectural choices were designed to allow for a standardized comparison of the deep learning models on the IMDB dataset. This standardization facilitates an accurate analysis of each model’s performance, enabling a comparison of their accuracy and loss values.

Table 2 shows the result of different deep learning models on IMDB review dataset based on various metrics including Accuracy, Precision, Recall, F1-Score, and Time of training.

To compare the performance of these models, we utilized accuracy, validation-accuracy, loss, and validation-loss diagrams. These diagrams provide insights into how well the models are learning from the data and help in evaluating their effectiveness for sentiment classification tasks.

Fig. 28 shows the accuracy and validation-accuracy diagrams where the accuracy, provides a visual representation of how the different deep learning models perform in terms of accuracy during the training process and validation-accuracy shows the trend of accuracy values on the testing set across multiple epochs for each model.

Fig. 29 illustrates the loss and validation-loss diagram where the loss diagram is a visual representation of loss values during the training process for six different models, and the validation-loss diagram depicts the variation in loss values on the testing set during the evaluation process for the different models. The loss function measures the discrepancy between the predicted sentiment labels and the actual labels.

Furthermore, the confusion matrices for the various deep learning models are displayed in Fig. 30. These matrices provide a detailed breakdown of each model’s performance, highlighting how well the models classify different classes. By closely examining these confusion matrices, we can gain insights into the precision of the models and identify patterns of misclassification for each class. This analysis helps in understanding the strengths and weaknesses of the models’ predictions.

Additionally, Fig. 31 displays the ROC-AUC (Receiver Operating Characteristic-Area Under Curve) diagrams for eight different deep learning models. These diagrams offer valuable insights into the classification performance of the models, aiding in the assessment of their effectiveness. By analyzing the ROC-AUC curves, we can make informed decisions regarding model selection and threshold adjustments, ensuring a more accurate and effective classification approach.

Based on the results provided, it can be concluded that the Transformer and Bi-GRU models achieved the best performance on the IMDB review dataset for sentiment analysis. Both models demonstrated high accuracy in classifying the sentiment of movie reviews. However, it is worth noting that the training time of the Transformer model was significantly less than that of the Bi-GRU model. This suggests that the Transformer model was faster to train compared to the Bi-GRU model while still achieving excellent performance. Additionally, the GRU model also exhibited good accuracy in sentiment classification and required less training time than the Bi-GRU model. Overall, the results suggest that the Transformer, and GRU models are effective deep learning models for sentiment analysis on the IMDB review dataset, with varying trade-offs between performance and training time.

Based on the provided information, the ARAS dataset [301] is a valuable resource for recognizing human activities in smart environments. It consists of data streams collected from two houses over a period of 60 days, with 20 binary sensors installed to monitor resident activity. The dataset includes information on 27 different activities performed by two residents, and the sensor events are recorded on a per-second basis.

Eight distinct deep learning models were used in our investigation to recognize human activities: CNN, RNN, LSTM, Bidirectional LSTM, GRU, Bidirectional GRU, TCN, and Transformer. A structural overview of the deep learning model designed to analyze the performance of eight different models on the ARAS dataset is shown in Fig. 32.

The first phase involves preprocessing the sensor data to ensure it is in a suitable and standardized format for deep learning models. The initial task in this phase is data cleaning, where any recorded instances where all sensor events are zero, and the resident is inside the house, are removed from the dataset. Next, a time-based static sliding window technique is applied for segmenting sensor events. This method groups sequences of sensor events into intervals of equal duration. Optimizing the time interval is crucial for effective segmentation; after evaluating intervals ranging from 30 to 360 s, a 90-s interval was determined to be optimal for the ARAS dataset. The segmentation task aids in decreasing training time and increasing accuracy for the deep learning models.

After preprocessing, the data is passed through an input layer. In the second layer, one of eight models: CNN, RNN, LSTM, Bi-LSTM, GRU, Bi-GRU, TCN, or Transformer is employed for feature extraction and training. This layer plays a vital role in capturing patterns and dependencies within the data. To mitigate overfitting, a dropout layer follows, which randomly deactivates a portion of the neurons during training, thereby improving the model’s generalization. Subsequently, a flatten layer is used to convert the multi-dimensional vector into a one-dimensional vector, making it compatible with fully connected layers. Finally, the output passes through a fully connected (Dense) layer, which uses a Softmax function for classification, transforming the model’s predictions into probability distributions across the classes.

In the experimental phase, we split the data from the first resident of house B, allocating 70% for training and 30% for testing, using a random split. Additionally, 20% of the training data was set aside for validation. The models were trained with a fixed set of parameters: 30 epochs, a batch size of 64, a dropout rate of 0.2, the “Categorical Crossentropy” loss function, and the Adam optimizer. For the CNN model, we used 100 filters with a kernel size of 3 and the rectified linear unit (ReLU) activation function. The RNN, LSTM, Bi-LSTM, GRU, and Bi-GRU models were configured with 64 units each. The TCN model was set with 16 filters, a kernel size of 5, and dilation rates of [1, 2, 4, 8]. The Transformer model utilized 2 attention heads, a hidden layer size of 64 in the feedforward network, and the ReLU activation function.

Table 3 illustrates the results of experiments on ARAS dataset with various metrices including Accuracy, Precision, Recall, F1-Score, and Time of training.

Also, Fig. 33 presents the accuracy diagram and validation-accuracy diagram for the deep learning models, while Fig. 34 shows the loss diagram and validation-loss diagram for deep learning models.

Since we performed preprocessing tasks like data cleaning and segmentation, the data is nearly normalized and balanced, leading to consistent and closely grouped results across all models. However, the results indicate that the Transformer and TCN models outperformed the others on the ARAS dataset. This outcome aligns with the dataset’s nature, which comprises spatial and temporal sequences of sensor events. Among the models, the Transformer exhibited the highest performance in terms of accuracy, recall, and F1-Score, while the Bi-LSTM model excelled in the precision metric. Moreover, the Transformer model demonstrated a notable advantage in training time, second only to the CNN model, underscoring its efficiency in processing and learning from time-series data. Additionally, when examining the accuracy and loss curves, it is evident that the Transformer, TCN, and CNN models stabilized earlier than the others. Overall, the Transformer model proved to be the most effective for working with the ARAS dataset, striking a balance between accuracy, training time, and consistency throughout the training phases, making it the optimal choice for recognizing human activities based on sensor data.

Since images are not sequential or time-dependent, recurrent models were less effective for these tasks. CNN-based models, on the other hand, are highly valuable for image analysis due to their ability to capture spatial relationships. Consequently, the analysis of deep learning models on the Fruit-360 dataset for image classification focused on eight CNN variants: VGG, Inception, ResNet, InceptionResNet, Xception, MobileNet, DenseNet, and NASNet. These models use deep transfer learning technique for training image data and improving classification accuracy. Fig. 35 provides a structural overview of the deep learning models used to evaluate the performance of these eight variants on the Fruit-360 dataset.

First, the fruit images are passed through an input layer. In the second layer, one of eight models (VGG, Inception, ResNet, InceptionResNet, Xception, MobileNet, DenseNet, or NASNet) is employed for feature extraction and training. Next, a Global Average Pooling 2D (GAP) layer is applied, which significantly reduces the spatial dimensions of the data by collapsing each feature map into a single value. To combat overfitting, a dropout layer is then introduced, randomly deactivating a portion of the neurons during training, which enhances the model’s ability to generalize. Finally, the output is passed through a fully connected (Dense) layer, where a Softmax function is used to classify the fruit images.

The dataset comprises 55,244 images of 81 different fruit classes, each with a resolution of 100 × 100 pixels. For the experiments, a subset of 60 fruit classes was selected, containing 28,484 images for training and 9558 images for testing. Non-fruit items such as chestnuts and ginger root were removed from the dataset.

All models were trained with a consistent set of parameters: 20 epochs, a batch size of 512, a dropout rate of 0.2, the “Categorical Crossentropy” loss function, and the Adam optimizer. Additionally, all models utilized the “ImageNet” dataset for pre-training.

Table 4 presents the experimental results for various models on the Fruit-360 dataset, including VGG16, InceptionV3, ResNet50, InceptionResNetV2, Xception, MobileNet, DenseNet121, and NASNetLarge. The table includes metrics such as Accuracy, Precision, Recall, F1-Score, and Time of training.

Furthermore, the accuracy, validation-accuracy, loss, and validation-loss diagrams were used to compare the performance of various models. When assessing the models’ performance for tasks involving the categorization of fruit photos, these graphs offer valuable insights into how effectively the models are learning from the data. Fig. 36 shows the accuracy and validation-accuracy diagram of the deep learning models, while Fig. 37 illustrates the loss diagram and validation-loss diagram of the deep learning models.

Based on the results, it can be concluded that the DenseNet and MobileNet models achieved the best performance for fruit image classification on the Fruit-360 dataset. Both models demonstrated high accuracy in classifying fruit images. Notably, MobileNet had a significantly shorter training time compared to DenseNet, indicating that it was faster to train while still delivering performance close to that of DenseNet. Additionally, the Xception model also showed good accuracy and required less training time than DenseNet. Overall, the MobileNet model stands out as a favorable choice due to its balance between accuracy and training efficiency.