This section describes the training framework used in the proposed Adaptive Knowledge Distillation (AKD) model. The system consists of a pretrained wav2vec 2.0 teacher network and a lightweight student network. During training, the student network learns from both the ground-truth emotion labels and the soft targets produced by the teacher model. The overall training objective combines the standard cross-entropy loss with the adaptive distillation loss. During training, the teacher model remains fixed while the student model parameters are optimized using the combined loss function. For each input segment, the teacher produces soft targets that guide the student network. The student model is trained using the Adam optimizer until convergence.

Let x denote an input speech segment and y is corresponding emotion label. The teacher network produces logits zt, while the student network produces logits zs. The probability distributions obtained after the softmax operation are denoted by pt and ps, respectively. The temperature parameter T controls the softness of the probability distributions used during distillation. The standard cross-entropy loss is formulated as (1)LCE=−∑i=1Cyilog⁡(ps,i),where C denotes the number of emotion classes. This loss measures the discrepancy between the student model’s predicted emotion distribution and the ground-truth labels, and the knowledge distillation loss is defined as (2)LKD=KL(ptT∥psT),where KL(⋅) denotes the Kullback–Leibler divergence between the softened teacher and student probability distributions. This loss enables the student model to learn informative knowledge from the teacher model. Finally, the model’s overall training objective is expressed as (3)L=αLCE+βLKD,where α and β are balancing coefficients that control the relative contributions of the classification loss and the distillation loss. The detailed descriptions of each component are provided in the subsequent sections.

The authors introduced a relatively new technique for transferring knowledge according to the Kullback-Leibler divergence theory in [8]. This divergence measures results from differences between two probability distributions over the same variable and is minimized for the teacher-student model. This technique has been tested on speech and image recognition tasks, which have proven effective. This concept incorporates the distillation of knowledge kd into the probability equation, along with the classification probability q=[q1,q2,…,qi,…,qc]∈R1×c. Here, qi represents the probability of the i-th class, and c indicates the total number of classes with logit of the i-th class in this concept, which can be calculated as:(4)qi=exp⁡(zi/kd)∑j=1cexp⁡(zj/kd)where zi represents the logit and distilled knowledge shown by kd, which is set to 1, and the label is classified one at a time. The disadvantage of this approach is that it makes neural network training too rigid, resulting in a loss of information about incorrect classes. As kd increases beyond 1, classes with a probability of 0 acquire a modest probability. However, logit distillation is limited despite this strategy. Furthermore, the teacher network is based on a pretrained wav2vec 2.0 model trained on large-scale speech corpora such as LibriSpeech. None of the evaluation datasets used in this study (IEMOCAP, EmoDB, and RAVDESS) is included in the pretraining data, ensuring that the reported results are not influenced by data overlap between pretraining and evaluation datasets.

Teacher: The method employs a pre-trained wav2vec-2.0 (Large) [15] model to encode the audio waveform using the feature encoder, capturing the low-level embedding features of the input waveform. The resulting embedding is normalized and activated with the Gelu function before being fed into the context network, comprising 12 transformer blocks with 12 attention heads each. Afterward, the soft label and weight for emotion classification prediction are determined from the feature vector obtained from the last layer of the context network.

Student: Our proposed student is based on the hybrid transformer [21], which incorporates skip connections between encoders and decoders. In the case of smaller datasets, the authors observed that employing knowledge adoption and distillation yields improved results. Following this insight, the authors constructed a student model (shown in Fig. 1) using an encoder-decoder architecture with a residual learning strategy, followed by convolutional layers with skip connections across layers [22]. In our method, the Transformer model takes the lead in analyzing speech and is encoded with a self-attention mechanism that analyzes these features, focusing on essential parts of the speech and considering long-range dependencies. This improves understanding compared to traditional methods. Finally, the system decodes the encoded features, enabling the model to consider all parts of the speech signal simultaneously and potentially improving its ability to capture complex relationships within the data, which is crucial for extracting discriminative features. The outputs undergo post-processing using connected components and employ spatial and temporal learning to target emotion effectively, capturing a comprehensive learning pattern from a micro perspective.

Limitations: The traditional transformer approach uses arbitrary convolutions for volumetric input data. However, these convolutions can only capture short-range spatial-temporal features, limiting their capacity to model broader global contextual dependencies beyond the designated receptive field. The spatial and temporal channels of the Transformers encode long-range dependencies by comparing feature activations throughout space and time. This mechanism transcends the limitations of conventional filters’ receptive fields. However, combining self-attention with convolutional layers proves advantageous for various tasks [23]. However, the authors are unaware of prior attempts to design spatio-temporal self-attention exclusively as an essential component for SER, as described in the literature.

To capture the inherent qualities of teacher networks, the authors developed Adopter (as shown in Fig. 1) as a latent representation of teacher networks. This approach draws inspiration from latent factor models frequently employed in recommendation systems (as discussed in [24]). A latent factor represents their inherent characteristics in the model. Our approach extracts instance representations from the final layer of the student network’s output. This results in a value of the input, where the number of channels, height, and width of the student’s feature map ensure consistent alignment of input representations; the authors employ an essential operation to select the most significant value within each channel, as demonstrated below:(5)di=Conv(bi,s)

The method has a set of values represented by the variable di, where each value is a vector in Rc. Similarly, bi∈Rc×h×w is a result for the i-th input, where c, h, and w represent channels, height, and width of the feature map. For simplicity, set di (a vector) equal to c (the number of channels), compute the weight for the i-th input attributed to the teacher model, and normalize it using the loss function to ensure fairness. (6)wt;i=loss(ct;i)=exp⁡(ct;i)∑t′=1mexp⁡(ct′;i)

Eq. (6) defines a loss function for a model’s predictions, particularly useful in scenarios with multiple possible outputs at each step. Here, wt;i represents the loss assigned to a specific element (i) at a particular time (t). The model first calculates a score (ct;i) for this element. The higher the score, the more likely the model is to believe this element is the correct output. An exponential term (exp⁡(ct;i)) emphasizes this preference. To create a proper probability distribution, the exponentials of all scores across all elements and timesteps are then summed. Finally, dividing the element’s own exponential by this total sum gives us the loss (wt;i). This loss is likely used during training to adjust the model and improve its ability to identify the most probable output at each step. Furthermore, the weighted addition operation to obtain the integrated soft-target y~Ti, contrasting with traditional learning methods, is calculated as:(7)y~Ti=∑t=1mwt;i⋅y~Tt;i (8)LKD=∑i=1N(H(yi,yS,i)+k⋅DKL(y~Ti,y~Si)) (9)D(xi,xj,xk)=cos−1⁡(xi−xjxi−xj⋅xi−xkxi−xk)

The variable y~Tt;i represents the soft-target generated by the t-th teacher for the i-th input. The system guided the students through two methods: (i) loss of distillation of standard knowledge, incorporating the teacher model’s knowledge according to Eq. (8). This equation involves two main components: y~Si and y~Ti (student and teacher), respectively. The former is the soft-target of the i-th input produced by the student network, while the latter is the integrated soft-target computed using Eq. (7). Furthermore, facilitate the transfer of relational information across different datasets through structural knowledge, implemented as Eq. (9), inspired by [25]. Furthermore, represent the loss for the student-teacher scenario by obtaining the integrated soft targets. This approach will ensure accurate and efficient learning, leading to better outcomes, which are computed as follows:(10)L=∑i,j,kldD(y~Ti,y~Tj,y~Tk)⋅D(y~Si,y~Sj,y~Sk) (11)LKD=∑t=m1∑l=f(g)kut−st(vl)2

Our Adaptive Knowledge Distillation (AKD) framework prioritizes robust learning, particularly when the student model learns from the teacher’s outputs. In order to prevent a student regression from being negatively impacted by the noise from their teacher’s predictions, the authors replace the original regression loss function (MSE, or Mean Squared Error) with the Huber loss (denoted as ld). The reason is that MSE is not strong to outliers in the same way that regression problems with MAE are. Huber loss is MSE for small errors and MAE for large errors, improving the student regression’s robustness.

In the core AKD process, the students are trained using the “soft targets”, i.e., the probabilities of the teacher model. The soft targets y~Ti, y~Tj, and y~Tk are computed using the three inputs or the inputs in the complex example (for all three inputs). The soft targets provide the students with the opportunity to exploit the additional knowledge learned by the teacher model. The computation of these targets, as well as their use in the distillation loss ld is defined in Eq. (10).

Furthermore, as shown in Eq. (11), our knowledge transfer mechanism helps to close the divide not only at the final output. As ut is the last rich feature map of a teacher model and vl is the l-th layer of a student model, Eq. (11) is probably used in a simple comparison process (a projection and a distance/similarity computation, for example) or a dynamic weighting based on how much do ut and vl agree with each other. This mechanism is then used to dynamically weigh the loss term of the main distillation term (ld in Eq. (10)). This is based on the idea that if the student’s intermediate characteristics vl are already close to the teacher’s characteristics ut, the distillation weight at the output level (ld) could be relaxed to allow student characteristics across the architecture to be more expressive.

A critical challenge in KD is “negative transfer,” in which a student model inadvertently learns incorrect patterns from a faulty teacher. To mitigate this, introduce a mechanism to selectively allow knowledge transfer based on the teacher’s reliability for a given instance. This is governed by Eqs. (12) and (13):(12)γ=1−ρteacherc

Here, ρteacherc represents a measure of the teacher’s confidence or correctness for a specific class c or, more generally, its prediction quality on a given input instance (often determined by comparing the teacher’s prediction to the ground truth). Consequently, γ quantifies the teacher’s error or uncertainty; a low γ indicates a reliable teacher prediction for that instance. (13)β={γ,if γ≤θ1,otherwise

The variable β acts as a modulating factor, determined by γ and a predefined threshold θ. This threshold θ defines an acceptable level of teacher error.

If the teacher’s error γ is within an acceptable range (i.e., γ≤θ), then β is set to γ. This means the influence of certain loss components (like Lccc in Eq. (14)) will be scaled by the teacher’s error.

Crucially, as stated in the original context: “If the value of γ is greater than θ, i.e., the teacher prediction is ‘wrong’ beyond the threshold, Lcos is set to zero for that sequence.” This is a key part of the prevention of negative transfer. It implies that a specific component of the distillation loss (here, Lcos, probably a cosine similarity-based loss encouraging alignment between student and teacher outputs/features) is entirely disregarded if the teacher is deemed too unreliable for that particular sample.

After incorporating this negative transfer mitigation module, the overall joint loss function, which guides the student’s training, can be expressed as (revising/clarifying the role of the original Eq. (10) context):(14)Ljoint=α⋅Lcos′+β⋅Lcccwhere:

Lcos′ is the cosine similarity based distillation loss, which is effectively Lcos if γ≤θ, and 0 if γ>θ. This term encourages the student to mimic the teacher’s output representation when the teacher is reliable.

Clarification on β’s role based on typical KD practice: Usually, if the teacher is reliable (γ≤θ), one would want to increase the student’s learning from the teacher. If β weights Lccc (which could be the student’s direct task loss or a KD loss related to the teacher), the current definition of β is a bit unusual and needs careful consideration based on its intended effect:

If γ≤θ (teacher good), then β=γ (teacher error). So, Lccc is scaled by teacher error. This would reduce its impact if the teacher is very good (low γ).

This might be intended if Lccc is the student’s own task loss, and when the teacher is unreliable (and Lcos′ is zeroed out), the student should focus more on their own task loss. If Lccc is also a distillation loss, this setup for β would require careful justification in the main text.

In this manner, the student model selectively learns from the teacher. It primarily imitates the teacher’s outputs (via Lcos′) only when the teacher’s predictions are deemed accurate (i.e., γ≤θ). If the teacher’s predictions are significantly “wrong” (i.e., γ>θ), this specific distillation path is shut off, preventing the student from internalizing erroneous knowledge. This method implicitly incorporates ground-truth information into the KD loss landscape by using ρteacherc (derived from comparing teacher output to ground truth) to gate or modulate the learning process.

This article uses the IEMOCAP [26], EmoDB [27], and RAVDESS [28] corpora to ensure the proposed method’s robustness and efficiency for SER. The IEMOCAP corpus is recorded in American English by 10 professional speakers, covering four emotions. Similarly, EmoDB is a German-language recorded corpus by ten German speakers that covers seven emotions, and RAVDESS is a recorded corpus in British English by twenty-four speakers across twelve sessions, covering eight emotions. These are scripted corpora in which male and female actors deliver pre-designed scripts while portraying various emotions. More explanations and details are available in [26–28].

To assess the predictive capacity of our proposed model, utilize two metrics: weighted accuracy (WA) and unweighted accuracy (UA). UA represents the mean accuracy across emotional categories, while WA gauges the accuracy across all samples. These metrics are widely utilized in contemporary SER research to assess performance.

We adopted a true nested cross-validation protocol for evaluation. In the outer loop, the data were divided into Kouter speaker-grouped folds. In each outer iteration, one fold was held out as the test set (approximately 20% of the data), while the remaining folds (approximately 80% of the data) formed the outer-training partition. The split was performed at the utterance level before segmentation, so that all segments derived from the same utterance remained in a single partition. In the inner loop, the outer-training partition was further divided into Kinner speaker-grouped folds for hyperparameter selection. For each candidate hyperparameter configuration, the model was trained on the inner-training folds and validated on the inner-validation fold, and the average validation performance across inner folds was used for model selection. After selecting the best configuration, the model was retrained on the full outer-training partition and evaluated once on the corresponding outer-test partition. Final performance was reported by averaging the results across all outer folds, which is shown in Algorithm 1. To create this outer loop, we performed a division at the utterance level, before actually segmenting the speech as shown in Table 1. This way, all segments of a given utterance ended in either the training or testing set. If the segmentation was performed before splitting into the partitions, leakage between the two can occur. To ensure speaker-independent evaluation, speaker IDs were used as grouping constraints in both the outer and inner cross-validation loops. The split was applied at the utterance level so that speakers present in the training set were not included in the testing set for their own partition. Moreover, each speaker was assigned to only one outer fold, yielding disjoint speaker sets for training and testing. As shown in Table 2, grouped speaker-based folds were used for IEMOCAP, EmoDB, and RAVDESS. IEMOCAP and EmoDB used an 8/2 train-test split with a batch size of 64 and a learning rate of approximately 19–20/4–5, with train-test speaker splits per fold. Inner cross-validation was performed only on the outer training speakers, using the same grouping rule, thereby preventing speaker leakage during hyperparameter tuning. The fold assignments in Table 2 illustrate the exact speaker or session partitions used in the experiments. The Log-Mel spectrum with 40 Mel filters was used as an input feature, producing a sequence of 40-dimensional feature vectors for each time frame. The resulting feature vectors are then fused to form a final 128-dimensional feature vector. This approach is less computationally intensive than the more commonly used Mel-frequency cepstral coefficients (MFCCs), and has improved feature correlation.

The proposed model is compared with state-of-the-art techniques using the same dataset and evaluation metrics, as shown in Table 3. The output highlights the robustness of our method with innovative architecture in the SER domain. Our model outperformed the recent SER model and especially beat [39], knowledge distillation, and [38] multilayer attention mechanism with distillation for speech recognition, which achieved the best results recently. However, our model has a reasonable recognition rate and outperforms the recent baseline methods. These findings underscore the uniqueness and broad applicability of the features acquired through our proposed encoder-based adaptive knowledge distillation architecture. Furthermore, our model effectively captures salient information in emotion recognition tasks, as demonstrated by the confusion matrices in Figs. 2–4, which provide intuitive visualizations of its performance across all evaluated datasets.

The study set out to create an AKD model tailored for resource-constrained devices, such as those on the edge. To make it lean and efficient without sacrificing accuracy, carefully applied several techniques: pruning away unnecessary parts, simplifying calculations through quantization, and using distillation to learn from larger models. This method used the ONNX standard to neatly package the model’s parameters and weights for smooth, real-time deployment, with the results of this optimization detailed in Table 5. Beyond that, it further boosted its speed by adapting it to lower-precision numbers (like fp16 floating-point), ensuring performance wasn’t compromised. Ultimately, aimed to build a quick model that doesn’t drain much power and still makes excellent predictions.

Our model can occasionally become perplexed and incorrectly label similar or closely related emotions, such as confusing Frustration with Anger or Happiness with Excitement. Furthermore, when dealing with highly imbalanced data, our model tends to misclassify emotions as the one with the most data samples.