Since ResNet [20] solved the difficult training of deep convolutional neural networks, convolutional neural networks have achieved important advances in feature extraction. In order to break down the feature map into a multi-scale pyramid representation, Lin et al. [21] suggested using the Adaptive Context module. Li et al. [22] proposed a cross-layer feature distribution module that adaptively generates weights between multi-layer features and fuses them according to the weights. Res2Net [23] takes split feature maps and improves multi-scale representation through multipath superimposed small kernel convolution, thereby achieving good results in multi-scale feature extraction and fusion. However, most of these techniques use a fixed 3 × 3 convolution kernel to extract features from images, which limits the effective receptive field. Consequently, when managing downstream jobs, the model’s performance degrades. The dominance of convolutional neural networks in computer vision is challenged by Vision Transformer.

More and more models based on the visual Transformer, such as Segformer [24] and Swin Transformer [7], have shown superior performance in downstream visual tasks compared to CNN due to their excellent overall architecture and multi-head self-attention mechanism. In contrast, multi-head self-attention is essentially dense pairwise processing, which leads to higher computational complexity. Currently, in the spatial domain, ConvNeXt [18] imitates the Transformer architecture and introduces large-kernel convolution, which has similar effects to Transformer architecture models of the same scale, but reduces training time. The subsequent VAN [15] network proposes to use the Large Kernel Attention (LKA) mechanism to establish channel and spatial attention, and decompose the large kernel convolution to achieve it at a lower computational cost. SegNeXt [19] uses convolutional attention to improve original features and introduces multi-size branch convolution. It is demonstrated through a series of experiments that convolutional attention performs better in region feature extraction and augmentation than the Transformer’s self-attention. Table 1 shows the pros. and cons. of several spatial information extraction models.

Inspired by the aforementioned models, we propose a multi-scale region-aware module. This module adopts the four-layer architecture of Transformer; however, we substitute self-attention with convolution-based multi-scale region-aware attention to reduce the computational load and improve the emotion region extraction. By employing large kernel convolutions, we can expand the sensory field and extract spatial information from a larger neighborhood range within the image. At the same time, stripe convolution [25] helps alleviate the model’s computational burden while perceiving the stripe regions. Consequently, the proposed method yields the most diverse shapes and comprehensive information about extracted emotion regions. Additionally, the attention weights are generated for each scale of emotion regions by pooling operations in the whole connected layer, which highlights important emotion regions and takes inspiration from humans’ selective attention mechanism. Lastly, the weights of these multi-scale emotion zones are summed.

With the in-depth study of Transformer and self-attention, VTN [26] applied sliding window to focus on temporal correlation in the time dimension and tried to introduce Transformer into the temporal series. Wang et al. [9] achieved good success in the video emotion identification community by combining TimeSformer for global dependency modeling between frames and using ResNet with graph convolution for spatial feature extraction. However, the extensive pairwise computation of self-attention without a priori knowledge of localization and translational isotropy still results in a larger computing cost. The introduction of the time dimension in Transformer also brings about the problem of feature similarity, that is, the self-attention probability matrix drives the input feature matrix to converge to rank 1 at a double exponential rate, resulting in an increase in sequence similarity and thus affecting the recognition performance of the model. Shi et al. [13] discovered that convolution and residual concatenation can help to solve this issue. Previously, TCN [27] attempted to handle the temporal series task by expanding the convolution with residual concatenation, in which features were extracted directly across time steps using convolution. PredCNN [28] combines a cascade multiplication unit (CMU) with CNN to capture the inter-frame dependency of the temporal series. At the moment, SimVP [29], which is totally based on CNN, extracts inter-frame dependency by concatenating large kernel convolutions in a bi-directional manner without adding any further tricks or intricate plans. Table 2 shows the pros. and cons. of several temporal information extraction models.

Motivated by the aforementioned method, we present a temporally interactive sample learning module. Layer by layer, the lengthy data are broken down using iterative odd-even down-sampling. To enable interaction between pairs of odd-even sub-series and further capture the temporal correlation information within them, each layer uses convolutional processes with element-wise multiplication. Moreover, the convolution is utilized as the interaction weights to enhance feature diversity and mitigate potential issues related to the feature similarity in self-attention. Additionally, each sub-series undergoes another round of convolution, enabling effective fusion of surrounding pixels and enhancing information fusion. The global temporal dependency emerges as the sampling-interaction develops. Global temporal dependency can be better modeled by the module thanks to the linear complexity of sampling and convolution with respect to the number of pixels, in contrast to the quadratic complexity of dense pairwise computing in Transformer.

The architecture of the proposed model is shown in Fig. 2, which mainly consists of a multi-scale region-aware module and a temporal interaction sampling learning module. To get the combined region characteristics for each frame, fixed-length series are dynamically sampled from the source video, down-sampled by the Stem layer, and supplied to the multi-scale region perception module. The structure of the multi-scale region-aware module is shown in Fig. 2. The module adopts the four-layer architecture of Transformer but uses convolution-based multi-scale region-aware attention to replace self-attention for region extraction. A region attention aggregation submodule and a branching convolution submodule make up multi-scale region-aware attention. Using convolution kernels of various sizes on each branch, the branching convolution submodule extracts emotion regions of various scales in each frame. Subsequently, the region attention aggregation submodule calculates the attention weights of the emotion regions at each scale and generates combined region features by aggregating regions according to the weights. We build the spatial portion of the entire model by stacking multi-scale region-aware modules using a multi-stage pooling method, which produces new temporal series. The temporal interaction sampling learning module then receives the new series and uses multi-layer odd-even down-sampling to progressively break down the lengthy series. Each layer uses convolution and elemental multiplication to achieve interaction between pairs of odd-even series and learn the temporal correlation information between the two series. Additionally, by using convolution as the interaction weight, the feature similarity issue is resolved, and feature diversity is increased. Each series is then subjected to a second convolution, which blends the surrounding pixels to improve the information fusion. The global timing-dependent properties of the deep series are concatenated.

The preprocessed video series is fed into the multi-scale region-aware module (MRA). This module’s multi-scale region-aware attention extracts emotion regions at various scales using deep convolution kernels of varying sizes. The model’s attention to important regions is then increased by applying attention weights to the feature maps of these regions. Fig. 3 depicts the branching convolution submodule (BCS) and the region attention aggregation submodule(RAAS) that make up the multi-scale region attention structure. Firstly, the branching convolution submodule extracts emotional regions of different scales in each frame. Subsequently, the region attention aggregation submodule computes the attention weights for each scale emotion region. Aggregating regions based on the weights yields the combined region features.

To evaluate the performance of our method on the video emotion recognition task, we choose four methods for comparison experiments.

ResNet+Bi-LSTM uses ResNet50 for spatial feature extraction, followed by Bi-LSTM to complete temporal feature extraction. PredRCNN [35] extracts and memorises spatial and temporal feature representations at the same time using ST-LSTM. ResNet3D [36] replaces each residual block internally with multiple 3D convolutional layers on top of ResNet while learning spatial relationships between video frames and temporal dynamics. ConvLSTM is added to ResNet3D via ResNet3D+ConvLSTM, enabling spatiotemporal attentional augmentation of previously extracted spatiotemporal characteristics. TimeSformer [8] uses the Transformer for spatiotemporal feature extraction. Dynamic Facial Expression Recognition Transformer (Former-DFER) [37] models temporal sequences’ global dependency by first extracting spatial features using a basic ResNet and then adding a two-stage TimeSformer. The model’s global time capacity allows it to produce good results. Deep Emotional Arousal Network (DEAN) [38] extracts temporal features using Bi-LSTM after using Transformer to aggregate relationships across various regions.

The input size is uniformly 224 × 224, and the sequence length is adapted according to the dataset; Among them, ResNet+Bi LSTM adopts ResNet50, combined with randomly initialized Bi LSTM layers, fine-tuned to convergence on the dataset of this study. ResNet3D (+ConvLSTM) uses ResNet3D-18 randomly initialized with ConvLSTM layers, and independently trains from scratch on three datasets. DEAN retains the original model’s multi-scale convolution+attention vision module, removes the audio branch, and trains from scratch; PredRNN references the basic metrics of the original paper (Kinetics-400 pre-training), while Former DFER removes the audio/text branches from the original multimodal and only retains the video ViT module. After fine-tuning the video branch parameters in this dataset, only video performance is obtained. The results of the comparative trials on emotion recognition are shown in Tables 2–4. The performance of multiple models is compared in Table 5 in terms of memory overhead and FLOPs per frame.

The data in Tables 4–6 show that ResNet+Bi-LSTM and PredRNN have already achieved good results in processing the video task by virtue of the spatio-temporal model structure. The subsequent ResNet3D+ ConvLSTM results demonstrate that another round of spatio-temporal enhancement of the extracted spatio-temporal features can be more effective in enriching the affective spatio-temporal features and enhancing the recognition accuracy. ResNet3D depends on the residual connections to stack the network deeper, thereby obtaining more temporal memories and achieving better results. After Transformer was added, the model as a whole performed better on large sample datasets like CMU-MOSEI and Hume-Reaction. This was due to the model utilizing Transformer’s global self-attention modeling capability and fully extracting correlation information within images and between video frames. Nevertheless, the limited improvement effect of Transformer-based models in smaller datasets is influenced by the lack of past knowledge. And such models have high memory overheads, high computational requirements, and long training times.

This paper combines the advantages of the above models and fully extracts and utilizes information from spatiotemporal and spatial dimensions through the deep stacking of the proposed modules. In the CMU-MOSEI and Hume Reaction datasets with sufficient samples, the Acc-7, Acc-2, F1, and Corr of our model have all been improved. This is because the model introduces multi-scale large kernel convolution and adaptive region attention to achieve multi-scale emotional region perception and fusion, and subsequently uses multi-layer sampling, convolution, and interaction in temporal feature extraction to establish global temporal dependency. However, in the CMU-MOSI dataset with insufficient data, due to the reliance on sufficient samples for multi-scale region-aware convolution and temporal interaction sampling to learn spatiotemporal feature distribution patterns, the Acc-7 metric has decreased compared to Transformer-based models. Meanwhile, analyzing the recognition failure cases, the model performs poorly on videos with extreme motion blur and severe occlusion.

Table 7 reflects the temporal and spatial overheads of different models. The model in this paper is based on sampling and convolution as a whole, compared to the square-level overhead brought about by dense pairwise computation in Transformer, the overheads of sampling and convolution are linear with the number of pixels. At the same time, convolution possesses the assumptions of a priori knowledge of locality, translation invariance, etc., so that the cost of the model is significantly reduced in both time and space, with Flops being reduced by nearly 2.5 times compared to Transformer-based models.

Fig. 6 shows the experimental results of the model on the three datasets of CMU-MOSI, CMU-MOSEI, and Hume-Reaction using bar charts in order to clearly depict the performance of the model provided in this chapter on the three datasets. Because CMU-MOSEI has a larger sample size, its indication identification is superior.

The proposed video emotion recognition model, which combines spatial multi-scale region-aware and temporally interactive sampling, is mainly consisted of multi-scale region-aware module and temporal interactive sampling learning module. We created ablation experiments that remove each module one at a time for testing and examine how each module affects overall performance in order to confirm the necessity of each module. Table 8 displays the ablation experiment findings,

In this work, we aim to eliminate the branching convolution sub-module for multi-scale emotional region extraction and utilize ResNet 50’s standard conv 1 through conv 4 exclusively for feature extraction. The model’s performance on CMU-MOSEI declines by 0.007, 0.006, and 0.013, according to the results. The above results show that using large kernel convolution of different sizes to extract multi-scale emotion regions and obtain varied emotion information can improve the accuracy of emotion recognition. The region attention aggregation submodule, which produces region attention weights, was then eliminated. This time, the spatial components of affective feature extraction and fusion are not used; only the temporal interaction sample learning module is used. At this moment, the CMU-MOSEI model’s accuracy was still declining by 0.009, 0.007, and 0.012. The additional decline in accuracy suggests that, once the multi-scale affective regions have been recovered, significant regions should be emphasized and highlighted throughout the fusion process.

Secondly, this paper tries to remove some functional parts of the temporal interaction sampling learning module to test their effects. After removing the weight multiplication and inter-sequence interaction fusion, the accuracy of the model on CMU-MOSEI shows a different degree of decrease, including a decrease of 0.01 in the accuracy of seven classes, which indicates that the interaction of temporal information achieves good performance in multi-class recognition tasks. The accuracy of the model in seven classes on CMU-MOSEI continues to drop by 0.12 with each further removal of the quadratic convolutional reinforcement module.

The entire ablation experiment shows that attention must be paid to both the temporal and spatial dimensions. The recognition outcomes are influenced by global temporal dependency in the temporal dimension and multi-scale region extraction and fusion in the spatial dimension. The need for interaction and fusion is particularly clear in activities involving multiple classes.