Contextual sentiment analysis requires the combination of semantic information and visual features in images for emotion classification, such as analyzing the interaction between characters and the environment. Common approaches include deep learning-based methods [25], CNNs [26], and attention mechanisms [27].

The attention mechanism is a crucial component in image classification. Spatial attention (SA) and channel attention (CA) [28] are two widely used attention mechanisms in deep learning. Their basic structures are illustrated in Fig. 2. In the CA module, input features F represent feature maps generated by convolutional operations. After undergoing Global Maximum Pooling (GMP) and Global Average Pooling (GAP), the results are fed into two shared-parameter MLP. Then outputs are multiplied and summed and then activated using the Sigmoid function to produce the final CA feature map, Mc. The SA module takes F′ as input, applies GAP and GMP along the channel dimension, concatenates the resulting 2D maps, and uses a convolutional layer followed by Sigmoid activation to produce the SA feature map, Ms.

The CBAM [29] module enhances CNNs by sequentially applying CA and SA, enabling the network to focus on “what” and “where” in feature maps. As shown in Fig. 3, the CBAM first models each channel through the CA module, and then models the spatial dimension of the feature map through the SA module. Finally, they are multiplied to obtain a final attention map. It is used to weight the original features, allowing the network to focus on important channels and spatial locations and extract more discriminative features.

Given an intermediate feature map F′=RC×H×W as input, CBAM sequentially infers a 1D channel attention Mc=RC×1×1 and a 2D spatial attention Ms=R1×H×W, the whole attention process can be summarized as: (1)F′=Mc(F)⊗FF′′=Ms(F′)⊗F′where ⊗ denotes element-by-element multiplication. C represents channels, H and W respectively represent the height and width of the feature map. During the multiplication, the attention values are copied accordingly: the channel attention values are broadcast along the spatial dimension and vice versa. F″ is the final refined output.

Face detection is the method of identifying and locating faces. It aims to establish the position and bounding box of faces in an image. Traditional approaches include machine learning techniques such as AdaBoost [30] and SVM [31], which rely on handcrafted features like Haar [32] and HOG [33]. More recent methods are based on deep learning, including Faster R-CNN [34], MTCNN [22].

Face feature extraction involves creating a discriminative feature vector representation from a face image. Commonly used face feature extraction methods include Principal Component Analysis (PCA) [35], which reduces image dimensionality while preserving key variations for identity or emotion recognition. Local Binary Patterns (LBP) [36] capture local texture by comparing pixel neighborhoods, while more discriminative representations of face features can also be learned through the use of CNNs [37], such as VGGFace [38] and EfficientNet [23].

When extracting emotional features from a scene, the entire image is fed into the channel. To capture crucial regions of the image, the CBAM attention mechanism is employed, which integrates channel and spatial attention mechanisms.

In this study, we choose ResNet18 [42] as the backbone network for both the body and context feature extraction channels. Compared with deeper models such as ResNet50 and ResNet101, ResNet18 achieves better performance while significantly reducing training time and computational resource usage. Its relatively simple architecture also makes it well-suited for integration with the CBAM attention mechanism.

Incorporating the CBAM attention mechanism on the network structure of ResNet18 can enhance the feature representation capability of the model. By adaptively learning the importance of channels and space, CBAM can provide a more accurate and robust feature representation, enabling the model to better capture the features that are most useful for the task. Fig. 7 shows the exact location of the CBAM module when it is integrated in a ResBlock. We apply CBAM to the convolutional output in each block.

The complete network structure of context feature extraction channel after adding CBAM is shown in Fig. 8, the input of the channel is the original image that contains both characters and scenes. The CBAM attention module is added in all four stages of ResNet18, and finally the features of the scene channel are obtained through the average pooling layer and the fully connected layer.

Since more than 25% of the images in the EMOTIC dataset fail to detect clear faces, an adaptive face channel is introduced to perform face detection and feature extraction directly from body frames. As shown in Fig. 9, MTCNN is used to detect faces within the body bounding boxes. If one or more faces are detected, the extracted facial frames are input into EfficientNet for feature extraction. The features from the scene, body, and face channels are then fused for subsequent classification. If no face is detected, the face channel is deactivated (weight set to 0), and classification proceeds using only the body and context features.

The complete model structure of adaptive multi-channel image sentiment analysis network with Focal Loss (AMSA) is shown in Fig. 10.

Both the body feature extraction module and the context feature extraction module use pre-trained ResNet18 models. Additionally, the context feature extraction part incorporates the CBAM attention mechanism to extract scene-relevant features. The face feature extraction part takes the body box as input. When a face is successfully detected by MTCNN, the corresponding facial region is extracted and processed by EfficientNet, and the features from all three channels (face, body, and context) are fused for classification. If no face is detected, the face channel is deactivated (the weight is set to zero), and the model switches to a dual-branch fusion mode, using only body and context features for emotion prediction. To ensure stable performance under both conditions, we adopt fixed fusion weights determined through validation set optimization. The fusion output y is computed as follows: (14)y=σ(0.4⋅fbody+0.6⋅fcontext)y=σ(0.5⋅fface+0.2⋅fbody+0.3⋅fcontext)

Finally, the fusion network module combines the features from the three feature extraction modules and carries out fine-grained sentiment representation regression using two fully connected layers. The output results in both discrete and continuous dimensions.

The EMOTIC dataset is a collection of images of people in unconstrained environments, where over 25% of people’s faces are partially occluded or have very low resolution. It contains 23,571 images and 34,320 annotated characters. The dataset uses a combination of queries containing keywords for different locations, social environments, diverse activities, and various emotional states, and it provides manually annotated body regions. The annotations comprise 3 continuous emotion dimensions: Valence, Arousal, and Dominance and 26 discrete emotions: anticipation, engagement, confidence, happiness, surprise, fatigue, embarrassment, anger and more. The detailed definitions of these categories can be found in [13,43], and Fig. 11 shows example images of these categories.

In our experiment, the dataset is split into three subsets: 70% for training, 20% for validation, and 10% for testing. The split ensures a balanced distribution of emotion categories across the subsets.

Images are annotated according to the VAD model [44], which represents emotions through a combination of three continuous dimensions, each of which is an integer value within the range of 1 to 10. The VAD model is defined as follows: valence (V) is used to measure the degree of positivity or pleasantness of an emotion, ranging from negative to positive; arousal (A) is used to measure an individual’s degree of arousal, ranging from inactive/calm to agitated/ready for action; and dominance (D) is used to measure a person’s degree of control over a situation, ranging from non-control to control. Fig. 12 illustrates the various values assigned to each dimension.

We utilize average precision (AP) to measure the performance of discrete category classification and the Jaccard coefficient (JC) [45] to test the similarity between two sets. Due to the classification of EMOTIC dataset belongs to multi-label classification, the final output may have some overlap with Ground Truth or vary completely. Therefore, we use JC to represents the ratio of the size of the intersection of A and B to the size of the concatenation of A and B. It is defined by Eq. (15), the higher the JC, the better the result. The maximum value of the JC is 1, where the detected category and the Ground Truth category are identical. (15)JC(A,B)=|A∩B||A∪B|=|A∩B||A|+|B|−|A∩B|

Performance of VAD in three continuous dimensions is measured through the MAE. A lower calculated value indicates a smaller error and better predictive performance of the model as defined in Eq. (16). (16)MAE=1m∑i=1m|yi−f(xi)|

The distribution of the 26 emotion categories in the EMOTIC dataset is presented in Fig. 13, with a descending order from left to right.

To evaluate the contribution of each module in our model, we conducted an ablation study involving various combinations of CBAM, the adaptive face channel, and Focal Loss in six experimental configurations. As shown in Table 1, the results demonstrate that each module contributes positively to the overall performance. (iii) and (iv) demonstrate the benefit of incorporating adaptive face recognition channel and confirm that the focal loss effectively enhanced performance on imbalanced data. And the full AMSA model achieves the highest accuracy, while configurations with only partial components show reduced performance.

To evaluate our model on a fine-grained dimension, we conducted classification experiments on 26 discrete emotion categories and compared our AMSA model with several representative methods, including those proposed by Zhang et al. [14], Lee et al. [15], Mittal et al. [16], Kosti et al. [43], Wang et al. [46], Li et al. [47] and Yang et al. [48].

As shown in Table 2, AMSA achieved the highest mean Average Precision (mAP), which was 2.53% higher than that of the best baseline paper [48]. This significant improvement demonstrates the effectiveness of the proposed adaptive multi-channel framework. Additionally, our model achieved the highest classification accuracy in 10 emotion categories, including Anger, Annoyance, and others. These gains can be attributed to the introduction of Focal Loss, which addresses the class imbalance issue in the EMOTIC dataset by giving more weight to underrepresented classes during training. This is especially important given the unbalanced distribution of emotion categories in the dataset. Moreover, we observed that in complex scene categories with subtle emotional cues, such as Annoyance and Sympathy, our model achieved 24.22% and 35.30%, respectively, significantly outperforming other methods. This confirms that our model does not overly rely on a single modality, but is capable of adapting to diverse visual cues present in complex scenes.

In addition to the aforementioned models, several studies have reported only the mAP without disclosing the classification accuracy for individual emotion categories. For completeness, we provide a comparative summary of these results in Table 3. de Lima Costa et al. [49] proposed a lightweight single-stream model focused on computational efficiency, but showed limited accuracy improvement. Etesam et al. [50] adopted a large language model for caption-based reasoning. Zhang et al. [51] proposed a training paradigm that extracts visual affective cues from communication using unedited data and topic-aware contextual encoding. In contrast, our AMSA reaches the highest mAP of 35.04%, wihch shows a balanced and accurate recognition result across both common and rare classes.

To evaluate our model in the continuous emotional dimension, we conducted regression analysis on the VAD space. As shown in Table 4, our model achieves a 10% lower average error rate compared to [43], and performs comparably to [14]. This result proves the effectiveness of using Mean Squared Error (MSE) loss in our regression setting. Compared to both approaches, AMSA adopts a more comprehensive image analysis framework by incorporating adaptive face detection and multi-channel feature fusion. Unlike [43], which uses SL1 Loss, our method leverages MSE Loss to better optimize continuous predictions. While Zhang et al. [14] also adopt MSE, AMSA achieves similar overall performance in average error but exhibits more balanced results across the three VAD dimensions.

To provide a more intuitive result for our experiment, we displays three representative samples from the EMOTIC dataset in Table 5, including two successful cases and one challenging case with facial occlusion. The classification predictions are denoted in black for agreeing with those in Ground Truth, and in red for misclassification. We evaluated the results’ efficacy using JC coefficients and MAE. Our model correctly detects the majority of ground-truth categories in the first two examples. Compared to [43], the JC coefficients are improved by 0.15 and 0.30, respectively, while the MAE values are reduced by 1.23 and 1.57. In the third case, the face was obscured and model relied solely on body and context features for judgment, so it is not very accurate in terms of emotions, resulting in misjudgments such as fear and pain. Although its performance has declined, the gap compared to [43] remains small. This indicates the validity of the AMSA model presented in this paper.

This example also shows the importance of our adaptive strategy. When the face is blocked, the model automatically switches to rely more on body and context features. Although this fallback approach cannot fully replace facial information, it still enables the model make reasonable predictions by using other visual cues in the image.