Text sentiment analysis is one of the earliest and most mature research directions in affective computing and natural language processing. Early studies mainly relied on sentiment lexicons and traditional machine learning methods, whereas the recent rise of deep learning models has significantly advanced the performance of this field.

First, lexicon-based methods [13] primarily rely on precompiled sentiment lexicons to determine overall sentiment by identifying sentiment-bearing words and their intensities in text. For example, Alwan et al. [14] proposed a hybrid model based on rough set theory and sentiment lexicons for polarity analysis of Arabic political articles. To address low classification accuracy on Arabic political texts, they constructed a dedicated corpus of 206 annotated documents and performed classification via three core steps—preprocessing, feature extraction, and hybrid-model construction—achieving an accuracy of 85.483%. In a comment-classification task, Hamouda et al. employed the SentiWordNet lexicon [15], which assigns positive, neutral, and negative sentiment scores to lexical items [16]. Although lexicon-based methods offer interpretability, require no training data, and are grounded in manual rules, they struggle to capture contextual shifts, sarcasm or metaphor, and domain-specific expressions. Traditional machine-learning approaches commonly use algorithms such as k-nearest neighbors (k-NN), support vector machines (SVM) [17], and naive Bayes (NB) [18]. Goel et al. [19] presented a real-time Twitter sentiment-analysis system that combines naive Bayes with SentiWordNet, improving classification accuracy by incorporating SentiWordNet’s scoring scheme. Rathor et al. [20] compared SVM, NB, and maximum entropy (ME) for sentiment analysis of Amazon product reviews and found SVM achieved the highest accuracy. More recently, the rise of deep learning has substantially advanced text sentiment analysis. Kim [21] applied convolutional neural networks (CNNs) to sentence-level sentiment classification with strong results. Zhou et al. [22] proposed a bilingual LSTM model with an attention mechanism for cross-lingual sentiment classification. Wang et al. [23] introduced a tree-structured region-dividing CNN–LSTM hybrid model for dimensional sentiment analysis; by partitioning text into multiple semantic regions, extracting local features via CNNs, and capturing long-range cross-region dependencies with LSTMs, the model produces continuous-valued predictions of sentiment intensity.

Image sentiment analysis aims to identify and interpret the emotions conveyed in images by analyzing their visual content. Similar to text sentiment analysis, image sentiment analysis has also evolved from traditional approaches to deep learning–based methods.

Early approaches typically relied on hand-crafted visual features, such as low-level cues including color, texture, shape, and illumination, combined with machine learning classifiers for sentiment prediction. For example, Siersdorfer et al. [24] investigated the relationship between the affective content of images in social media and their metadata and visual content. By analyzing more than 586,000 images from Flickr, they explored the feasibility of predicting image sentiment polarity by combining visual features (e.g., color distributions and SIFT descriptors) with textual sentiment labels extracted using the SentiWordNet lexicon. In [25], a large-scale Visual Sentiment Ontology (VSO) containing more than 3000 adjective–noun pairs (ANPs) was constructed, and a SentiBank library with 1200 ANP detectors was developed on this basis, which significantly improved the accuracy of sentiment prediction for visual content by detecting affective concepts in images. Yang et al. [26] proposed a graph-based sentiment analysis method that jointly models user-posted images and comments from friends to infer users’ affective states. However, these methods often struggle to capture complex high-level semantic and affective information in images. With the advent of deep learning, image sentiment analysis has achieved substantial progress. Song et al. [27] proposed a visual-attention-based image sentiment analysis method (SentiNet-A), which integrates convolutional neural networks (CNNs) with a visual attention mechanism to improve image sentiment classification performance. Islam and Zhang [28] adopted transfer learning by initializing the network with parameters from a pretrained GoogLeNet [29] and, combined with data augmentation, achieved an accuracy of 86.1% on a Twitter image dataset.

Although previous studies have demonstrated the effectiveness of unimodal approaches for emotion recognition, comparative analyses in the literature consistently show that multimodal strategies achieve superior performance [30].

MSA seeks to infer sentiment by integrating heterogeneous modalities—such as vision, text, and audio—to provide a more comprehensive understanding of human affect. The recent surge in multimodal content has driven renewed interest in MSA. You et al. [31] proposed a Cross-modal Consistency Regression (CCR) model that uses CNNs to extract visual features and a paragraph-vector model to obtain textual features; consistency across modalities is then enforced via Kullback–Leibler (KL) divergence to enable joint optimization. Zhao et al. [32] introduced an image–text consistency–driven approach that first uses an SVM classifier to assess whether image and text semantics align, and then combines intermediate visual features from SentiBank [25] with textual and social features for adaptive MSA. To strengthen inter-modal interaction, several works employ attention mechanisms. Zhang et al. [33] proposed an attention-based model with a symmetric architecture that processes text and image data in parallel: a denoising autoencoder extracts robust textual features, while an improved variational autoencoder with attention extracts salient visual features. They further design a cross-feature fusion module that uses attention to learn complementary information between text and image features rather than relying on simple concatenation or one-way influence. Zhou et al. [34] introduced CAHFW-Net, a method for video-based emotion recognition that combines cross-attention with a hybrid feature-weighted neural network. Its hierarchical attention encoding network—particularly the cross-attention module—effectively captures complementary cues between facial expressions and scene context, mitigating emotion confusion and misinterpretation. Khan and Fu [35] developed a dual-stream model based on input-space translation (EF-CapTrBERT): an object-aware Transformer first translates images into natural-language descriptions, which are then formed into auxiliary sentences to inject multimodal information into BERT. This approach addresses image noise and modality fusion without modifying BERT’s internal architecture. With the rapid progress of vision–language models (VLMs) [36], their strong multimodal capabilities have seen wide adoption. For example, Huang et al. [37] proposed CLIP-MSA, a CLIP-based MSA model that introduces CLIP’s cross-modal encoder as the second branch of a dual-branch feature extractor. By leveraging CLIP’s pretrained semantic alignment and external knowledge, CLIP-MSA substantially improves the quality of unimodal representations.

In summary, although existing methods have achieved notable progress on MSA, they still exhibit clear limitations in hierarchical modeling of visual–textual features and in the fine-grained characterization of dynamic inter-modal interactions. Most current approaches follow a single-path feature extraction paradigm and perform multimodal fusion via simple concatenation or summation. Such coarse-grained modeling is insufficient to fully explore discriminative intra-modal representations from multiple perspectives, and it fails to effectively capture the complementarity and alignment between images and text in a fine-grained semantic space, which often leads to suboptimal sentiment recognition in complex real-world scenarios. Recent CLIP-based methods (e.g., FAMEAC [10], SentiCLIP [38]) and large vision-language models have made important advances; however, they have not yet simultaneously achieved multi-scale hierarchical representation, bidirectional fine-grained cross-modal alignment, and adaptive dynamic fusion in a unified framework. To bridge these remaining gaps, the proposed MB-CMCAG network performs multi-path collaborative modeling to learn hierarchical visual–textual representations at both coarse and fine granularities, employs bidirectional cross-modal cross-attention for deep semantic interaction, and introduces a novel cross-modal interactive gating mechanism that adaptively selects and reweights cross-modal information, thereby effectively suppressing redundant noise and highlighting key cues critical for sentiment discrimination.

Given a multimodal sentiment analysis dataset 𝒟={(Ii,Si,yi)}i=1N, each sample consists of an image Ii, an original text Si, and a sentiment label yi∈{positive,neutral,negative}. To better exploit the semantic and emotional cues embedded in the visual modality, an image caption generation model g(⋅) is employed to generate a descriptive caption Ci=g(Ii) for each image. The generated caption is concatenated with the original text to form the final textual input Ti=Concat(Si,Ci). The goal of this task is to learn a multimodal sentiment classification mapping function f:(I,T)↦y based on the image Ii and the augmented text Ti. By incorporating image-generated captions, the model can jointly learn complementary emotional representations from both visual and linguistic modalities, thereby achieving more accurate multimodal sentiment understanding.

The proposed MB-CMCAG model consists of five main components: an image caption generation module, a multi-branch feature-extraction layer, a cross-modal cross-attention interaction layer, a cross-modal cross-gating fusion layer, and a sentiment classification layer. As illustrated in Fig. 1, the image captioning module converts an image into a textual description, which is then concatenated with the original text to form the final composite text. The multi-branch feature-extraction module leverages different pretrained models to obtain multi-granularity feature representations from both images and text. The cross-modal cross-attention interaction module enhances information exchange between modalities through bidirectional attention, i.e., image-to-text and text-to-image. The cross-modal cross-gating fusion module adaptively controls the fusion strength of features from different modalities and integrates contextual information. Finally, the fused multimodal representation is fed into a multi-layer classifier for sentiment prediction.

To evaluate the performance of the proposed MB-CMCAG model, this study utilizes two widely adopted Multi-View Sentiment Analysis (MVSA) datasets: MVSA-Single and MVSA-Multiple [42]. Both datasets are sourced from Twitter and consist of image-text pairs, each annotated with sentiment labels: positive, neutral, and negative. The MVSA-Single dataset contains 5129 image-text pairs, each labeled by a single annotator. After preprocessing to remove samples with inconsistent sentiment labels across modalities, 4511 valid samples remain, including 2683 positive samples, 470 neutral samples, and 1358 negative samples. The MVSA-Multiple dataset comprises 19,600 image-text pairs, labeled by three annotators. The final sentiment label is determined through majority voting. After preprocessing, 17,024 valid samples are obtained, including 11,318 positive samples, 4408 neutral samples, and 1298 negative samples. To facilitate comparisons with other models, both datasets are randomly split into training, validation, and test sets in an 8:1:1 ratio. Table 1 provides detailed information on the dataset distribution.

All experiments were conducted on a server equipped with an NVIDIA RTX 4090 GPU (24 GB VRAM). The model was implemented in PyTorch. All experimental results are averaged over three runs with different random seeds. For feature extraction of the original text and image captions, we used a pretrained BERT tokenizer with a maximum sequence length of 128; sequences longer than this were truncated and shorter ones were padded. For visual feature extraction, we employed an ImageNet-pretrained ViT-B/16 model with a patch size of 16×16. For the additional branch, we used the CLIP-ViT-B/32 pretrained model to extract complementary semantic features. Models were trained with the Adam optimizer using a learning rate of 5×10−5 and a weight decay of 1×10−4. Owing to differences in dataset sizes, the batch size was set to 32 for MVSA-Single and 128 for MVSA-Multiple. Training proceeded for up to 30 epochs with early stopping based on validation performance and a patience of 5 epochs. Detailed hyperparameter settings are reported in Table 2.

In terms of evaluation metrics, we employ Accuracy and Macro-F1 as the primary measures, which are standard metrics for multi-class sentiment analysis tasks. Accuracy measures the overall proportion of correctly classified samples, while Macro-F1 provides a balanced evaluation of class-wise precision and recall, making it more suitable for datasets with class imbalance. The calculation formulas are as follows:(21)Accuracy=1N∑i=1NI(y^i=yi) (22)Precisionc=TPcTPc+FPc (23)Recallc=TPcTPc+FNc (24)F1c=2×Precisionc×RecallcPrecisionc+Recallc (25)Macro-F1=1|𝒞|∑c∈𝒞F1cwhere N denotes the total number of test samples, y^i and yi denote the predicted label and the ground-truth label of the i-th sample, respectively, I(⋅) is the indicator function, and 𝒞 denotes the set of sentiment classes. For each class c, TPc, FPc, and FNc are computed in a one-vs.-rest manner.

To demonstrate the effectiveness of the proposed MB-CMCAG model, we conduct comparative experiments against representative baseline methods from the literature. The performance differences are quantitatively evaluated from both unimodal and multimodal perspectives using two core metrics, namely Accuracy and Macro-F1. The comparative results are reported in Table 3. The baseline methods included in the comparison are described as follows:

BERT and BiLSTM [22]: BERT and BiLSTM represent classical methods for modeling textual modalities, with extensive applications in text classification tasks.

ViT and ResNet-50 [40]: ResNet-50 and ViT represent the most prevalent benchmark models in image classification.

MultiSentiNet [43]: This model introduces visual semantic features (objects and scenes) as auxiliary information and employs a visually guided attention LSTM to mitigate semantic loss and weak image-text correlation caused by naive feature fusion.

Co-Memory [44]: This work addresses insufficient image-text interaction in multimodal sentiment analysis by proposing a co-memory network, where text guides visual feature selection and images attend to key words, with stacked layers for iterative refinement.

DMAF [45]: This model uses unimodal attention to highlight sentiment regions and words, and integrates intermediate and late fusion for robust multimodal sentiment prediction.

MVAN [46]: This paper proposes a Multi-View Attention Network that jointly models object-level and scene-level views of images and employs an interactive learning mechanism to improve multimodal sentiment analysis performance.

CLMLF [47]: This method employs a Transformer-based multi-layer fusion module to align and fuse textual and visual features at the token level, and incorporates label- and data-driven contrastive learning to enhance the learning of sentiment-discriminative shared representations.

CLIP-CA-CG [48]: This author builds a CLIP-based cross-modal sentiment model that extracts visual and textual features with ResNet50 and RoBERTa, enhances cross-modal interaction via multi-head attention, and fuses multi-level features using a cross-modal gating module.

SentiCLIP [38]: This multimodal sentiment analysis model leverages the CLIP encoder to construct a unified semantic space and employs a feature interaction module to enhance cross-modal emotion understanding.

ITIN [49]: The paper proposes an image-text interaction network that enhances multimodal sentiment analysis accuracy through fine-grained alignment of image regions and text words, along with adaptive feature fusion.

VILA [50]: The paper significantly improves visual language model performance and surpasses LLaVA-1.5 by unfreezing the LLM, adopting interleaved data, and using mixed text instruction fine-tuning.

LLaVAC [51]: The paper designs a structured prompt incorporating independent image and text sentiment labels as well as a joint multimodal label, and performs single-round LoRA fine-tuning on LLaVA to directly use it as a multimodal sentiment classifier.

The results in Table 3 show that MB-CMCAG achieves statistically significant improvements over the strongest baselines on the corresponding metrics, further confirming the robustness of the proposed method. Although MB-CMCAG achieves the best accuracy on MVSA-Multiple, its F1-score is slightly lower than that of some comparison methods. This is mainly because MVSA-Multiple is a more challenging dataset, where image-text semantic inconsistencies, ambiguous sentiment expressions, and class imbalance are more pronounced. Since F1-score is more sensitive to minority-class recognition than Accuracy, a small fluctuation in difficult categories may lead to a lower F1 even when overall accuracy remains strong. Therefore, this result does not contradict the effectiveness of MB-CMCAG, but rather reflects the difficulty of the dataset and the remaining room for improvement in minority-class discrimination.

Furthermore, to more intuitively illustrate the discriminative capability of the MB-CMCAG model across different categories, we plot the corresponding confusion matrices on the MVSA-Single and MVSA-Multiple datasets, respectively. The confusion matrices indicate that the model achieves higher classification accuracy on positive and negative samples, whereas its performance on the neutral category is relatively weaker, as shown in Fig. 4. This observation is consistent with the slightly lower F1-score on MVSA-Multiple, since errors in minority classes have a larger impact on Macro-F1 than on overall Accuracy. Future work may explore targeted data augmentation and cost-sensitive learning to further improve neutral-class recognition and alleviate class imbalance.

To systematically evaluate the contribution of each core component in the proposed MB-CMCAG, we conducted a seven-group ablation study on both the MVSA-Single and MVSA-Multiple datasets. In each variant, one key module is removed or replaced while all other components remain unchanged. The seven settings are: (1) Text Only, (2) Image Only, (3) MB-CMCAG without CLIP (w/o CLIP), (4) MB-CMCAG without BERT-ViT (w/o BV), (5) MB-CMCAG without Cross-Modal Cross-Attention (w/o CMCA), (6) MB-CMCAG without Gated Cross-Attention (w/o GCA), and (7) MB-CMCAG without Caption Generation Module (w/o CGM). Performance is evaluated using Accuracy and Macro-F1. The detailed experimental settings and complete results are reported in Table 4.

The ablation results clearly show that every module contributes positively to the final performance. Removing any single component leads to a consistent decrease in both Accuracy and Macro-F1 on the two datasets, confirming the necessity and complementarity of the proposed design. Specifically, the unimodal variants (Text Only and Image Only) suffer the largest performance drops, with average accuracy decreases of 4.73% and 5.82%, respectively, highlighting the importance of multimodal fusion. Removing the CLIP branch (w/o CLIP) or the fine-grained BV branch (w/o BV) reduces average accuracy by 2.59% and 3.59%, respectively, indicating that coarse-grained global alignment and fine-grained local modeling are both essential. Eliminating the cross-modal cross-attention module (w/o CMCA) causes a 3.52% drop, showing that bidirectional fine-grained interaction is critical for semantic alignment. Replacing the gated mechanism with simple concatenation (w/o GCA) leads to a 1.43% decline, demonstrating the effectiveness of adaptive noise suppression and feature re-weighting. Finally, removing the image caption generation module (w/o CGM) results in a 0.70% decrease, confirming that visual-to-textual semantic enrichment provides useful complementary information. Overall, these results verify that the performance gains of MB-CMCAG arise from the coordinated interaction of its modules rather than from any single component alone, which further supports the effectiveness of the proposed hierarchical design.

To further evaluate the computational efficiency of the proposed method, we report the model size, computational cost in terms of Floating Point Operations per Second (FLOPs), inference time, and GPU memory usage in Table 5. All measurements are conducted on the same server under the same settings as Section 4.2, with a batch size of 1 to ensure fair inference time comparison.

Although MB-CMCAG integrates three pre-trained backbones, its total parameter count (347M) and inference time (45.3 ms) remain significantly lower than recent large vision-language models such as LLaVAC and VILA (both over 7B parameters). The cross-modal gated mechanism effectively suppresses redundant computation, resulting in a practical trade-off between performance and efficiency. Compared with these large-scale VLMs, our model achieves competitive accuracy while requiring only about 5% of the parameters and less than 20% of the inference time, demonstrating its suitability for real-world deployment on standard GPUs.

To further illustrate the effectiveness of MB-CMCAG in multimodal text-image classification tasks, we selected four representative cases from the MVSA-Single test set. These examples are used to qualitatively analyze the model’s ability to capture subtle emotional cues in text-image pairs. As shown in Fig. 5, we compare our proposed model with CLIP-CA-CG and SentiCLIP on these four cases. In addition, we input image captions as auxiliary sentences to further examine their effect on sentiment prediction.

First, all three models correctly classify the first and second examples as positive and negative, respectively, since the image-text pairs in these cases convey clear and consistent emotional signals. In the third example, where both modalities express no obvious sentiment tendency, CLIP-CA-CG predicts negative and SentiCLIP predicts positive, whereas our model correctly identifies the sample as neutral. This result suggests that the proposed multi-branch feature extraction strategy can better preserve complementary semantic information from both modalities, thereby improving the recognition of neutral samples with ambiguous emotional cues. In the fourth example, both CLIP-CA-CG and SentiCLIP classify the sample as positive, mainly because the image itself does not explicitly express a negative tendency while the text suggests positive sentiment. In contrast, our model correctly predicts negative, which indicates that the image captioning module can provide additional visual semantic cues and help the model handle image-text inconsistency more effectively.

In summary, these representative cases show that MB-CMCAG can better capture complementary sentiment cues from images and texts, especially in challenging samples with weak or inconsistent modality alignment. The results also indicate that image captions as auxiliary textual descriptions can further enhance cross-modal understanding and improve sentiment classification in complex scenarios.