In the proposed framework, we adopt RoBERTa [32] as the textual encoder and Faster R-CNN [33] as the visual backbone. This choice is motivated by following the requirements of multimodal aspect-based sentiment analysis. The textual inputs in the applied datasets (e.g., Twitter-15 and Twitter-17 datasets) are typically short, noisy, and lack grammatical regularity, which demands a backbone capable of stable representation learning under limited data conditions.

On the textual side, RoBERTa [32], a robustly optimized variant of BERT [4], has demonstrated strong contextual modeling and reliable fine-tuning in low-resource sentiment tasks. By contrast, Decoding-enhanced BERT with Disentangled Attention (DeBERTa) [34] achieves gains on large-scale benchmarks but usually requires abundant data to exploit its disentangled attention fully. When applied to small and noisy corpora such as Twitter, DeBERTa [34] often suffers from unstable convergence and overfitting, whereas RoBERTa [32] provides proven robustness.

On the visual side, sentiment cues are often localized in specific regions, such as faces, trophies, or symbolic objects, rather than spread across the entire image. Faster R-CNN [33] is particularly suitable for this scenario because its region proposal mechanism yields object-level features that align naturally with aspect–image correspondence. In contrast, patch-based encoders such as the Vision Transformer (ViT) [35] or holistic vision–language models such as Contrastive Language–Image Pre-training (CLIP) [36] and Bootstrapping Language–Image Pre-training (BLIP) [37] primarily produce global embeddings that dilute fine-grained sentiment cues, thereby limiting their effectiveness in aspect-level alignment.

Another key consideration is efficiency and compatibility. Pre-extracted region proposals from Faster R-CNN [33] enable efficient training on a single GPU, avoiding the latency and memory overhead of large vision–language frameworks. More importantly, the discrete region-level features extracted by Faster R-CNN [33] and the stable token representations produced by RoBERTa [32] provide complementary granularity that aligns naturally with our Gated-Linear Aspect-Aware Fusion (GLAF) and alignment objectives, both of which require fine-grained and structurally consistent inputs to achieve effective cross-modal integration. In contrast, global embeddings generated by models such as ViT [35], CLIP [36], or BLIP [37] are less amenable to alignment-based supervision, while DeBERTa’s [34] instability on small and noisy corpora undermines downstream fusion. Therefore, the combination of RoBERTa [32] and Faster R-CNN [33] provides both efficiency and natural compatibility with GLAF and alignment-driven optimization, making it a well-suited backbone for our framework.

We utilize a pretrained RoBERTa language model [32] to encode the input sentence in an aspect-aware manner. For the original input sentence S and an aspect term a_m ∈ A, a modified sequence S^′ is constructed by concatenating the aspect term with a masked version of the sentence, separated by a special delimiter: (1)S′=Concat([am],[SEP],Smasked)where S_masked is derived by replacing the aspect span a_m in S with a placeholder token. The token [SEP] is a special separator symbol used in RoBERTa to mark boundaries between different segments. The function Concat (·) denotes the token-level sequence concatenation (not embedding concatenation), and the square brackets [·] indicate token sequences rather than scalar values or mathematical sets.

The resulting sequence S^′ is then encoded to produce contextualized token embeddings: (2)HT=RoBERTa(S′)∈Rh×lwhere H_T serves as the aspect-aware textual embedding for subsequent cross-modal reasoning modules, h is the hidden dimension, and l is the input length. All visual and textual feature matrices are organized in a column-wise format, and each column represents a single token or region embedding vector of dimension ℎ.

To extract region-level visual features relevant to the aspect, a pretrained object detection model named Faster R-CNN [33] is used to process the input image V. The detector identifies the top K regions with the highest confidence scores, Subsequently, the detected object proposals are ranked based on their category-wise confidence scores, and the top 100 proposals are retained to preserve finer-grained objects, thereby facilitating precise alignment with aspect targets. Let R, R ∈ R^2048×100 denotes the regional representations, we have: (3)R=FasterR−CNN(V)

In order to capture the inter-object dependencies, the regional feature matrix R is projected and subsequently input into a Transformer encoder, producing object-level visual representations: (4)HV=Transformer(WR⊤R)where W_R ∈ R^2048×h and H_V ∈ R^h×100.

Let us start with the construction of visual representations that are guided by the aspect-aware textual semantics. We adopt a Cross-Modal Transformer layer [39] to model the interaction between the aspect and the image, which regards image representation H_V as queries, and contextualized aspect representations H_T as keys and values as follows: (5)HV′=CrossModal-Transformer(Q=HV,K=HT,V=HT)where HV′ ∈ R^h×100 is the generated aspect-based image representation. This cross-modal attention allows each image region to query the aspect-aware textual features and retrieve semantically aligned context, thereby generating visual representations that are conditioned on the corresponding aspect term.

To assess whether the image is semantically relevant to the aspect, we apply max-pooling to the visual representation HV′ to extract the most salient features for relevance prediction. This downsampling operation selects the maximum activation across spatial dimensions for each feature channel, highlighting informative regions while suppressing noise: (6)hv=MaxPool(HV′)∈Rh

The aggregated vector h_v is subsequently fed into a binary classifier to compute the relevance score P (r) ∈ (0, 1), which represents the probability that the image is semantically aligned with the current aspect: (7)P(r)=σ(Wrhv+br)where W_r ∈ R^1×h and b_r ∈ R are learnable parameters of the classifier, and σ (·) denotes the sigmoid activation function.

A high relevance score indicates that the visual content provides meaningful context for the given aspect, while a low score suggests semantic misalignment. This scalar value P (r) serves as the global relevance signal and would be utilized in subsequent gating to suppress noisy or unrelated visual information during fusion.

While cross-modal fusion enables rich semantic interactions, irrelevant or noisy visual information may still introduce confusion if indiscriminately combined with textual representations. we leverage the scalar relevance score P (r) ∈ (0, 1) to suppress uninformative or misleading visual content. The score is then broadcast and expanded to match the shape of the visual representation HV′, producing a gating matrix G ∈ R^h×100, and each entry in G equals to P (r). The relevance-guided visual features are computed via element-wise multiplication: (8)HV′′=G⊙HV′=P(r)⋅HV′where ⊙ denotes the Hadamard (element-wise) product.

When P (r) is approaximate to 0 (indicating low image-aspect relevance), the visual signal is effectively suppressed. Conversely, when P (r) approaches to 1, the model retains full visual representation. This gated modulation acts as a soft attention filter, allowing the model to downweight noisy or irrelevant visual cues while preserving informative regions for downstream fusion.

To estimate semantic relevance between an image and a given aspect, we follow the coarse-grained region-aspect alignment [38], using manually annotated relevance tags to supervise visual–textual correspondence. The predicted relevance score P (r) is optimized with binary cross-entropy loss: (9)ℒRE=−1M∑k=1[Mlog⁡P(rk)]where M denotes the number of training samples with relevance annotations; k denote the k-th training instance. A higher penalty is imposed when the model predicts high relevance scores for semantically unrelated image–aspect pairs, or low scores for relevant pairs. This loss supervision guides the model to distinguish informative visual signals from noisy or irrelevant content.

While the visual modality provides abundant contextual signals for multimodal sentiment analysis, the sentiment expressed toward a given aspect is usually grounded in specific regions of the image rather than the entire visual scene. To be specific, for an individual mentioned in the text, the associated image may contain unrelated elements such as background objects, surrounding people, or environmental noise.

In this part, we employ a region alignment supervision module [38] to compute attention over localized image regions and optimize it toward a ground-truth alignment distribution based on manually annotated aspect-relevant regions. Through this supervision, the model is encouraged to focus on semantically relevant regions while suppressing noisy or unrelated visual content, thereby enhancing aspect-aware representation learning.

Cross-Modal Transformer layer [39] is applied to obtain the aspect-aware attention distribution over 100 object proposals from Faster R-CNN. We use the representation of the first token in the aspect input HT0 as queries, and the filtered image representations HV′′ as keys and values: (10)HT′=CrossModal-Transformer(Q=HT0,K=HV′′,V=HV′′)where HT′ ∈ R^h×1 is the generated image-based aspect representation. This aspect-aware visual representation HT′ serves as both the region-level visual cue for multimodal fusion and a supervision target for alignment training. It enables the model to focus on sentiment-relevant regions in a fine-grained and interpretable manner.

We denote the attention weights from the i-th head of the CrossModal Transformer as D_i. To obtain the final attention distribution across the K candidate regions, we compute the mean of all m attention heads, resulting in D=1m∑i=1mDi, D ∈ R^K.

To supervise the attention mechanism in learning semantically meaningful region-aspect alignments, we leverage the ground-truth (GT) bounding boxes that is manually annotated by human experts [40,41]. These GT regions serve as soft alignment references, enabling the model to calibrate its attention distribution toward sentiment-relevant visual areas. For each training instance, we construct the alignment distribution based on the degree of spatial overlap between the detected candidate regions and the annotated GT box. We construct the supervised alignment distribution α∈K by computing the Intersection over Union (IoU) between each of the K = 100 candidate regions—extracted from image V using Faster R-CNN—and the ground-truth (GT) bounding box B_GT. For the i-th region B_i, we calculate its IoU score with B_GT, denoted as s_i = IoU(B_i, B_GT). If s_i exceeds a predefined threshold τ = 0.5, it is retained; otherwise, it is set to zero. The generated alignment distribution α serves as a soft supervision signal to guide the model’s region-level attention mechanism toward aspect-relevant visual regions.

To strengthen the alignment between the model-generated attention distribution D and the human-annotated alignment distribution α, we adopt the Kullback–Leibler (KL) divergence as the training objective. The region-level alignment loss is formally defined as: (11)ℒATT=1N∑i=1Nαilog⁡(αiDi)where N is the number of aspect–image samples for region-level alignment training, we use manually annotated supervision with global image–aspect relevance labels and fine-grained bounding boxes to guide attention learning. These annotations help the model distinguish sentiment-relevant regions from noise, providing structured guidance for cross-modal alignment.

However, while the alignment module improves aspect–region correspondence, it lacks a fine-grained fusion of textual and image-derived aspect features. To address this, we introduce the Gated-Linear Aspect-Aware Fusion (GLAF) module, which adaptively integrates multimodal cues based on aspect relevance.

To dynamically regulate the contribution of different textual features under aspect guidance, we develop a gating-based adaptive fusion strategy. Specifically, given the aspect-aware textual representation H_T and the generated image-based aspect representation HT′, we first compute the fusion gating matrix G as: (12)G=σ(W1HT′+W2HT+b)where W₁, W₂ ∈ R^h×h are learnable transformation matrices; b ∈ R^h is the bias term; and σ (·) denotes the element-wise Sigmoid activation function.

The gating coefficient G ∈ (0, 1) indicates the contribution degree of the visual modality to the fusion representation. This gating formulation is a widely adopted linear gating strategy commonly used in multimodal representation learning [16]. Based on the gating matrix G, the fused representation is computed as: (13)HF=G⊙HT′+(1−G)⊙HT,HF∈Rh×l

This fusion mechanism allows the model to automatically adjust the relative contributions of individual textual tokens according to their aspect relevance, effectively mitigating the impact of noisy or irrelevant textual signals.

Although the gating-based fusion mechanism adaptively regulates token contributions based on aspect relevance, it does not explicitly model fine-grained semantic dependencies across the fused sequence. To address this, we propose a lightweight semantic interaction module based on multi-head linear attention, building upon ReLU-based linear attention [42] with structural simplification and reduced computational overhead. Specifically, we remove the softmax in self-attention and replace it with ReLU-activated queries and keys for more efficient, stable computation. This design reduces numerical instability and strengthens cross-modal semantic interaction without overfitting to dominant features, while a residual connection preserves original fused features for fine-grained refinement guided by learned attention weights.

Let H_F denote the fused representation obtained from the previous stage, we project H_F into the query, key, and value spaces as follows: (14)Q=ReLU(WQHF),K=ReLU(WKHF),V=WVHFwhere W_Q, W_K, W_V ∈ R^h×h are learnable projection matrices.

The ReLU activation introduces non-linearity and enhances the representation capacity. These projected matrices are then fed into a linear attention layer to compute contextualized embeddings, which are subsequently used for sentiment classification.

Next, the attention weights are computed using a linear normalization strategy. Specifically, we define the unnormalized attention matrix as QK^T. The normalized attention weights are calculated as: (15)Attn=QiKjT∑j=1lQiKjT+εwhere Q_i ∈ R^h and K_j ∈ R^h denote the i-th query and j-th key vector, respectively, and ε is a small constant (e.g., 10⁻⁶) added to prevent numerical instability.

The attention-weighted representation is computed as: (16)HF′=Wo(Attn⋅V+HF)where HF′ ∈ R^h×l denotes the final enhanced fused representation incorporating contextual information via semantic interaction and W_o is a learnable projection matrix used as a linear transformation function. The residual connection with H_F helps preserve the original modality-aware features while injecting interaction-aware refinements.

This semantic interaction mechanism enables to model fine-grained global dependencies across tokens within the fused representation HF′, thereby enhancing the precision and contextual richness of the aspect-level semantic encoding.

The fused representation is fed into a sentiment classification layer. For aspect-level prediction, we apply mean pooling over token-level representations, a common method in sentence representation learning [4,32] for effectively aggregating embeddings. (17)HA′=MeanPooling(HF′),HA′∈Rh

The resulting vector HA′ is then fed into a linear classifier to predict the sentiment polarity distribution: (18)P(y|X)=Softmax(WAHA′+b),WA∈R3×h,b∈R3where P (y│X) ∈ R³ denotes the probabilities for three sentiment classes: positive, neutral, and negative.

To incorporate external sentiment supervision at the aspect level, we reformulate the original dataset into an aspect-specific instance structure. While each original sample is defined as a triplet X = (S, V, A). Specifically, we query the LLM to infer a sentiment polarity label y^GPT ∈ {0, 1, 2} for each training sample based solely on its textual input. These predicted labels are then mapped into a continuous vector space via a learnable embedding function Emb: ℤ → R^h, yielding sentiment-guided embedding vectors: e_g = Emb (y^GPT), and e_g ∈ R^h represents the knowledge-guided sentiment prior for the g-th sample. The embedding matrix is jointly optimized with the rest of the network parameters during training, allowing the model to learn how to incorporate external sentiment knowledge effectively.

To obtain high-quality sentiment supervision signals, we leverage the reasoning capabilities of a pretrained large language model (GPT-4o) to generate auxiliary sentiment labels for each training instance. For a given sample X = (S, V, A), a carefully designed prompt is constructed and fed into GPT-4o to infer a sentiment polarity label: y^GPT ∈ {negative, neutral, positive}.

Inspired by the prompt-based causal reasoning design in Multimodal Dual Cause Analysis (MDCA) [27], we design a set of structured prompt templates to elicit aspect-level sentiment understanding from the LLM.

The prompt templates include a system prompt, defining the LLM’s role as a multimodal sentiment analysis expert, and a task prompt, providing text and visual context for aspect-level sentiment prediction. Table 1 shows representative examples illustrating their structural diversity and task relevance.

For each training sample, the overall procedure for external sentiment label generation via GPT-4o is formally in Algorithm 1.

Specifically, a combined system and task prompt is constructed based on the input text, image, and aspect term, and subsequently fed into GPT-4o to generate a predicted sentiment label. The resulting label is stored in an external sentiment label repository, which serves as prior supervision for downstream knowledge-guided training.

In the multimodal fusion stage, the Gated-Linear Aspect-Aware Fusion (GLAF) module generates a fused sentiment representation HF′, which integrates textual and visual features through gated fusion and linear attention mechanisms. To improve the alignment between the model’s sentiment predictions and external knowledge, we introduce a cosine embedding loss that encourages semantic consistency between H’ F and the sentiment-guided vector e. The loss is formally defined as: (19)ℒguide=1N∑g=1N[1−cos⁡(HF′,eg)]

This loss encourages the model to produce sentiment representations that are semantically aligned with external guidance, thereby leveraging high-level knowledge to improve robustness and accuracy during multimodal fusion. This design bridges implicit commonsense reasoning from LLMs with low-level multimodal representations, enabling the model to handle sentiment ambiguity more effectively.

We employ a multi-task training strategy integrating image–aspect relevance prediction, region-level alignment, sentiment classification, and external knowledge guidance to fully exploit multimodal signals at multiple granularities and enhance generalization via language model priors.

The overall training objective is formulated as a weighted sum of all sub-losses:(20)𝒥=ℒMASC+λ1ℒRE+λ2ℒATT+λ3ℒguidewhere ℒMASC is the primary loss for multimodal aspect-based sentiment classification, ℒRE supervises the prediction of image-aspect relevance, ℒATT enforces semantic alignment between image regions and aspect terms via region-level attention, ℒguide promotes consistency between the fused sentiment representation and external language model guidance.

The coefficients λ₁, λ₂, λ₃ are hyperparameters that control the relative contribution of each auxiliary loss. These values are selected based on validation performance to ensure that the auxiliary objectives enhance—but do not overpower—the primary sentiment classification task.

The bold entries in Table 3 indicate the optimal results of the obtained evaluations. Table 3 provides a performance comparison for the proposed GLAMSNet with several representative baseline models on the Twitter-15 and Twitter-17 datasets, with evaluations based on Accuracy (Acc) and Macro-F1 Score (F1). The baselines encompass text-only, image-only, and multimodal approaches. To ensure equitable comparison, all models are assessed under identical experimental settings.

The results show that GLAMSNet consistently outperforms most existing methods on both datasets. Compared with the strongest baseline group (GLFFCA, MDCA, and DQPSA) [16,27,28], our model yields further improvements. Specifically, GLAMSNet achieves F1 gains of 0.51% on Twitter-15 and 2.56% on Twitter-17 compared with GLFFCA [16], while also surpassing the more recent MDCA [27]and DQPSA [28]. MDCA frames MABSA as dual-cause analysis with large language model prompting, reaching 78.74% accuracy and 74.23% F1 on Twitter-15, and 73.51% accuracy and 71.25% F1 on Twitter-17. DQPSA introduces a dual-query prompt with an energy-based expert mechanism and delivers 79.02% accuracy and 74.86% F1 on Twitter-15, and 73.94% accuracy and 71.82% F1 on Twitter-17. Despite their competitiveness, GLAMSNet consistently surpasses both, achieving margins of 0.34%–0.82% in accuracy and 0.28%–1.20% in F1. This confirms the effectiveness of combining hierarchical alignment supervision, gated-linear aspect-aware fusion, multi-head linear attention, and external knowledge guidance into a unified framework.

Finally, EMT-DLFR [29] introduces efficient multimodal transformers with dual-level feature restoration for video-based sentiment analysis. As it addresses different modalities and datasets, direct comparison is infeasible. Nevertheless, its global–local fusion and restoration strategies provide promising directions for extending GLAMSNet to video sentiment analysis in future work.

To evaluate GLAMSNet’s performance, we track Accuracy and Macro-F1 across epochs 0–8 on the validation and test sets of Twitter-15 and Twitter-17, as shown in Fig. 3. On validation sets, accuracy rises rapidly in early epochs—63.73% to 74.96% on Twitter-15 and 43.88% to 73.13% on Twitter-17—before stabilizing, with similar trends in F1 scores. On test sets, accuracy peaks at 79.56% for Twitter-15 and 74.31% for Twitter-17, while F1 reaches 75.14% and 73.02%, respectively, maintaining stable performance thereafter. GLAMSNet converges slightly faster on Twitter-15, but its peak results on Twitter-17 demonstrate adaptability to complex data. These trends confirm the effectiveness of the GLAF module and external guidance in achieving robust, high-accuracy sentiment prediction.

As shown in Fig. 4 (3 × 3 grid layout), each sample presents the original image, attention-weighted regions, and predicted (red) vs. ground-truth (blue) bounding boxes. The first row in Fig. 4 shows a sports achievement from Twitter-15, depicting Kevin Durant and Russell Westbrook in a high-scoring game. The attention map [45] focuses on faces and hand gestures but partially covers the background, resulting in an IoU of 0.617, indicating background interference in dynamic scenes. The second row shows a sports honor from Twitter-17, where Kevin Durant receives the NBA Championship and Finals MVP awards. The model attends to the player, trophy, and podium with minimal background noise, achieving an IoU of 0.722 and reliable spatial focus in positive scenarios. The third row presents an entertainment performance from Twitter-17, featuring Mark Ronson and Lady Gaga at the Met Gala. The model focuses on both performers and the central stage with little background attention, yielding an IoU of 0.895, which reflects precise spatial focus and strong cross-modal alignment for accurate sentiment localization.