Multimodal discrimination detection integrates image and text information to comprehensively identify discriminatory content. It addresses the limitations of single-modality approaches, which often fail to capture the complexity of discriminatory information. For instance, Kiela [12] proposed an early fusion method combining image and text modalities to detect discriminatory internet memes. Ma [13] utilized a self-supervised label generation module alongside Bidirectional Encoder Representations from Transformers (BERT) and Residual Network (ResNet) models to enhance feature learning without requiring additional annotations. Similarly, Chen and Pan [14] implemented the OSCAR+ model with Optical Character Recognition (OCR) technology to improve detection performance.

Nie et al. [15] proposed MAGIC, a multimodal dialogue system that interprets user intent within multimodal contexts and dynamically determines the response type and modality. This context-aware and modality-adaptive approach provides useful inspiration for improving multimodal discrimination detection in complex semantic environments.

Furthermore, models such as HateCLIPper [16] and InterCLIP-MEP [17] explored various modality interactions in Contrastive Language-Image Pre-training (CLIP)’s visual and linguistic representations to address the challenges of hate meme detection. Recent advances in multimodal pre-trained models, including Vision-and-Language BERT (ViLBERT) [18], VisualBERT [19] and Universal Image-Text Representation (UNITER) [20], have employed Transformer architectures to enhance cross-modal interactions, significantly boosting task performance. Nevertheless, challenges persist, such as modality heterogeneity, noise in training data, and the diverse manifestations of discriminatory content. Moreover, the lack of fine-grained annotations in existing datasets hinders models’ ability to process metaphorical and contextually complex discrimination.

Multilingual discrimination detection focuses on identifying discriminatory content across diverse languages and cultural contexts. Multilingual pre-trained models, such as mBERT, XLM-R, and multilingual Text-to-Text Transfer Transformer (mT5) [21], offer an effective solution for cross-lingual detection by leveraging large-scale corpora to capture both commonalities and differences between languages. However, these models still exhibit limited generalization capabilities, particularly in low-resource languages and implicit discrimination detection, due to variations in linguistic expression and cultural context. Montariol [22] enhanced the models’ ability to transfer across languages by introducing auxiliary tasks like sentiment analysis and named entity recognition. However, semantic differences remain a significant challenge. Röttger [23] introduced the MULTILINGUAL HATCHECK framework, which covers 10 languages and highlights the complexity of cross-cultural applications through 36,582 comparison cases, offering a more comprehensive perspective on multilingual discrimination detection.

To enhance the cross-lingual generalization of CLIP, we replace its original text encoder with XLM-R, a multilingual pre-trained model supporting over 100 languages. Built on RoBERTa, XLM-R effectively captures multilingual semantics and performs well in low-resource settings, making it suitable for this task. Given a batch of image-text pairs {(Ii,Ti)}i=1N, where Ii and Ti denote the image and text features of sample i, each text Ti is encoded by XLM-R into a multilingual high-dimensional representation ti, as defined in Eq. (1). (1)ti=XLM-R(Ti)=FFN(SelfAttention(E(Ti)))where E(Ti) denotes the word embedding matrix of the input text. SelfAttention (⋅) is the multi-head self-attention mechanism for modeling contextual dependencies, and FFN (⋅) is the feedforward neural network. The output ti∈Rdt is the high-dimensional text feature.

For visual encoding, we adopt the original Vision Transformer (ViT) backbone from CLIP. ViT splits images into fixed-size patches and applies Transformer layers to capture global semantics, outperforming CNN-based models such as ResNet and ConvNeXT in vision and multimodal tasks. The resulting global image feature is denoted as vi, as shown in Eq. (2). (2)vi=ViT(Ii)=FFN(SelfAttention(PatchEmbedding(Ii)))

The above process separately extracts image and multilingual text features, laying the foundation for the subsequent feature alignment and fusion of text and image modalities.

In order to solve the difference in dimension between image and text features, a linear transformation operation is used to map the two modal features into a unified feature space, as shown in Fig. 3. The specific form of the transformation is given in Eq. (3). (3)V′=Wvvi+bv;T′=Wtti+bt

where Wv∈Rd×dv and Wt∈Rd×dt are learnable linear transformation matrices, and bv and bt are bias terms. V′∈Rd and T′∈Rd represent the image and text features after dimensional alignment.

After completing the feature dimension alignment according to Eq. (4), a cross-attention mechanism is utilized to achieve effective interaction between image and text features. Specifically, image feature V^′ is used as Query and text feature T^′ is used as Key and Value. By calculating the attention weight Av of an image feature to a text feature, the text feature can be made to focus on the part related to the semantics of the image; similarly, the attention weight At of a text feature to an image feature can guide the image feature to focus on the part related to the semantics of the text. The specific calculation process is as follows: Computation of Attention Weights for Text Guided by Images: (4)Av=softmax(V′(T′)⊤d)

The attention weights for images guided by text are computed as shown in Eq. (5): (5)At=softmax(T′(V′)⊤d)

Calculation of fused image and text features is given in Eq. (6): (6)Fv=AvT′;Ft=AtV′

In order to further refine the fused representation, a weighted summation integrates the image and text features into the final multimodal representation F. Learnable weighting coefficients α and β (α+β=1) dynamically adjust the contributions of each modality during the fusion process, ensuring a balanced representation of image and text information. The final fusion formula is given in Eq. (7): (7)F=αFv+βFt

This feature alignment and fusion strategy effectively mitigates the challenges posed by dimensional differences and insufficient cross-modal interactions in multilingual multimodal discrimination detection tasks.

Given a batch of image-text pairs {(Ii,Ti)}i=1N, where Ii represents the image features of the sample i and Ti represents the text features of the sample i. The label of each sample indicates whether it is discriminatory (i.e., discriminatory or non-discriminatory). Unlike traditional contrastive learning methods, in this study, the input features are no longer the original image and text features, but the fusion representation obtained by fusing the image and text features, and the label information is used as the classification basis to guide the contrast learning.

The complex discrimination signal detection process based on the dynamic memory mechanism is shown in Fig. 4. This module is based on the fused multimodal features obtained from the multimodal feature alignment and fusion module. By constructing a memory bank that stores the feature representations of historical samples along with their label information, it enables similarity retrieval and similarity-weighted aggregation between the current sample and historical samples. Specifically, the memory bank M storing historical samples is continuously updated. Cosine similarity measures the similarity between the current sample h~f and historical samples h~fi within M. The Top-K most relevant samples are retrieved and aggregated using similarity-weighted values to generate enhanced feature representations h^f. The concatenated current and enhanced features are then fed into the gated recurrent unit (GRU) module, which dynamically integrates historical and current features through update and reset gates, producing the final representation. These integrated features are subsequently classified by a Softmax classifier to predict category labels. Finally, the prediction results and feature representations are used to update M, ensuring continuous learning and adaptation.

Multi³Hate dataset [24]: A multimodal, multilingual, and multicultural dataset designed for hate speech detection. It contains 300 parallel meme samples across five languages—English, German, Spanish, Hindi, and Chinese—resulting in a total of 1500 examples. Each sample is annotated by at least five native speakers from diverse cultural backgrounds and classified as either hate speech or non-hate speech. To partition the dataset, a 5-fold cross-validation approach is used to maximize the utility of limited samples. Specifically, the 300 samples in each language are randomly divided into five subsets, with four subsets serving as the training set and one as the validation set in each iteration. This process is repeated over five rounds of training and evaluation. The final model performance is evaluated by averaging the results from all validation rounds, thereby reducing bias introduced by dataset partitioning.

In the visual model, ConvNeXT outperforms both ViT and ResNet across all languages. On the BHM dataset, ConvNeXT achieves an F1 score of 0.695, surpassing ViT’s 0.674 and ResNet’s 0.576. VRCL (our) further exceeds ConvNeXT, improving the F1 score to 0.714. Additionally, in the Multi³Hate dataset across five languages, the VRCL demonstrates an average F1 score improvement of 0.0618 over ConvNeXT, underscoring its superior performance and multilingual adaptability in visual modeling. A deeper analysis of these baseline models reveals several critical limitations. For instance, while ViT achieves a moderate average F1 score (0.674), its performance drops significantly on Chinese (F1 = 0.541), indicating challenges in handling linguistic diversity. ResNet performs even worse, with an average F1 score of 0.564, suggesting limited capacity to extract discriminative features from visual data. Although ConvNeXT demonstrates relatively better generalization (average F1 = 0.616), it still encounters difficulties with certain languages, such as Chinese (F1 = 0.608). These observations further highlight the effectiveness and robustness of the proposed VRCL model in managing multilingual visual inputs.

In evaluating text model performance, the multilingual pre-trained model mBERT consistently outperforms traditional models such as TextCNN and Bi-LSTM. For example, on the BHM dataset, mBERT achieves an F1 score of 0.658, compared to 0.611 and 0.643 for TextCNN and Bi-LSTM, respectively. Moreover, mBERT demonstrates strong performance across all languages in the Multi³Hate dataset, particularly in Hindi (IN), where it achieves an F1 score of 0.674. A closer examination of the text-only baseline models further underscores the value of multimodal integration. TextCNN achieves a relatively low average F1 score of 0.588, suggesting that relying solely on textual inputs may overlook important visual cues essential for accurate discrimination detection. Although Bi-LSTM performs slightly better than TextCNN, with a comparable average F1 score of 0.588, it still faces challenges in modeling complex linguistic patterns–particularly in multilingual contexts. These findings emphasize the limitations of traditional text models in capturing nuanced and cross-lingual semantics. Notably, while mBERT alleviates some of these limitations with its strong multilingual capabilities, our proposed VRCL model still surpasses mBERT’s performance in all languages except Hindi, further demonstrating VRCL’s broader adaptability and effectiveness in detecting multilingual discriminatory content.

In the task of multimodal discrimination detection, VRCL demonstrates significant advantages across multiple datasets and language settings, consistently outperforming mainstream models such as ALBEF, CLIP, and G²SAM. Specifically, on the BHM dataset, VRCL achieves an F1 score of 0.714, notably higher than ALBEF’s 0.676, indicating stronger capabilities in multimodal feature fusion and semantic representation. On the Multi³Hate dataset, VRCL attains the highest F1 score of 0.721 on the English (US) subset, while also showing stable and robust performance across other language subsets: 0.719 on German (DE) and 0.678 on Chinese (CN), both surpassing the best results of existing methods.

Although G²SAM, as one of the latest fusion models, achieves a competitive average F1 score of 0.662, close to VRCL, it still falls short. This gap is mainly attributed to G²SAM’s current and historical feature fusion mechanism, which has not been fully optimized and thus fails to effectively leverage historical semantic context. Consequently, its performance in cross-modal semantic alignment and discrimination signal recognition is limited. In fact, mainstream multimodal models such as ALBEF, CLIP, and G²SAM mostly adopt a “single-step” feature fusion strategy, lacking the ability to remember and model semantic evolution over time. This limitation makes it difficult for these models to accurately capture subtle cross-modal semantic changes in the presence of semantic drift, implicit discrimination, or complex contextual variations.

In contrast, the superior performance of VRCL primarily benefits from its innovative incorporation of a historical feature memory mechanism and a GRU-based sequential modeling architecture. By dynamically retrieving and integrating the most semantically similar historical samples with the current features, VRCL effectively captures the semantic evolution process, greatly enhancing feature expressiveness and semantic continuity. As a result, VRCL is able to more accurately identify cross-modal semantic biases and discriminatory signals, demonstrating stronger robustness and generalization capabilities.

To assess the contribution of each module to the model’s performance, this study conducted ablation experiments by systematically removing individual modules and tracking changes in the F1 score. Table 3 presents the experimental results.

Experiment 1: Replacing the original CLIP model’s text extractor with XLM-R resulted in a decrease in the F1 score to 0.685. This indicates that XLM-R is essential for cross-linguistic feature capture and semantic understanding, and its absence significantly impacts performance in multilingual tasks.

The proposed VRCL model demonstrates strong performance in integrating multilingual textual and visual features. However, its computational complexity warrants careful consideration. Both the text encoder (XLM-R) and image encoder (ViT) are based on the Transformer architecture, resulting in a time complexity of approximately 𝒪(n2⋅d) for textual inputs and 𝒪(p2⋅d) for visual inputs, where n denotes the input text length, p is the number of image patches, and d is the hidden dimension. In addition, the introduction of the semantic memory module improves contextual reasoning but also incurs additional storage and retrieval overhead. While the overall resource consumption is higher than that of traditional unimodal models, the VRCL model remains feasible for training and inference on modern GPU infrastructures. The advantages of VRCL lie in its robust multilingual adaptability, enhanced cross-modal alignment, and improved generalization through memory-based semantic enhancement. Nevertheless, the model still faces challenges such as relatively high computational costs and limited performance on certain low-resource languages, indicating room for further optimization in efficiency and cross-lingual generalization.