In the domain of Arabic image captioning, researchers in [29] proposed an innovative approach tailored to generating image captions specifically for clothing images. Leveraging deep learning techniques, their model proficiently generates Arabic captions describing clothing items, enhancing accessibility to fashion-related visual content. Meanwhile, researchers in [30] investigated Arabic image captioning, focusing on the impact of text pre-processing on attention weights and BLEU-N scores. Their work sheds light on optimizing the caption generation process in Arabic, taking into account the nuances of text pre-processing. Authors in [31] ventured into automatic Arabic image captioning using a combination of Recurrent Neural Networks (RNNs) and Long Short-Term Memory (LSTM) language models, alongside Convolutional Neural Networks (CNNs). Their model demonstrated the feasibility of generating Arabic image captions, marking a significant milestone in the field of Arabic image description.

In the realm of English image captioning, researchers in [32] proposed a novel approach for automatic caption generation for news images. They employ a multimodal approach that integrates both image content and associated news articles to create coherent and informative image captions. Researchers in [33] introduced a compact image captioning model with an attention mechanism, focusing on the efficiency of caption generation. Their research contributes to streamlining image captioning models for practical applications. In addition, researchers in [34] provided valuable insights from lessons learned during the 2015 MSCOCO Image Captioning Challenge, highlighting key takeaways and challenges in image captioning.

Researchers in [37] introduced a novel approach, “Wikily” Supervised Neural Translation, tailored to Arabic-English cross-lingual tasks, including image captioning. Their model leverages Wikipedia as a resource for cross-lingual supervision, showcasing its efficacy in generating accurate image captions across the Arabic-English language barrier. In addition, researchers in [38] contributed to Arabic-English cross-lingual image captioning by providing valuable resources and end-to-end neural network models, enriching the accessibility and understanding of visual content for Arabic speakers.

Researchers in [39] introduced COCO-CN, a resource for cross-lingual image tagging, captioning, and retrieval tasks involving Chinese and English. Their work underscores the significance of bridging the linguistic gap between these two languages in the context of visual content. Additionally, researchers in [40] explored fluency-guided cross-lingual image captioning, particularly focusing on Chinese-English pairs. Their approach highlights the importance of fluency in generating high-quality image captions that resonate with speakers of both languages.

Researchers in [41] presented multimodal pivots for image caption translation, addressing the German-English cross-lingual challenge. Their work explores strategies for effectively translating image captions between these languages using multimodal approaches. Furthermore, researchers in [42] contributed to the field with the creation of Multi30k, a multilingual English-German image description dataset. Their work serves as a valuable resource for cross-lingual image captioning research, fostering improved understanding and communication between German and English speakers.

Researchers in [43] presented the STAIR captions dataset, a substantial resource for Japanese image captioning. Their work advances the availability of image description data for Japanese speakers, contributing to the field’s progress in Japanese-English cross-lingual image captioning. Moreover, researchers in [44] delved into cross-lingual image caption generation with a focus on Japanese-English pairs. Their work explores techniques to generate image captions that transcend language barriers, enhancing cross-cultural communication.

In generating Table 1, we employed a meticulous and systematic literature review process to ensure the precision and comprehensiveness of the information presented. This process entailed the following steps:

Keyword-Based Search: We initiated our literature review with a keyword-based search in major academic databases. The keywords were carefully chosen to encompass the core themes of our study, namely ‘cross-lingual image captioning’, ‘multimodal learning’, and ‘semantic matching’.

Through this rigorous process, Table 1 was crafted to provide a detailed and accurate summary of existing literature in the field of cross-lingual image captioning, serving as a foundational reference for our study and future research in this domain.

For heterogeneous images and sentences, the first step is to map the images and sentences into a common embedding space and measure semantic relatedness. As shown in Fig. 2, the image’s semantic embedding network, denoted as, (fl) consists of a CNN encoder (using the pre-trained ResNet-101 model) and a fully connected layer (referred to as FCE). The text’s semantic embedding network, denoted as (fS), is composed of a single-layer LSTM (denoted as *** LSTME). The final hidden vector of (LSTME) at the last time step defines the semantic vector in the common embedding space for the input sentence.

By inputting image-sentence pairs ((I), ST), you obtain the image’s feature embedding, (fl(I)), in the common semantic space and the sentence’s embedding feature, (fS(ST)), in the common semantic space. For matching pairs ((I), ST), negative examples are found within the same batch. Specifically, sentences (ST′) that do not match with (I) and images (I′) that do not match with (ST) within the same batch are identified. The pretraining process involves minimizing a bidirectional ranking loss in the common semantic space.

The goal of this process is to align the semantics of images and sentences in a shared embedding space and measure their semantic relatedness. (2)L(θμ)=∑I∑ST′max(0,Δ−fl(I)fS(ST)+fl(I)fS(ST′))+∑I′∑STmax(0,Δ−fl(I)fs(ST)+fl(I′)fS(ST))

Eq. (2) describes a bidirectional ranking loss, a common approach in cross-modal semantic matching. It is designed to fine-tune the semantic alignment between images and their corresponding textual descriptions, following a methodology widely used in multimodal learning tasks. For further theoretical background and application of similar loss functions, readers are referred to [53].

In the equation, (Δ) represents a boundary hyperparameter, and (θμ) represents the learning parameters for the (FCE) and (LSTME) layers in this module.

In addition, this study also has axis language sentence-pseudo-label sentence pairs (Sp, ST), which can provide data support for measuring the semantic similarity between target language sentences and axis language sentences. In this section, cross-lingual semantic matching is introduced to enhance the semantic relevance of sentences, using a similar semantic embedding network mechanism as in Section 3.3.1 to align the embedding vectors of the target language and axis language. Both the encoder for the target language and the encoder for the axis language use a single-layer BG-RU (Bidirectional Gated Recurrent Unit) structure, with the hidden vector at the end of BGRU used as the sentence feature vector. fp is the axis language feature mapper (BGRUPE), and fT is the target language feature mapper (BGRUTE). Similarly, pretraining is performed by minimizing bidirectional ranking loss to align the common semantic space. (3)L(θρ)=∑Sp∑ST′max(0,Δ−fP(SP)fT(ST)+fP(SP)fT(ST′))+∑Sp′∑STmax(0,Δ−fP(SP)fT(ST)+fP(Sp′)fT(ST))

In the equation, for matching pairs (SP, ST), ST′ is the negative example from the pseudo-label sentence set in the same batch that does not match (SP), and (SP′) is the negative example from the axis language sentence set in the same batch that does not match (ST). (θP) represents the learning parameters for the (BGRUPE) and (BGRUTE) layers in this module.

We utilized two benchmark datasets for our experiments, as outlined in Table 2.

Flickr8k (English Dataset): This dataset consists of 8092 images, with each image accompanied by five annotated English descriptions. To ensure data consistency, we divided the dataset into three sets: 6092 images for the training set, 1000 images for the validation set, and another 1000 images for the test set. English word segmentation was performed using the “Stanford Parser” tool (https://stanfordnlp.github.io/CoreNLP/index.html), retaining English words that appeared at least 5 times and truncating sentences exceeding 20 words in length.

AraImg2k (Arabic Dataset): This dataset comprises 2000 images, each associated with five manually annotated Arabic descriptions. To maintain uniformity, we split this dataset into three subsets: 1500 images for training, 250 images for validation, and 250 images for testing. Arabic word segmentation followed the method proposed by [55], retaining Arabic words occurring at least 5 times. The segmentation data was extracted from the ATB and stored in text files, with each sentence treated as a time-series instance. Each file contained information for a single sentence.

It is important to note that the images and sentences in AraImg2k and Flickr8k are distinct from each other.

For evaluating the generated image descriptions, we employed semantic evaluation metrics commonly used to assess the quality of machine-generated text compared to human references. These metrics provide insights into the model’s performance in generating accurate and fluent image descriptions. The following evaluation metrics were used:

BLEU (Bilingual Evaluation Understudy): Measures the quality of machine-generated text by comparing it to human references. We reported BLEU-1, BLEU-2, BLEU-3, and BLEU-4 scores.

We utilized the pre-trained ResNet-101 model and a fully connected layer to extract image features vI resulting in a dimension of d = 512. These features were then used as the initial input to the decoder LSTMG at time step 0.

The image semantic embedding network consists of the pre-trained ResNet-101 model and a fully connected layer. The target language encoder employed a single-layer LSTME structure.

Both the axis language and target language encoders used single-layer BGRUPE and BGRUTE frameworks with a hidden layer dimension of 512. The output dimension of BGRU was set to 1024.

The language sequence model utilized a single-layer LSTML. The hidden layer dimension and word embedding dimension for all LSTM structures in this study were set to d = 512.

Throughout the model training process for both subtasks, dropout was set to 0.3, the batch size during pre-training was 128, and during reinforcement training, it was 256.

After the pre-training of the Semantic Matching module (Section 3.3) and the Language Optimization module (Section 3.4), the learning parameters θµ, θρ, and θω remained fixed. Both of these modules provided rewards to guide the Image Description Generation module (Section 3.2) in learning more source-domain semantic knowledge and target-domain language knowledge.

For the task of generating image descriptions in English using Arabic as the axis language, the learning rate for pre-training the Image Description Generation module is 1E-3. The learning rates for pre-training the source-domain semantic matching module and the target-language domain evaluation module are set to 2E-4. When training with language evaluation rewards and multi-modal semantic rewards, the learning rate for the Image Description Generation module is 4E-5. The values of α, β, and γ are set to 1, 1, and 0.15, respectively.

However, for the task of generating image descriptions in Arabic using English as the axis language, the learning rate for pre-training the Image Description Generation module is 1E-3. The learning rates for pre-training the source-domain semantic matching module and the target-language domain evaluation module are set to 4E-4. When training with language evaluation rewards and multi-modal semantic matching rewards, the learning rate for the Image Description Generation module is 1E-5. The values of α, β, and γ are set to 1, 1, and 1, respectively.

To assess the effectiveness of the Image & Cross-Language Semantic Matching module and the Target Language Domain Evaluation module, ablation experiments were conducted. Table 3 presents the results of ablation experiments for the task of cross-language image description from Arabic to English and from English to Arabic.

In Table 3, the baseline model employed Eq. (1), L(θG) as the objective function. The model trained with the rr∣vl reward represents participation in the Cross-Modal Semantic Matching module (Section 3.3.1). The model trained with the rr∣vP reward represents participation in the Cross-Language Semantic Matching module (Section 3.3.2). The model trained with the r∣g reward represents participation in the Target Language Domain Evaluation module (Section 3.4). The model that jointly uses rewards rr∣vl, rr∣vP, and r∣g is also evaluated.

The results of the ablation experiments shed light on the impact of various reward components on our model’s performance for both cross-language English and Arabic image captioning tasks.

Figs. 3 and 4 illustrate the detailed evaluation metrics for cross-language English and Arabic image captioning, respectively, further elucidating the findings of our study.

According to Table 3, introducing the Multi-Modal Semantic Relevance Reward rr∣vl and Cross-Language Semantic Matching Reward rr∣vP led to improvements in several performance metrics. Notably, the CIDEr scores increased for English and Arabic by 1.2% and 1.4%, respectively, compared to the baseline. These results indicate that the Image & Cross-Language Semantic Matching module enhanced the semantic relevance of the generated sentences.

The Target Language Domain Evaluation Reward r∣g played a positive role in both cross-language English and Arabic image description tasks. For cross-language English and Arabic image captioning, CIDEr scores increased by 2.3% and 2.2%, respectively, compared to the baseline.

Furthermore, the combined effect of the Target Language Domain Reward r∣g and Image-Sentence Semantic Matching Reward rr∣vl resulted in substantial performance improvements in both tasks. For cross-language English and Arabic image descriptions, CIDEr scores increased by 10.3% and 3.6%, respectively, compared to the baseline. This indicates that combining these rewards results in descriptions that are more semantically consistent with the images.

Finally, when all rewards rlg, rrlvl, and rrlvP were considered jointly, significant improvements were observed across all metrics. In comparison to the baseline, CIDEr scores increased for English and Arabic by 13.8% and 4.9%, respectively. This highlights the effectiveness of incorporating guidance from both the Image & Cross-Language Domain and Target Language Domain in improving fluency and semantic relevance in generated sentences.

It is worth noting when comparing experiments involving only the rlg reward to those with both the rlg and rrlvl rewards, CIDEr scores for English and Arabic increased by 8.0% and 1.4%, respectively. The differential impact of the rrlvl reward on the two subtasks is noticeable. This difference is because, in the cross-language English image description subtask, the test set from Flickr8k contains various scenes, such as people, animals, and objects, making the visual semantics richer and more diverse. In this scenario, the Image-Sentence Semantic Matching Reward rrlvl demonstrates excellent semantic complementing ability (resulting in an 8.0% increase).

However, in the cross-language Arabic image description subtask, the test set AraImg2k primarily features images with a single visual scene (mostly focusing on people). Consequently, there is limited visual semantics to complement. Despite this, the method still improved performance by 1.4%.

The data in Table 3 was obtained through experimental trials conducted using the Python programming language. The trials were designed to evaluate the impact of different reward mechanisms in our cross-lingual image captioning model. The experiments were carried out on the Flickr8k dataset for English captions and the AraImg2k dataset for Arabic captions. These results demonstrate the effectiveness of our model in enhancing cross-lingual image captioning through a combination of semantic matching and language evaluation rewards.

Table 4 provides a comparative analysis of various methods, including our study, for cross-language English image description tasks. The performance metrics presented were derived from experimental results on the English Flickr8k dataset. These results were obtained through systematic testing and evaluation of our model against established benchmarks in the field.

A comparison between our work and previous studies based on the data in Table 4 demonstrates our model’s superior performance. Our approach consistently surpasses prior methods across diverse evaluation metrics. Notably, it excels in BLEU and CIDEr scores, signifying its improved accuracy and diversity in generating English image descriptions.

Fig. 5 shows the visual results of this model on the cross-language English image description task using the Flickr8k test set. The red font indicates semantic errors from the baseline model’s translation, while the green font represents correct semantics from this model’s translation. The figure illustrates that this model generates descriptions closer to the visual content of the images. For example, it can identify object attributes, replace the incorrect ‘women’ with ‘men’, and infer object relationships, correcting “A red Jeep driving down in a mountainous area” to “driving down a rocky hill.” Additionally, this model’s generated sentences have fewer stylistic differences from the target language. For instance, the sentences generated by this model tend to follow the target language’s style of “someone doing something somewhere,” while the baseline model prefers to add attributive modifiers to objects.

Similar to Table 4, Table 5 presents a comparative analysis of various methods for cross-language Arabic image description tasks. The data was derived from testing our model on both the Arabic Flickr8k and AraImg2k datasets, providing a comprehensive performance evaluation across multiple metrics.

Comparing our work with previous studies based on the data in Table 5 reveals a significant advancement. Our model consistently outperforms existing methods across all evaluation metrics. Notably, it achieves remarkable improvements in BLEU, CIDEr, and SPICE scores, reflecting its superior accuracy, diversity, and linguistic quality in generating Arabic image descriptions.

Fig. 6 indicates that the descriptions generated by the proposed model are more semantically relevant to the visual content. For example, the proposed model is capable of supplementing and correcting missing or incorrect visual information, resulting in more coherent and accurate sentences. Additionally, the sentences generated by the proposed model align more closely with the style of real descriptions, presenting a continuous and concise style.

In summary, the performance analysis of the cross-language Arabic image description task shows that the proposed model consistently outperforms baseline and state-of-the-art methods in various evaluation metrics. It generates descriptions that are not only more semantically accurate but also stylistically aligned with the target language, making it an effective solution for cross-language image description tasks.

In conclusion, this study represents a significant contribution to the field of cross-lingual image description. Our method’s ability to generate culturally relevant and semantically coherent captions across languages is not just an academic advancement; it has practical implications for enhancing multilingual understanding and communication. The introduction of the AraImg2k dataset, along with our novel methodologies, sets a new benchmark in the field and lays the groundwork for future research in this area.