Within our captioning network, the visual features are initially extracted from the input image to enable their utilization in subsequent processing by the language model. The initial phase of generating a description for an image involves acquiring the visual representations of the input image. A Faster-RCNN network utilizing ResNet-101 extracts object features from input images, producing an object feature matrix, represented as V, which contains N object feature vectors. (1)V={v1, v2,…,vN}

In this representation, vi∈Rd refers to a vector that represents the features of an object, in which i ranges from 1 to N, and V∈RN×d represents the matrix of features of the objects. In addition, a mean-pooled object features v¯ of the input image is also used as an extra input to the image captioning system, with the following definition: (2)v¯=1N∑i=1Nvi

Here, v¯∈Rd.

According to the framework we propose, local visual features of the input image are essential for boosting the model’s performance. By accurately capturing these details, we can significantly enhance the overall effectiveness of the image captioning process. To focus on these local visual features, our model relies on a conventional visual attention module, which enables the model to selectively emphasize important aspects of the image. This visual attention process can be described by a set of formulas that enable precise attention to the most important visual features as a result of this operation. (3)αti=Wc⋅tanh⁡ (Wa⋅hta+Wb⋅vi) (4)βt=softmax⁡ (αt) (5)v^t=∑i=1Nβti⊙viwhere v^t∈Rd, βt∈RN, and αt∈RN. Wc∈Re, Wa∈Rg×e, and Wb∈Rd×e are trainable weights. hta∈Rg is the hidden state of the attention LSTM.

A diagram illustrating the language model architecture is shown in Fig. 2. As a basis for the approach we propose, the UpDown framework is used as the baseline structure, known for its effectiveness in image captioning tasks. There are two LSTM layers in the framework: one for language LSTMs, denoted LSTMlan, and another for attention LSTMs, denoted LSTMatt. As a result of leveraging these LSTM layers, the model can better capture and integrate sequential information from the input data. The hidden states of the attention LSTM, represented as hta∈Rg, and the language LSTM, represented as htl∈Rg, can be determined by the following equations, which ensure that the interaction between visual features and language generation is precise and dynamic.

(6)hta=LSTMatt (ht−1a; [ht−1l,E⋅yt−1,v¯]) (7)htl=LSTMlan(ht−1l;[hta,v~t,v^t])

Here, E∈Rm×q represents the word embeddings matrix, and yt−1∈Rm denotes the token generated in the previous time step. These embeddings play a crucial role in capturing the semantic meaning of the tokens. v~t∈Rd is an updated context vector which will be explaned in Section 3.5. To predict the next token, the hidden state of the language LSTM, htl, is fed into a fully connected layer with a softmax activation function. This setup allows the model to generate a probability distribution pt over the entire vocabulary, ensuring that the most likely subsequent token is selected based on the context provided by htl. The process is described by the following equation: (8)pt=softmax⁡ (htl⋅Wg)where pt∈Rm. Wg∈Rg×m represents the weights to be trained.

The input word to the attention LSTM at each time step during the training phase of our models is taken from the ground truth annotation. This approach ensures that the model learns from accurate data. Conversely, the input word for the attention LSTM in the testing phase, as defined in Eq. (6), comes from the word predicted in the previous time step. This method allows the model to generate sequences based on its learned predictions. As an initial input in both training and testing, a special “begin-of-sequence” token is used. As the generation of the description’s words continues, a special “end-of-sequence” token will be predicted or a maximum description length will be reached indicating the end of the generated description, ensuring a coherent and contextually relevant output.

The visual feature detector module (VFD) is an ordinary neural network that serves as a detector to dynamically select the most relevant visual features at each time step, in our case the number of top features is referred to as k. The VFD module consists of a concatenation layer, a fully connected layer (FC), and a softmax layer. The input to VFD comprises the output of the conventional visual attention module v^t and the hidden state of the attention LSTM hta. The VFD module generates an output matrix known as the selected top-related features matrix (STF), denoted as Ut, which consists of the top-k selected local visual features of the input image. The internal structure of the VFD module is illustrated in Fig. 3. Given the object features matrix V, the context vector v^t, and the hidden state of the attention LSTM hta∈Rg as input to the VFD module, we first concatenate v^t and hta as follows:(9)xt=[hta,v^t]where [ ] refers to the concatenation operation. Then, a fully connected layer is used to map the resulted vector xt∈Rd+g into another vector x¯t∈RN whose length is equal to the number of local visual features N as follows: (10)x¯t=Wx⋅xtwhere Wx∈R(d+g)×N is a learnable parameter matrix. Next, we apply the softmax activation function on x¯t to generate a new vector x^t∈RN which represents a probability distribution over N as follows: (11)x^t=softmax⁡ (x¯t)

The elements of x^t are probability values and each element of x^t represents the probability of its corresponding local visual feature. After that, the indexes of the k elements of x^t with the highest probabilities are determined and their corresponding local visual features in V∈RN×d are selected to form the matrix Ut which is given by: (12)Ut={ut1, ut2, …,utk}where uti∈Rd and Ut∈Rk×d. Ut represents a subset of the local visual features the most related to the current linguistic context at the current time step t. Fig. 4 provides an illustration of the selection of the local visual features according to their corresponding probabilities in the VFD module.

After generating the STF matrix from the VFD module, we have developed an additional module that takes the STF as its input. This module is originally a visual attention module and is referred to as the Visual Feature Visual Attention (VFVA). The main purpose of this module is to attend to the visual features of the STF matrix, yielding an updated context vector known as the Updated Context Vector (UCV). The UCV is then fed into the language LSTM of the language model to guide the generation of the subsequent word. Considering the updated context vector v~t as defined in the following equations, the VFVA module’s formulas demonstrate its role in attending to the importance of each of the various visual feature vectors. (13)δti=Wd⋅tanh⁡ (We⋅hta+Wf⋅uti) (14)γt=softmax⁡ (δt) (15)v~t=∑i=1kγti⊙utiwhere v~t∈Rd, δt∈Rk, and γt∈Rk. We∈Rg×e, Wf∈Rd×e, Wd∈Re are learnable weights.

Our image annotating network is trained using two stages: cross-entropy (XE) and CIDEr optimization. In the first stage, standard cross-entropy loss is applied, which is determined as follows: (16)LossXE=1T∑t=1T−log⁡ (pt(yt∣y1:t−1,V))

Second stage, Self-Critical Sequence Training (SCST) is utilized alongside CIDEr-D for optimizing and training the model. The loss function at this stage is specified as follows: (17)LossRL=−Ew1:T∼θ[r(w1:T)]

In this context, w1:T refers to the sampled annotation, while r indicates CIDEr-D score of sampled annotation. Gradient approximation for LossRL, symbolized as ∇θLossRL, is detailed in Eq. (18). Here, r(w^1:T) refers to the CIDEr reward for the maximally sampled annotation, and r(w1:Ts) denotes the CIDEr reward corresponding to the randomly sampled annotation. This approach enables the model to refine the generated annotations by aligning them more closely with human references. Utilizing the CIDEr-D score helps the model to effectively grasp the intricacies and relevance within the sampled annotations. Consequently, this method enhances the overall quality and accuracy of the generated captions. Additionally, it provides a robust framework for training models in tasks that require a nuanced understanding of content similarity. (18)∇θLossRL=− (r (w1:Ts)−r (w^1:T)) ∇θlog⁡ (p (w1:Ts))

Algorithm 1 outlines the training process for the VFDICM model. It begins by extracting feature representations from the input images using Eq. (1). Then, for each word position in the caption, the algorithm computes the output of the conventional visual attention module, the selected top-related feature matrix from the VFD module, and the output of the VFVA module using Eqs. (5), (12), and (15), respectively. These computations aid in generating the probability distribution for the next word using Eq. (8). The algorithm continues updating the cross-entropy loss and reinforcement learning loss based on Eqs. (16) and (17) until convergence is achieved. Finally, it returns the generated word probabilities and the model parameters.

Extensive evaluations were conducted using the MS-COCO dataset [5] in image captioning. As a popular dataset, due to its diversity and numerous caption-image pairs, this dataset served as a solid base for evaluating the effectiveness of the models. The MS-COCO dataset, frequently employed in tasks involving image annotation, is comprised of 123,287 images. Following Karpathy’s well-established data splitting method [46], 113,287 images were allocated for training, 5000 images for validation, and another 5000 images for testing. Notably, most of the MS-COCO images are associated with a total of five ground truth captions, resulting in a substantial collection of about 616,747 different ground truth captions for all the MS-COCO images. These captions vary in length, spanning a length of 5 to 49 words, offering a wide spectrum of scenarios and contexts for model training and evaluation.

To assess our model’s performance, the model was tested using a series of evaluation metrics, including CIDEr [47], METEOR [48], BLEU [49], ROUGE-L [50], and SPICE [51]. As a result, each of these metrics offers unique insight into a variety of different characteristics of the performance of the model. BLEU, a precision-based metric originally intended for machine translation, has been shown to be highly correlated with human evaluation, emphasizing n-gram precision. METEOR evaluates machine-generated translations by employing a generalized concept of unigram matching with human reference translations, which allows for a balanced assessment of precision and recall. As part of its similarity calculation, CIDEr uses cosine similarity between words in candidate and reference captions based on Term Frequency-Inverse Document Frequency weighted n-grams, thereby taking both recall and precision into account, which is particularly useful for image captioning tasks.

ROUGE, on the other hand, measures the quality of a summary by comparing it with a human-generated summary, which helps in understanding how well the model can generate concise and relevant descriptions. SPICE assesses how effectively captions represent attributes, objects, as well as their relationships, offering a more semantic assessment of captions generated. To simplify the representation of these metrics, we denote METEOR, CIDEr, ROUGE-L, SPICE, and BLEU-n (n = 1, 2, 3, 4), as M, C, R, S, and B-n (n = 1, 2, 3, 4), respectively.

To extract features of objects from images, Faster-RCNN is used based on ResNet-101, resulting in object features of dimensions 36 × 2048. To manage longer sentences, those exceeding 16 tokens are truncated. In constructing our vocabulary, only words occurring more than five times are included, yielding a vocabulary of 9487 words for MS-COCO. Word embeddings are 1000-dimensional, and hidden states are set for both LSTMs to 1000. During our experiments, we select k to be equal to 10 since it gives us the best scores.

In our image captioning model, we utilize a visual features vector of size d=2048 to represent the input image. LSTM hidden state sizes g=1000 allow for the capture of complex linguistic patterns during caption generation. Words in our vocabulary are embedded in vectors of length q=1000. The features of N=36 objects in the image are used for data extraction. The vocabulary for our captioning task consists of m=9487 unique words for MS-COCO. Additionally, we incorporate an internal hidden attention mechanism of size e=512 to improve the model’s ability to generate captions for relevant sections of an image. The number of selected most relevant visual features is set to k=10.

For training our annotation networks, we employ the Adam optimizer and conduct training for 50 epochs in the cross-entropy stage, followed by 100 epochs in the CIDEr optimization stage. Initially, the learning rate is set at 0.0005 and subsequently decreases by a factor of 0.8 every 5 epochs during cross-entropy training and every 10 epochs during CIDEr optimization. Regarding the batch size, a batch size of 40 is chosen. The scheduled sampling percentage increases by 5% every 5 epochs until it reaches 25% during cross-entropy training. The gradients are clipped to an absolute maximum of 0.1. We employ a dropout ratio of 0.5 in our model. Testing is conducted with a beam size of 3 with the beam search strategy. Our networks are developed using the PyTorch framework.

To construct the vocabulary for our proposed model, we underwent several processing steps with the ground truth caption corpus. Initially, we analyzed all the ground truth sentences, totaling 6,454,115 words, and identified 27,929 unique words. Following word counting, we retained words that appeared more than five times, resulting in a vocabulary of 9486 unique words, while discarding 18,443 unique words. These discarded words constituted approximately 66% of the total unique words in the caption corpus. However, in terms of overall word count, they represented merely 0.5%. While the percentage of discarded unique words may seem high, the actual impact on the model’s performance is negligible due to their small contribution to the total word count. All discarded words were replaced with the ‘Unknown’ special token, which was subsequently added to the vocabulary, expanding it to 9487 words. Additionally, we tokenized all ground truth sentences, replacing each word with its corresponding unique index or integer value assigned from the vocabulary. Sentences longer than 16 words were truncated, and those shorter than 16 words were padded using a ‘0’ digit, serving as zero padding. Furthermore, we prepended the beginning-of-sequence token to the start of each ground truth caption and appended the end-of-sequence token to the end. These preprocessing steps prepared the ground truth captions for training our models.

As shown in Table 1, our model performed in cross-entropy and CIDEr optimizations using MS-COCO data. The cross-entropy results indicate that our proposed image captioning model surpasses the baseline in most evaluation metrics, notably excelling in METEOR, BLEU-4, SPICE, CIDEr, and ROUGE. Furthermore, experimental results underscore the superiority of our model over the baseline in terms of CIDEr optimization scores across various evaluation metrics, with significant differences observed in BLEU-4, BLEU-1, ROUGE, METEOR, and CIDEr metrics. These scores unequivocally demonstrate the superior performance of our proposed framework, highlighting the effectiveness and value of our approach.

VFDICM demonstrated significant enhancement during the CIDEr phase, where optimization considers caption quality and diversity. Significant advancements were observed in the BLEU-4, BLEU-1, METEOR, CIDEr, and ROUGE metrics. The integration of VFD and VFVA within our network significantly improves the image captioning algorithm’s ability to capture rich visual representations and features, resulting in improved overall performance and description generation. The VFD module updates the visual features matrix, while the VFVA module directs attention to this matrix, generating an updated context vector for the language model to produce informative descriptions.

The comparison of other models with our model using the MS-COCO dataset is detailed in Tables 2 and 3, covering cross-entropy optimization and CIDER stages. Table 2 focuses on the cross-entropy results, where our model demonstrates remarkable performance. Specifically, our model outperforms most other models in several key metrics, including ROUGE-L, BLEU-4, BLEU-3, and SPICE. Additionally, it achieves the second-highest scores in CIDEr, BLEU-1, and BLEU-2. It is noteworthy that the r-GRU model secured METEOR’s highest score in this phase. Significant differences in scores between our model and others across the majority of metrics underline its greater capabilities during the cross-entropy training phase. Moving on to Table 3, which displays the results from the CIDEr optimization stage, our model continues to excel. It achieves top scores in several metrics such as ROUGE-L, BLEU-3, BLEU-4, BLEU-2, BLEU-1, and METEOR. However, it ranks second in the CIDEr and SPICE metrics. The significant variance between our model’s performance and that of other models in metrics like METEOR, BLEU-3, BLEU-2, and BLEU-1 further highlights the effectiveness of our model at this optimization stage.

In more detail, the cross-entropy phase results in Table 2 reveal that our model is not only competitive but often leading in many critical areas. For example, its performance in BLEU-3 and BLEU-4 indicates strong capabilities in generating accurate and contextually appropriate sequences. The high scores in ROUGE-L and SPICE metrics suggest that our model excels in generating linguistically rich and semantically relevant annotations. Despite the r-GRU model’s achievement of the highest METEOR score, our model’s close performance in this metric indicates its robustness and reliability. Similarly, the CIDEr optimization results shown in Table 3 confirm our model’s strong performance across multiple evaluation criteria. Its leading scores in BLEU-1 through BLEU-4 metrics indicate consistent and high-quality output. The model’s top performance in METEOR and ROUGE-L underscores its ability to produce both precise and contextually appropriate annotations. While the Stack-VS model attains the highest CIDEr score, our model’s second-place ranking in this metric and others like SPICE demonstrates its overall superior performance across the board.

These findings strongly underscore the significant improvements our proposed method brings to model performance. The incorporation of the Visual Feature Decoder (VFD) and Visual Feature Visual Attention (VFVA) modules plays a crucial role in this enhancement. By integrating these modules, our model can more effectively harness visual features from input images, which leads to the generation of more detailed and informative descriptions. The VFD module is designed to dynamically identify and extract relevant features from the visual input. This process results in the creation of a new visual matrix that encapsulates the most important aspects of the image. In turn, the newly formed visual matrix is used by the VFVA module, focusing its attention on these critical features. In this way, VFVA updates the contextual data that the language model uses to predict the next word.

This sophisticated interplay between the VFD and VFVA modules significantly boosts model performance on a range of standard evaluation metrics. The VFD module’s ability to distill relevant visual information ensures that the model captures essential details from the input images. Meanwhile, the VFVA module’s attention mechanism allows the model to maintain a contextual understanding of these visual features, thereby enhancing the accuracy and relevance of the generated descriptions. A further advantage of this method is that it is designed to ensure that each word prediction is informed by a comprehensive understanding of the visual context, which in turn contributes to the model’s robust performance. This approach not only improves the model’s descriptive capabilities but also ensures consistency and coherence in the generated text. As a result, the model excels across multiple evaluation metrics, demonstrating superior performance compared to methods that do not leverage such advanced visual feature integration.

We conducted several experiments using MS-COCO dataset, refer to Table 4. In experiment VFDICM_sig2, we employed a sigmoid activation function within the VFD component, instead of the softmax activation function of the VFD. In the language LSTM, we concatenated only two inputs: hta and v~t, removing v^ from the inputs of the language LSTM. In contrast, in experiment VFDICM_soft2, we used the same two inputs in the language LSTM as the VFDICM_sig2 but applied a softmax activation function in the VFD component.

In experiment VFDICM_sig3, we again used a sigmoid activation function within the VFD component. However, in the language LSTM, we concatenated three inputs: the output of the first attention mechanism (v^t), along with hta and v~t. The VFDICM_soft3 experiment is our proposed model which uses softmax activation function in the VFD with three inputs into the language LSTM.

To demonstrate and analyze the significance of the second attention mechanism and the impact of the absence of v~t, we conducted several experiments using MS-COCO dataset, refer to Table 5, where the attention mechanism was replaced with the mean of features (v~t=1k∑i=1kuti).

In experiment VFDICM_sig2_m, we used a sigmoid activation function in the VFD component and two inputs in the language LSTM: hta and the mean of features (v~t). Conversely, in experiment VFDICM_soft2_m, we used the same inputs in the language LSTM but with a softmax activation function.

For experiment VFDICM_sig3_m, we utilized a sigmoid activation function in the VFD component and three inputs in the language LSTM: v^t, hta, and v~t. Similarly, in experiment VFDICM_soft3_m, we used the same inputs in the language LSTM as VFDICM_sig3_m but with a softmax activation function in the VFD component.

To evaluate the impact of selecting the most relevant visual features at every time step (top-k) on VFDICM performance, we performed ablation experiments varying the k parameters using MS-COCO dataset, refer to Table 6. Different versions were compared with top-k values of 1, 10, 15, 20. We observed a significant correlation between parameter k and overall performance. With k = 10, the version outperformed the others with k = 1, 15, and 20. Therefore, we chose the configuration with k = 10 since it yielded the best results for integration.

In addition to conducting quantitative score analysis, we must assess the quality of the captions produced by VFDICM. Fig. 5 displays a selection of sample images from the test dataset, accompanied by their respective captions. Each image in Fig. 5 is associated with two different types of description. First, we describe the image using our proposed model, followed by the ground truth image captions.

For instance, consider the picture located in row one, column one, top left. The caption for our proposed model, “two men are skiing on a snowboard in the snow,” describes the scene in greater detail by identifying “two men.” This caption is very similar to the human-generated caption, “Two men use their snowboards to go down a snowy incline,” demonstrating our model’s capability to generate human-like captions. In another example, look at the image in the first column and first row from the right. According to our model, “a teddy bear sitting on top of a wooden table,” accurately depicts the image. In this caption, the human-annotated ground truth caption is very similar, “A cake shaped as a Teddy Bear on a wooden table.” VFDICM’s performance and quality of generated descriptions remain high, significantly exceeding that of the baseline on standard evaluation metrics, as shown by the scores in Table 3. These examples illustrate the model’s ability to understand and describe various contexts effectively. Moreover, the consistency in generating accurate captions across different images underscores the robustness of our model.

In this work, we have introduced an innovative image captioning model that utilizes the visual characteristics of the input image to produce informative sentences. This model places a strong emphasis on the local visual features of the input image, enabling it to guide the prediction of the next word in the evolving caption. Through the integration of the Visual Feature Detector (VFD) and Visual Feature Visual Attention (VFVA) modules, the model effectively harnesses the visual representations of the input image, leading to a significant enhancement in the performance of the image captioning model.

VFDICM exclusively relies on the visual attributes of the input image, without depending on semantic information such as topics, attributes, or Parts of Speech (PoS). This reliance solely on the visual features empowers the model to exploit the richness of the visual content within the input image, ultimately generating high-quality text. Additionally, the proposed framework operates independently of any external data sources, other than the adopted dataset, allowing the model to concentrate entirely on the available visual data within the image content. As well as its simplified architecture and fewer hyperparameters, our model is a good choice as a baseline for more complicated models and architectures.

Furthermore, the VFDICM model, as proposed, makes a substantial contribution to improving the efficiency of image captioning algorithms and producing high-quality descriptions. This novel approach effectively exploits the visual features of the input image to improve the performance of image captioning models and generate more accurate and informative descriptions. Our model incorporates two additional modules, VFD and VFVA, to facilitate this. The VFD module dynamically selects the most relevant features of the input, creating a new visual matrix that consists of these relevant visual features in the current linguistic context. Meanwhile, the VFVA module attends to these selected visual features and generates a new visual context vector, which is then utilized for generating a new word in the evolving caption.