Solutions for image captioning are proposed upon the encoder-decoder in recent years. For example, Vinyals et al. [26] propose the CNN-LSTM architecture to encode the image into features and decode them into a caption. Anderson et al. apply the two-layer Long Short-Term Memory (LSTM) network to concentrate on the weighting stage. These methods all employ the RNN-based decoder, which may lose relevance if the two generating words have a large step interval [23]. Until Google proposes the Transformer [12] which applies the Self-attention to calculate the similarity matrix between vision and language, image captioning is trapped in this issue.

Transformer is still an encoder-decoder framework consisting of the attention and Feed Forward Network. Upon this, some optimized Transformers are proposed to obtain better features by improving the structure of the model. M²-Transformer [22] encodes image regions and their relationships into a multi-layer structure to fuse both shallow and deep relationships. Then, the generation of sentences adopts a multi-level structure by low- and high-level visual relations, which is better than the application of single semantic features. However, M2-Transformer is an optimization of Transformer, it still researches the feature and can’t solve the splitting problem of cross-modal feature conversion for image captioning. To address this issue, X-Transformer [20] focuses on the interaction between image and language by spatial and channel bilinear attention distribution. According to this improvement, X-Transformer achieves excellent performance in 2020. Furthermore, Zhang et al. [19] propose RSTNet to count the contribution of visual features and context features while generating fine-grained captions by novel adaptive attention. Meanwhile, RSTNet is the first to apply the grid features of the image for image captioning and obtain excellent performance. Since 2021, many pre-training methods for image captioning are proposed. For example, Zhang et al. [18] and Li et al. [25] research the visual representation of an image and propose the grid features and pre-training strategy of visual objects features respectively. The pre-training methods apply big data to construct relationships between the visual features and language features and achieve powerful performance for image captioning.

Self-attention is the sub-unit of Transformer, which maps the query, key and value to the output. Moreover, for each input token vi∈Rd, the Self-attention can output a corresponding sequence zi∈Rd which can be computed as follows:

(1)zi=∑j=1n∂i,j(vjWV)

(2)∂i,j=exp⁡(δi,j)∑k=1nexp⁡(δi,k)

(3)δi,j=(viWQ)(vjWK)Tdwhere the projections WQ,WK,WV∈R are trainable matrixes, ∂ is the SoftMax function and δ is the scaled dot-product attention.

The position encoding methods that we discuss are the absence of the image captioning encoder. As we know, the position encoding is initially designed to generate the order of sequence for the embedding token [12] named absolute position encoding, which can be formulated as:

(4)vi=vi+piwhere the pi is the positional encodings and vi∈Rvd. There are several methods to accomplish the encoding such as the sine and cosine functions and the learnable parameters [12,27].

Besides the absolute position encoding, researchers recently reconsider the pairwise relationships between the tokens. Relative position encoding is significant for the tasks that request distance or sequence to measure the association [24,28]. The relative position between tokens vi and vj is encoded to pi,jQ,pi,jK,pi,jV∈Rd and embedded into the Self-attention, which can be defined as:

(5)viQ=viWQ+pi,jQ

(6)vjK=vjWK+pi,jK

(7)vjV=vjWV+pi,jV

Although relative position encoding has been widely applied in object detection, it is hardly employed in image captioning. Considering the semantic information of the relative distance, we believe it can advance the interaction of vision and language.

As the extra information of the objects in an image, position encoding is always ignored for image captioning. To re-weigh the attention by the spatial information, we design a novel learnable spatial grid feature map to improve position encoding, as well as update the Self-attention to adopt it.

Given the objects detected in an image I, we first extract their center point position, height and width of regions as Pi={pi′,pi″,hi,wi}. Then, we design a learnable spatial grid feature map M and set its size to m×m, as shown in Fig. 2, and apply the data process θ to work out the absolute distance encoding diA via the objects’ positions Pi and the M, which can be formulated as:

(8)diA=Mi(θ(pi′),θ(pi″))+Mbias(pi′,pi″,hi,wi)where Mi(,) denotes the position grid feature with indexes (the blue box in Fig. 2) and Mbias is the compensation of the extra region which can’t be covered by a grid (the shaded area in Fig. 2). Finally, we apply the Euclidean distance and linearization to compute relative distance and obtain the relative distance encoding pi,j.

As the spatial information of objects, we refine the Self-attention of the encoder (which is inspired by the contextual mode in [24]) to embed the relative position encodings pi,j, as shown in the encoder in the green box of Fig. 1 and the optimized Self-attention in Fig. 3.

Considering the interaction of visual features of objects v, we regard the relative position encoding pi,j as the bias for the similarity matrix viQ(vjK)T of query and key. Therefore, δ in Eq. (3) can be refined as:

(9)δi,j=(viWpQ)(vjWK)T+(viWpQ)(pi,j)Tdwhere the projection WpQ∈R is trainable matrixes.

Grid Mapping. The reason that Transformer for image captioning abandons the position encoding while encoding is that image is 2D and the regions of objects are not a sequence. To calculate the position on a 2D image and define the absolute distance encoding diA, we propose an undirected mapping method, for the process θ and feature map Mi&Mbias.

Considering that each grid of feature map M possesses a fixed index which can be represented as a 2D sequence, what we should concentrate on is that an object region covers how many grids. Unfortunately, we can hardly map an object region to only a grid region entirely. Thus, we define a parameter Mbias to represent the partial feature map of neighbor grids. According to this issue, we first calculate the corresponding indexes of the covered grids for each of the detected regions and find out the center of the covered grids (the blue box in Fig. 2), which can be formulated as:

(10)θ(pi′)=⌈pi′hIm⌉

(11)θ(pi″)=⌈pi″wIm⌉where hI and wI refer to the height and width of I. Then, we collect the 8-neighbor grids of the computed Mi and the process can be followed as:

(12)Mbias=∑n=18(λnMin)where λn is the intersection between the nth-neighbor grid and Mbias, calculated by Pi={pi′,pi″,hi,wi}. Note that if the 8-neighbor grids may not cover the target entirely, we can calculate the all covered grid features for Mbias, except the center grid, by the approach of concentric circles with different weights. The 8-neighbor is the normal situation of concentric circles. Therefore, we can obtain the absolute position encoding diA.

Relative Position Encoding. The relative position encoding pi,j is determined by the relative distance calculation and the correlation between objects. Note that anyone relative distance isn’t mapped into an integer because the semantic distance between two objects can’t altogether be replaced with position distance. Therefore, we follow Eq. (8) to obtain the two mapped absolute distance encodings diA and djA, and measure their relative distance encoding di,jR. Considering that the order of the words responding objects can’t be predicted while generating, we regard mathematics vectors diA and djA as piA and pjA to measure their Euclidean distance and linearize them into the encoding pi,j influenced by captions:

(13)di,jR=(piA−pjA)2

(14)pi,j=di,jRWRwhere WR∈R is a trainable matrix.

In contrast to the positional encodings, classes are the semantic information of objects. Classes are explained as the two different forms of token: (a) they are the attributes of the visual objects; (b) they are the sources of words in the captions. According to these two characteristics, we propose the Classes Semantic Quantization strategy to quantize the objects-classes word, as well as the Object Classes Awareness network (OCA) to refine the encoder-decoder framework.

Classes Semantic Quantization

The objects-classes are essentially the words wCls. To quantize them and to ensure they hold the same semantics field as the captions, we utilize the words dictionary (Section 4.1) from the dataset to quantize the wCls to word embedding vectors vCls:

(15)vCls=wClsWewhere We is the word embedding matrix. Note that some classes are word groups gCls={wCls1,wCls2,…,wClsk|k∈N∗}, such as “hot dog”, “traffic lights” and “fire hydrant”. If the word group can be represented by the core word, for example the “fire hydrant” is almost equivalent to the word “hydrant”, we will crop the auxiliary word. Besides, if the word group retains the new semantics different from any word, we will add their vectors together. The process can be defined as a piecewise function:

(16)vCls={wClsnWe,∀wClsn≈gCls∑n=1k(wClsnWe),∃wClsn≠gClswhere gCls represents the word group and n∈{1,2,…,k}.

Encoder

For the encoder, we quantize the class words to the vectors vCls and accept them as the tokens which are homologous with the visual features v of detected objects. We refine the encoder by improving the encoder block. The OCA module (the purple box in Fig. 1) is proposed to build the relationship among classes, which is identical to the Self-attention for the v. We apply the extra multi-head Self-attention (MHSA), residual structure and layer-normalization (LayerNorm) to model the semantic relationship of objects, which can be defined as:

(17)znheadCls=SAnhead(vCls,vCls,vCls)

(18)ZheadCls=[z1Cls,z2Cls,…,znheadCls]WE0

(19)ZCls=LayerNorm(vCls+ReLU(ZheadCls))where nhead means the n-th head of the MHSA, the projection WE0∈R is a learnable matrix, the [,] is the concatenation of the vectors, ReLU is the activation function and ZheadCls,ZCls∈Rd are the results of MHSA for classes vectors and LayerNorm respectively. Besides, we have to calculate them without sharing layers because vCls possess a different modality from the v.

In this phase, we propose a fusion strategy to embed class information for the vision encoder, namely OCA_E. We provide ZheadCls for the visual features v to add semantic information and help the model optimize relationships among objectives. As the blue dashed in Fig. 1, we input ZheadCls and v simultaneously. Considering the improved Self-attention with position encoding, we can just renovate the v with ZheadCls, defined as:

(20)v=v+ZheadCls

Decoder

The decoder aims to generate the final captions with the visual feature and class information from the encoder. As shown in the blue box of Fig. 1, we refine the decoder with an additional feature processing module that can embed class information between language features and visual features, namely OCA_D. The refining decoder consists of N blocks each of which can be divided into three modules: (a) Language Masked MHSA Module, which can achieve the interaction in the generated words; (b) Bridge MHSA Module (words-to-classes), which includes an MHSA, the residual connection and a LayerNorm and can be regarded as the interaction between caption words and detected objects names; (c) Cross MHSA Module (classes-to-vision), which contains an MHSA, a feed-forward Network (FFN), the residual connections, the LayerNorms, a linear and a SoftMax function and generates the caption word by word eventually.

Language Masked MHSA Module. We apply this module to build the relations (words-to-words) among the words y1:t−1 that can be represented as:

(21)y∼t−1=LayerNorm(yt−1+MHSA(yt−1WlQ,yt−1WlK,yt−1WlV))where WlQ,WlK,WlV∈R are learnable matrixes and yt−1 indicates the vectors of the word at (t−1)-th step.

Bridge MHSA Module (words-to-classes). Because a detected object itself corresponds to a region and the class of this object is semantic, we propose the structure of words-to-classes-to-vision. This module aims to model the relationship between words y∼t−1 and class features ZCls. Therefore, we construct bridge attention to capture the class context information, which denotes the primary multi-modal interaction to bridge language and vision by classes and can be formulated as:

(22)κt−1=MHSA(y∼t−1WwcQ,ZClsWwcK,ZClsWwcV)

(23)Z∼t−1Cls=LayerNorm(ZCls+κt−1Wwc0)where WwcQ,WwcK,WwcV,Wwc0∈R are learnable matrixes, Z∼t−1Cls∈R(t−1)×d denotes the output of the Bridge MHSA with the y∼t−1 and is exploited as the input of Cross MHSA Module (classes-to-vision).

Cross MHSA Module (classes-to-vision). This module aims at modeling the relationship between the attended classes Z∼Cls and visual features Z, which refers to another multi-modal crossing to bridge language and vision by MHSA with classes and vision. The process can be given by:

(24)λt−1=MHSA(Z∼t−1ClsWcvQ,ZWcvK,ZWcvV)

(25)Z∼t−1=LayerNorm(Z+λt−1Wcv0)

(26)yt−1cv=LayerNorm(Z∼t−1+FFN(Z∼t−1))where WcvQ,WcvK,WcvV,Wcv0∈R are learned parameters, Z∼t−1Cls from the former module is input into MHSA as query, and visual features Z which are composed of position information from the encoder are fed into MHSA as key and value.

The distribution of the vocabulary is as follows:

(27)p(yt|y1:t−1)=Softmax(yt−1cvWy)where Wy∈R is a learnable matrix.

Train by Cross-Entropy Loss. First, we train our model by the Cross-Entropy Loss L_XE:

(28)LXE(θ)=−∑t=1Tlog⁡(pθ(yt∗|y1:t−1∗))where y1:T∗ represents the ground truth.

Optimize by CIDEr Score. Then, we employ Self-Critical Sequence Training (SCST) [29] to optimize:

(29)LRL(θ)=−Ey1:T∼pθ[r(y1:T)]where the reward r(.) is computed by CIDEr (Consensus-based Image Description Evaluation) [30]. The gradient can be defined as:

(30)∇θLRL(θ)≈−(r(y1:Ts)−r(y^1:T))∇θlog⁡pθ(y1:Ts)where ys refers to the result of sampled probability and the y^ means the result of the greedy algorithm.

We apply the MSCOCO dataset [31] to conduct experiments. The dataset has 123287 images (82783 for training and 40775 for validation) with 5 captions for each. We adopt the Karpathy split [32] to obtain the training set, the validation set and the testing set. Besides, we collect the words that occur more than 4 times in the training set and get a dictionary containing 10369 words. The metrics of BLEU (Bilingual Evaluation Understudy) [33], CIDEr [30], METEOR (Metric for Evaluation of Translation with Explicit ORdering) [34], SPICE (Semantic Propositional Image Caption Evaluation) [35] and ROUGE-L (Recall-Oriented Understudy for Gisting Evaluation) [36] are applied to evaluate our method. We compute these metrics with the public code from the MSCOCO dataset.

Differing from the image grid features, we demand accurate object classes and position information for our framework. Therefore, we exploit the Objects365 [37], MSCOCO [31], OpenImages [38] and Visual Genome [39] datasets to train the Faster-Rcnn model [7] for extracting objects features, and merge their classes to obtain a label list with more than 1800 classes, which is similar to VinVL [18]. These objects’ visual vectors are extracted in 2048-dimension and transformed into 512-dimension vectors to match the embedding size. The number of block N is set to 6. With Cross-Entropy Loss, we adopt the learning rate of 4e-4 decayed 0.8 every 2 epochs and ADAM during the total 20 epochs. While training with CIDEr Score Optimization in another 30 epochs, we set the learning rate to 4e-5 and decay it by 50%. Furthermore, the batch size is 10 and the beam size is 2.

We report the performances of the other methods and our method in Table 1. The compared methods include Show&tell (LSTM) [26], SCST [29], RFNet [11], UpDown [14], AoANet [40], Pos-aware [41], M²-Transformer [22], X-Transformer [20], RSTNet [19] and PureT [42]. These methods are operated with LSTM or Transformer.

We adopt the strategy of pre-training for the visual feature in VinVL [18] and Transformer [12] as our baseline. Therefore, our baseline achieves good scores because of the great pre-training of detection which also provides accurate position and class information for the proposed method.

For stability, we first present the results of a single model in Table 1. Our models with XE Loss and SCST training are both superior to others. With the XE Loss training, our single model with different terms (OCA_E and OCA_D) achieves the highest scores in all metrics, especially the CIDEr score which obtains advancement of over 1% to the X-Transformer and AoANet. With the SCST training, our models also achieve the best comprehensive performance. While comparing with the strong competitors M²-Transformer, X-Transformer and RSTNet, our two models are superior to them in all terms of metrics, especially the CIDEr score improved by over 2%. Besides, the BLEU-4 score of our methods reach 41.2% and 41.6% which achieve improvements of 0.3% and 0.7% to the latest PureT, respectively. Meanwhile, our methods surpass PrueT in terms of all metrics except METEOR.

In addition, we report the results of the ensemble of four models with SCST in Table 2. Our method also achieves excellent performance and advances the M²-Transformer and RSTNet by more than 6% in terms of CIDEr. Furthermore, our method and PureT are about equal in performance, as outlined in the case of the single model. We also present some generated captions in Table 3 to demonstrate the performance of our approach.

We conduct ablative experiments to understand the influence of each module in our model.

Influence of GMPE. To quantify the influence of GMPE in the refined encoder, we conduct experiments with different modules. We adopt 6 blocks of encoder and decoder and set the size of grid feature m×m to 16×16. Note that we adopt the Transformer as our baseline in row 1. In Table 4, we evaluate the performance of GMPE in three combinations including the baseline with GMPE (row 2), OCA_E with GMPE (row 5) and OCA_D with GMPE (row 6). As we can see, combining baseline and GMPE can achieve improvements of 1.4% in terms of the CIDEr score to the pure baseline. Furthermore, GMPE can increase the CIDEr score of pure OCA_D from 136.7% to 138.3% and improve the performance of OCA_E from 137.1% to 137.8%.

Influence of OCA_E and OCA_D. To better understand the influence of OCA_E and OCA_D in encoder and decoder respectively, we conduct several experiments to evaluate them. Note that the number of block N is set to 6 and m×m is set to 16×16. In Table 4, we present the results of OCA_E in rows 3, 5 and 7, as well as the performance of OCA_D in rows 4, 6 and 7. It can be seen that the baseline with only OCA_E or OCA_D (row 3 and row 4) can achieve improvements of 1.6% and 1.2% in terms of CIDEr score, respectively. Besides, the CIDEr score of OCA_E and OCA_D combined with GMPE (row 5 and row 6) can reach 137.8% and 138.3%, which achieve improvements of 0.9% and 1.4% to the baseline with GMPE (row 2), respectively. However, while combining OCA_E and OCA_D, we obtain a poor record (row 7) resulting from too much specific class information which can fragment the generated captions.

Influence of the Number of Blocks. We fine-tune the number of refining encoder-decoder blocks N and the size of the grid feature m×m. Note that we adopt the baseline while experimenting on N and the baseline with GMPE for m×m. As shown in Table 5, the baseline has a continuous improvement in the CIDEr score with the gradual increase of the value N. Besides, the baseline model tends to be stable when N is set to 6. Therefore, we set N to 6 as the final configuration. The baseline with GMPE also gets a significant improvement in all metrics and reaches peak performance while 16×16. Nevertheless, we still believe in the effectiveness of the other sizes and don’t suggest setting the size lower than 11×11, because the small size can result in too many objects in one grid and lose the advantage of GMPE.