We use BLIP as the backbone of our frozen pre-trained model. BLIP features a multimodal hybrid encoder-decoder structure (MED), unifying image-text matching and generation tasks. It employs a Vision Transformer (ViT) [40] as the image encoder and BERT [18] as the text encoder, with different components activated depending on the downstream multimodal task. For image-text matching tasks, cross-attention is added to the text encoder to fuse image and text features, and a special token [Encode] is prepended to the input text. For generation tasks, a [Decoder] token is inserted at the beginning of the input text, and the bidirectional self-attention layers are replaced with causal self-attention to generate captions for the given image.

LoRA utilizes low-rank matrices to approximate the weight changes during fine-tuning, effectively reducing the number of required parameters. Formally, for a pre-trained weight matrix W0∈R(m×n), the weight update can be constrained as a low-rank matrix decomposition, as shown in Eq. (2). During fine-tuning, the original weights W0 remain frozen, and only the low-rank matrices are updated via gradient descent. Due to the low-rank nature, the dimension r is typically small, making the size of the low-rank matrices significantly smaller than that of the original parameter matrix, where A∈R(r×n) and B∈R(m×r), and r≪min(m,n). While LoRA offers an effective solution for efficient parameter transfer, it has not been further explored in the context of multimodal models.

In the process of fine-tuning multimodal models, a factor that hinders modal alignment is that the multimodality fusion network is deep, which can lead to the loss of query information. To solve this problem, we designed a query information augmentation pipeline that runs parallel to the fusion network for adapted feature aggregation and information supplement.

Formally, as shown in Fig. 2, given the multimodality fusion branch network consists of N transformer blocks, the forward process can be expressed as xF=bN(bN−1(...b1(xT,xV),xV),xV), where bN represents the N-th transformer block, xV and xT represent the image features and the text features, respectively, and xF represents the fusion features. We apply VCRA to aggregate the fusion information from fusion branch network. Denote wi=ΛbBΛaA as the weight matrix accounting for the i-th block, with A and B are the shared random matrices, and Λa and Λa are the scaling vectors. The query-augmentation side network gradually collects information from each block: (5)hi+1=hi+xiFwiwhere hi+1 is the output of the h-th layer of QASN. At the same time, we will supplement the query information flowing in QSAN back into the fusion network: (6)xi+1F=bi(xiF,xV)+f(hi+1)

We did not take the query representation h as the residual of the fusion representation xF, but instead train an information filter to explicitly model the contribution of supplementary query information. Our filter design is simple and effective, and can be represented as f(x)=σ(α⊙x+β). Here, α and β represent learnable parameter matrices, which are initialized to 1 and 0, respectively. Finally, the filtered information is processed through the activation function σ. As a result, multimodality fusion branch network with supplementary query information is more conducive to transfer knowledge for downstream tasks.

In addition to using unsupervised contrastive learning during pre-training, BLIP also employs it during downstream task fine-tuning. As we know, with sufficient training samples, the contrastive loss function effectively brings positive pairs closer and pushes negative pairs farther apart. However, the scale of downstream task fine-tuning datasets is typically limited, which allows individual noisy samples to interfere with the feature representation process, potentially disrupting the previously aligned feature space. To address this issue, we introduce MCAM (Fig. 3), which captures fine-grained relationships between samples through rank consistency, mitigating the impact of noisy samples.

Using contrastive learning loss to align image and text features lacks modeling the degree of repulsion for negative samples. For example, pushing an picture of a dog ’equally’ away from the descriptions ”cat”, ”tiger”, and ”house” can lead to the loss of fine-grained similarity information between the image and text and affecting feature alignment. We introduce the Multimodal Consistency Alignment Module to solve this problem by matching the consistency of similarity rankings between image-text pairs.

Formally, for a mini-batch of image-text pairs denoted as (xV,xT), where xV represents the image and xT represents the corresponding text, image and text are processed through the visual encoder f(⋅) and text encoder g(⋅), respectively, to obtain modality-specific representations. BLIP uses LITC as the contrastive loss: (7)LITC=−∑i=1Nlog⁡eϕ(f(xiV),g(xiT))/τ1∑j=1Neϕ(f(xiV),g(xjT))/τ1−∑i=1Nlog⁡eϕ(g(xiT),f(xiV))/τ1∑j=1Neϕ(g(xiT),f(xjV))/τ1 (8)ϕ(f(xV),g(xT))=f(xV)⊤g(xT)‖f(xV)‖⋅‖g(xT)‖where τ1 is a temperature hyperparameter. While LITC is effective at distinguishing between positive and negative image-text pairs, it does not consider differences between highly relevant and moderately relevant pairs. Our proposed multimodal consistency alignment module introduces ranking information to capture fine-grained image-text relationships, enhancing modality representation and strengthening cross-modal alignment.

Specifically, for a given image-text pair within a mini-batch (xiV,xiT), we can obtain a list S(xiV)={ϕ(f(xiV),g(xjT))}j=1N that represents the cosine similarity between the image xiV and all texts in the batch, as well as a list S(xiT)={ϕ(g(xiT),f(xjV))}j=1N that represents the cosine similarity between the text xiT and all images in the batch. We aim for the corresponding elements in S(xiV) and S(xiT) to have the same ranking positions. This allows us to capture the fine-grained ranking information between the image and various negative text samples, as well as between the text and different negative image samples. We achieve ranking consistency by minimizing the the Jensen-Shannon (JS) divergence of the two top one probability distributions: (9)LITR=∑i=1NJS(S~τ1(xiV)||S~τ1(xiT))=12∑i=1N(S~τ1(xiV)log⁡(2S~τ1(xiV)S~τ1(xiV)+S~τ1(xiT))+S~τ1(xiT)log⁡(2S~τ1(xiT)S~τ1(xiV)+S~τ1(xiT)))where S~τ1(xiV)=softmax(S(xiV)/τ) and S~τ1(xiT)=softmax(S(xiT)/τ) represent the top-one probability distributions of S(xiV) and S(xiT), respectively, and τ is a hyperparameter. The final loss function is: (10)Lall=LITC+LITR

Tables 2 and 3 show the performances of UniTrans for image-text retrieval task on Flickr30K and MSCOCO. As shown, UniTrans achieves performance comparable to UniAdapter with only 0.2M parameters on both Flickr30K and MSCOCO, even outperforming it on certain metrics. Additionally, our method’s performance is very close to that of fully fine-tuning the BLIP backbone while exceeding previous fully fine-tuned methods.

Unlike image-text retrieval, the VQA task requires multimodal generation capabilities. Therefore, in addition to the encoder, a decoder is necessary for text generation. Due to the structural differences between the encoder and decoder, we did not share the random matrices parameters, resulting in a slight increase in trainable parameters. However, the total parameter count remains significantly lower than the baseline. According to Table 4, our method outperforms all fine-tuning methods, demonstrating that our approach is well-suited for multimodal generation tasks.

Table 5 shows the performance of our method on the Image Captioning task using the COCO caption dataset. The results indicate that UniTrans outperforms the baseline and is only slightly below the fully fine-tuned BLIP method. As shown in Table 6, our UniTrans also outperforms the baseline on video datasets.

From the comparative experimental results on multimodal benchmarks, it can be observed that Unitrans achieves competitive performance while reducing the trainable parameters of LoRA to just 5%. This is attributable to two main factors. First, we leverage VeRA to share a low-rank random matrix across modalities and employ scaling vectors to adapt weight updates. This shared mechanism reduces the number of large random matrices involved in training, effectively decreasing parameter size while enhancing inter-modal information exchange. Second, QASN adaptively supplements query information within the fusion network, and MCAM regularizes fine-grained image-text alignment during fine-tuning through contrastive learning loss, strengthening multimodal alignment. In comparison, UniAdapter enables modality interaction during fine-tuning by employing a shared MLP in the adapter. However, it lacks information sharing across layers, and the MLP itself involves a considerable number of trainable parameters, particularly as the rank r increases. Based on this, we extend the concept of UniAdapter to Low-Rank Adaptation and further optimize it by introducing QASN and MCAM to facilitate multimodal alignment during fine-tuning. As a result, we achieve performance comparable to UniAdapter while significantly reducing the number of trainable parameters.

As shown in Table 7, we present the training time and GPU memory costs for the three tasks: Image Captioning, Image-Text Retrieval, and Visual Question Answering. We consider the training time and storage cost of fully fine-tuning BLIP as one unit. From the table, it can be observed that our UniTrans outperforms both fully fine-tuned BLIP and UniAdapter in terms of training time and GPU memory cost.

We also explored the impact of rank size on UniTrans. As shown in Fig. 4a,b, experiments conducted on Flickr30K indicate that as the rank increases, performance of cross-modal downstream fine-tuning improves. However, when the rank reaches a certain threshold, the performance gains slow down or even slightly decrease. This may be due to the rank increase making the random matrix too large, introducing redundant parameters and leading to overfitting. As shown in Fig. 4c, as the rank R increases, the growth in parameters for our UniTrans is significantly slower compared to LoRA and UniAdapter, demonstrating the superiority of our method.

To intuitively understand the performance of UniTrans in multimodal downstream tasks, we present detailed visualization results in Figs. 5–7. Fig. 5 shows a comparison between the captions generated by UniTrans on the COCO Caption dataset and the ground-truth. It can be seen that UniTrans is capable of accurately generating image captions, with key words closely matching the ground-truth. Fig. 6 demonstrates the results of the Image-Text Retrieval task on the Flickr30K dataset, where UniTrans accurately retrieves the corresponding image given a textual caption. We also present the results of the VQA task on the VQAv2 dataset, where UniTrans provides accurate answers. However, the responses are somewhat brief, which may be attributed to the relatively smaller parameter size of the multimodal base model. In the future, we plan to conduct experiments with larger-scale models.

To assess the effectiveness of each module in UniTrans, we conducted ablation experiments on image-text retrieval using the Flickr30K benchmark and the DiDemo benchmark. We compared UniTrans with three baselines: a frozen BLIP, fully fine-tuned BLIP, and LoRA. As shown in Tables 8 and 9, when the random matrices are not shared (‘+scaling vectors’ means only scaling vector is used, and random matrices are shared in each transformer block of single modality), the number of trainable parameters is significantly reduced, but the performance deteriorates. However, our proposed VCRA demonstrates clear improvements, highlighting its contribution to modality alignment. The results also indicate that the QASN and the MCAM, both designed to enhance modality alignment, further boost performance. Notably, the latter achieves performance gains without introducing additional parameters.