Existing UIL studies can be categorized into unimodal data-based methods and multimodal data-based methods based on the data modality utilized.

Given that visual and linguistic modalities often convey complementary insights, the synergy achieved through joint multimodal representation learning has demonstrated remarkable efficacy across various tasks, including visual question answering, image captioning, and quotation interpretation.

The CLIP (Contrastive Language-Image Pre-training) model is a multimodal pre-training model proposed by OpenAI, designed to bridge the semantic gap between text and images through contrastive learning. The innovation of CLIP lies in its use of unsupervised learning methods, training on large-scale data pairs of image and text scraped from the Internet, thereby learning cross-modal representations. The CLIP model comprises two main components: an image encoder and a text encoder. The image encoder can be any pre-trained convolutional neural network (CNN), such as ResNet, while the text encoder is typically based on the Transformer architecture. These two encoders map input images and text into the same vector space, allowing for a direct comparison of their similarities.

During training, CLIP utilizes a contrastive loss function, which aims to maximize the similarity of positive pairs while minimizing the similarity of negative pairs. Suppose we have a set of images I and corresponding textual descriptions T, where each image i has a corresponding text description ti. The loss function can be represented as: (1)LCLIP=−logexp(S(i,ti)/τ)∑jexp(S(i,tj)/τ)−logexp(S(ti,i)/τ)∑kexp(S(ti,k)/τ)

Here, S(i,ti) and S(ti,i) denote the similarity score between the image i and its corresponding text description ti, and 𝜏 is the temperature parameter used to scale the similarity scores before applying the softmax function. For each image i, we compute its similarity with all text descriptions, and for each text description ti, we calculate its similarity with all images. Then, we normalize these similarities using the softmax function to obtain probability distribution. The goal of this loss function is to make the similarity of paired image-text descriptions significantly higher compared to non-paired ones. Minimizing this loss function allows the CLIP model to learn meaningful representations in the multimodal space, effectively bridging the semantic gap between text and images.

Building upon CLIP, the BLIP Series Models [13,27] represent an advanced multimodal pre-training architecture, incorporating refined training paradigms and architectural enhancements. Beyond joint representation learning, BLIP emphasizes improvements in text generation rooted in visual understanding, thus achieving a harmonious integration of visual analysis and linguistic expression, marking a significant advance toward comprehensive multimodal intelligence. BLIP2 is an upgraded version of BLIP, further enhancing the model's performance and efficiency while retaining its original strengths. BLIP2 achieves higher cross-modal matching accuracy and faster inference speed through improvements in model architecture and optimized training strategies.

We establish a set of symbols that will be used throughout our discussion to ensure clarity and consistency. Table 1 summarizes the main notations used in this paper.

In this work, we aim to address the problem of UIL based on multimodal UGCs across different social media platforms. Without loss of generality, we focus on UIL between 𝒪1 and 𝒪2 platforms, and we choose the popular social media Twitter and Instagram as 𝒪1 and 𝒪2 platforms for illustration of our scheme. Due to the different service focuses, on Twitter people tend to broadcast news or join topic discussions through text tweets, while on Instagram they tend to post images with short text descriptions to share their daily lives or ideas. To make the model more representative, we specifically assume that users prefer to generate unimodal text posts on 𝒪1 and multimodal paired image and text posts on 𝒪2.

Suppose we have a set of training user account pairs 𝒰={(u11,u21),(u12,u22),⋯,(u1N,u2N)} where each pair consists of two user accounts from different social media platforms (𝒪1 and 𝒪2) are composed of. Also, suppose that for each user account u1i on 𝒪1, we have his/her K unimodal text posts 𝒯i={t1i,t2i,⋯,tKi}. Similarly, for each user account u2i on 𝒪2, we collect his/her G multimodal posts 𝒮i={(v1i,c1i),(v2i,c2i),⋯,(vGi,cGi)}, where vgi and cgi denote the image and text of the i−th post, respectively. The training user account pairs are labeled by y=(y1,y2,⋯,yN)T∈RN, where yi stands for the real situation of the first i pair of users (u1i,u2i). Specifically, yi=1 if accounts u1i and u2i refer to the same identity in the physical world, and yi=0 otherwise. In a sense, our goal is to use labeled pairs of training user accounts to learn the projection f:u1i×u2i→{0,1}.

Diverse social media platforms are characterized by distinct functionalities; for instance, Instagram prioritizes visual content, with user-generated multimodal posts predominantly featuring images accompanied by brief or even absent textual accompaniments. To bolster textual content on such platforms and leverage the copious semantic information embedded in images, we introduce a text augmentation strategy grounded in the BLIP model. This approach taps into the decoder component of a pre-trained BLIP model to generate descriptive captions for individual images within posts. BLIP, a robust bidirectional text-image pre-training model, boasts a decoder capable of translating image substance into coherent natural language narratives. By doing so, it effectively transforms non-verbal image data into textual representations, thereby enabling the interpretation of image content through the lens of linguistic semantics and enriching the overall semantic context.

Specifically, for the G multimodal posts 𝒮i={(v1i,c1i),(v2i,c2i),⋯,(vGi,cGi)} for each user u2i on the 𝒪2 platform, a pre-trained BLIP model based on the pre-training BLIP model first used ResNet to process the input image vgi into a high dimensional feature vector. This process captures the spatial and appearance information of the image. Next, the decoder Transformer receives the image feature vector and generates a sequence of word embeddings to form the final caption dgi. The caption dgi not only captures the specific visual elements in the image such as objects, scenes, and actions, but also expresses deeper contextual information and implicit relationships. The generated caption dgi is paired with the original data (vgi,cgi) to get the extended data pair (vgi,cgi,dgi), which in turn constructs an extended set of posts notated as 𝒮i^={(v1i,c1i,d1i),(v2i,c2i,d2i),⋯,(vGi,cGi,dGi)}. 𝒮i^ contains all the modal information for each post: the image, the original text, and the caption of the image generated by BLIP.

By transforming image information into text, the semantic gap between two different modalities, image and text, can be narrowed, which facilitates the comparison and fusion of data from the two modalities in a unified semantic space, and is conducive to improving the accuracy of cross-modal UIL. Especially on social media platforms, image posts posted by users are often accompanied by short or even no textual descriptions. BLIP-generated captions can effectively supplement this lack of information and provide more valuable information clues for UIL. In addition, image-based caption generation can be viewed as a kind of data augmentation of text, which increases the diversity of data and helps to improve the generalization ability of the subsequent model and the ability to cope with unseen complex situations.

Traditional work tends to use targeted image models and language models to extract image and text features respectively, e.g., extracting 2048-D image feature vectors using the ResNet model and 768-D or 1024-D text features using the BERT model, and then training modal fusion networks or classification networks based on these two features. However, due to the semantic gap between text and images, high quality as well as a high quantity of multimodal data from anchor users is required to achieve the desired training results. Since social data tends to have a lot of noise and anchor users are expensive to acquire, the current results of training fused multimodal features based on unimodal models are not satisfactory.

To address the above problem, we design the alignment feature extraction module to provide a unified feature representation of text and images on social media with the help of a pre-trained cross-modal comparative learning model, CLIP. CLIP employs a comparative learning strategy, the basic idea of which is to maximize the similarity between positive sample pairs (an image and the text that describes it correctly), while minimizing the similarity between negative sample pairs (an image and an irrelevant text description). This approach motivates the model to learn representations that efficiently map images and text into a shared semantic space. The process of extracting features based on the CLLP model can be formalized as Eqs. (2) and (3): (2)Z1={Zv,Zc,Zd}={fimg(v),ftxt(c),ftxt(d)} (3)Z2=Zt=ftxt(t)where fimg and ftxt denote the image encoder and text encoder of CLIP, respectively. Specifically, for the image v of 𝒮i^ in the extended set, the image features are obtained using the image encoder fimage of CLIP, denoted as Zv∈RG×dimg, with dimg being the fimage output image feature’s dimension. For the original text c and image caption d in 𝒮i^, the text features are obtained using CLIP's text encoder ftxt, respectively, denoted as Zc∈RG×dtxt and Zd∈RG×dtxt, with dtxt being the dimension of the ftxt output text feature. Similarly, for text t in the set of 𝒯i, the text encoder of CLIP is also utilized to obtain its text features Zt∈RK×dtxt, which are more comparable between multimodal features Z1 and Z2 than those extracted by the unimodal model. CLIP can capture the unique features of each of the text and image while maintaining cross-modal semantic consistency, so that content expressing the same concept or topic will have similar representations even in different modalities, effectively bridging the semantic gap between text and image and providing high-quality input features for subsequent UIL.

The contents posted on different social media platforms by users are often different, and for the UIL task, some of the posts are extremely similar in content, which possesses a positive impact on the UIL effect, while some of the posts are equivalent to noise, which harms the UIL effect. Considering the different degrees of contribution of each post to end-UIL, we design a lightweight Adapter module to personalize and optimize the features extracted from the CLIP model to further enhance the performance of these features in the UIL task. The Adapter module is typically a pre-trained model that is supplemented by one or more layers of a small network structure, allowing for the targeting of the UIL without changing the parameters of the original model. Adapter modules typically add one or more layers of small network structures to a pre-trained model, allowing fine-tuning of the model's performance for a specific task without changing the original model parameters. In the multimodal UIL scenario, we design the multimodal Adapter module as a four-terminal, multi-headed self-attention module, with each end as shown in Eq. (4): (4)Z^=Adapter(Z)=MultiHead(MLP(Z))where MultiHead stands for Multi-Head Self-Attention Operation, MLP stands for Multilayer Vector Machine Operation, Z∈{Zv,Zc,Zd,Zt}, Zc is the text feature of user u1i, Zv,Zc,Zd are the image feature, text feature, and image caption feature of user u2i, Z^∈{Zv^, Zc^, Zd^, Zt^} which correspond to the outputs of the features in Z after passing through the Adapter module, respectively.

Specifically, we set up the Adapter module for the text end of the 𝒪1 platform, the text end of the 𝒪2 platform, the image end, and the caption end, respectively. First, the features of each end are passed into an MLP network, as shown in Eq. (5): (5)Z′=MLP(Z)=σ(W2⋅σ(W1⋅Z+b1)+b2)

The MLP consists of two fully connected layers, where W1 and W2 are the weight matrices of the first and second layers, respectively, and b1 and b2 are the bias terms of the corresponding layers. The σ(⋅) denotes the activation function. The MLP layer performs more complex nonlinear transformations on the multimodal features extracted by CLIP to fully explore and combine the high-level abstraction relationships among the features.

The feature Z′ output from the MLP network is then passed to the Multihead Self-Attention module, and the Multihead Self-Attention is calculated as shown in Eq. (6): (6)Z^=MultiHead(Z′)=Concat(head1,⋯,headh)WOwhere h denotes the number of heads, headi denotes the output of the i-th head, and WO is the output transformation matrix. The output of each head headi can be calculated by Eq. (7): (7)headi=Attention(Z′WiQ,Z′WiK,Z′WiV)=softmax(Z′WiQ(Z′WiK)Tdk)Z′WiV

Attention is the attention computation function, WiQ, WiK, WiV are the query, key, and value transformation matrices of the i-th head, dk is the dimension of the key vector Z′WiK, softmax normalizes the similarity, the weight of each key vector is computed, and then multiply the weights by the value vectors, and finally perform a weighted summation to get the Attention output.

The Adapter module utilizes a multi-head self-attention mechanism to assign dynamic weights to the features of each modality of each post so that those post features that are more representative and distinguishable in determining the identity of a user will receive higher attention and weighting. The multi-head self-attention mechanism processes the input features in parallel from multiple different perspectives, and each head outputs a weighted subset of feature vectors, and then stitches together the results from the individual heads to ultimately generate a new representation of the features that reflect the importance of each post.

The multimodal Adapter module makes full use of the powerful cross-modal features provided by CLIP and fine-tunes the differentiation of the importance of image features based on the actual application scenarios, which improves the accuracy of UIL, and does not require fine-tuning of the large multimodal model during training, but only needs to train the lightweight Adapter module, which improves the performance of UIL.

In response to the behavioral patterns of users posting content on different social media platforms, it is found that users tend to post content with some kind of intrinsic relevance within the same period. Based on this observation, we construct a multimodal similarity computation framework incorporating a time factor for measuring users' content consistency across platforms.

First, we construct a user similarity matrix based on text posts on the 𝒪1 platform and multimodal posts containing text, images, and captions on the 𝒪2 platform as shown in Eq. (8): (8)sim=[CosSim(Zt^,Zv^),CosSim(Zt^,Zc^),CosSim(Zt^,Zd^)]where CosSim(Zt^,Zv^) is the cosine similarity between the image content of u2j and the text content of u1i, CosSim(Zt^,Zc^) is the cosine similarity between the text content of u2j and u1i, and CosSim(Zt^,Zd^) is the cosine similarity between u2j's caption content and u1i's text content. Eq. (6) measures the content consistency of users between the two platforms.

However, considerations based solely on content similarity are not sufficient to fully capture the temporal nature of user behavior. Given that content posted by users within the same period is more relevant, we introduce a time factor r to adjust the similarity weights. This time factor r is defined as shown in Eq. (9): (9)r={2/(1+eα⋅x−α⋅t1),0<x<t11,t1<x<t2e−β⋅x+β⋅t2,x>t2where x represents the publishing time difference between the two posts, t1 and t2 are the set time difference threshold parameters, and α and β control the magnitude of the change in the decay function. The image of the time decay function is shown in Fig. 3, when the two posts are closer to each other in terms of publishing time, the larger the time factor value is, and the higher the corresponding similarity confidence is. Specifically, when the publishing time difference x between two posts is small enough, i.e., 0<x<t1, we assign a reward value greater than 1 to the similarity between the posts, and as the time difference increases, the time factor gradually decays, but because the time difference is still small at this time, the time factor decays slowly. When the time difference x between two posts is large, i.e., x>t2, we assign a penalty value less than 1 to the similarity between the posts, and as the time difference increases, the time factor gradually decays, and it decays rapidly, converging to 0. When the time difference is within the fuzzy period, i.e., t1<x<t2, we retain the original content similarity, i.e., we set the time factor to 1.

According to Eq. (7), we calculate the time factor matrix R between K textual posts and G multimodal posts, and finally, we combine the time factor-adjusted cross-modal similarity defined as a list of content similarities weighted by the time factor as shown in Eq. (10) as follows: (10)simfinal=[R⊙CosSim(Zt^,Zv^),R⊙CosSim(Zt^,Zc^),R⊙CosSim(Zt^,Zd^)]where ⊙ denotes multiplication by elements. The final prediction is shown in Eq. (11): (11)o=softmax(Wf⋅AvgPooling(simfinal)+bf)

AvgPooling(⋅) performs mean pooling on the final similarity matrix to reduce the dimensionality, thus forming a vector that reflects the overall similarity between user u1 and user u2. Finally, this dimensionally reduced user similarity vector is input to the fully connected layer network for classification, with Wf denoting the weight of the classification layer, bf denoting the bias of the classification layer, and the output of the fully connected layer transformed into a probability distribution by the softmax function. After optimized training, the proposed network can effectively predict whether the user pairs from two platforms are the same person.

The cross-modal similarity calculation method based on the time factor not only effectively addresses the limitations of single-modal similarity calculations but also successfully leverages temporal information to enhance the accuracy and effectiveness of the UIL task. By incorporating the time factor into the similarity calculation, this method provides a more nuanced understanding of the relationship between different modalities, particularly when dealing with unaligned image-text pairs. Algorithm 1 outlines the complete process of the EUIL algorithm.

Dataset. We validate our approach on the TWIN dataset [12] collected by Chen et al. The TWIN dataset collects information about users on two popular heterogeneous social media platforms, Twitter and Instagram. After filtering out some low-quality data, 5765 user pairs were obtained on Twitter and Instagram, along with 1,729,500 UGCs and corresponding timestamps.

Training settings. We divided the user account pairs into three parts: 80% for training, 10% for validation, and 10% for testing. These are considered as positive samples, while negative samples are randomly generated with the same number of positive and negative samples. We converted the UIL into a binary classification task, using accuracy as the evaluation metric. For optimization, we used the Adam optimizer with a learning rate of 0.0001.

Due to the limited research done on the issue of linking user identities in UGCs, we compare our scheme with the following baseline:

WSF-GBDT: This baseline is derived from the method in [28], which introduced four writing-style features, including lexical, syntactic, structural, and content-specific features, to characterize users and employed the support vector machine [29] for the UIL.

Table 2 shows the performance comparison of the different methods. Based on this table, we obtain the following observations: (1) Our proposed method and UserNet are significantly better than WSF-GBDT. this may be attributed to the fact that the former two methods integrate multimodal user representations, not just writing style features. (2) Our proposed method outperforms WHOLE, demonstrating the advantages of fine-grained similarity modeling in the context of user identity correlation, and can further incorporate temporal correlation. (3) Our proposed method significantly outperforms the optimal baseline UserNet. This may be attributed to the fact that UserNet extracts text features using the text-specific unimodal model BiLSTM and image features using the visual unimodal model ResNet, respectively, and computes the similarity matrices directly on top of the two unimodal features, ignoring the semantic gaps between the text and image. semantic gap between text and image. In contrast, our proposed method, which uses the text encoder and image encoder of the multimodal model CLIP to extract features, allows the features of different modalities to be in the same embedding space that can be compared and is more capable of capturing the similarity relationship between image and text. To visually demonstrate the effectiveness of the multimodal model in extracting aligned features on social media, we use a similarity matrix to show the similarity of the image-text features extracted by CLIP, where the lighter the color of the block of elements in the matrix, the higher the image similarity. Where image-text pairs come from heterogeneous multimodal posts made by users on social media, there is often a potential correlation between images and text. As shown in the Fig. 4 below, the features extracted by the CLIP model better capture this correlation.

To gain a deeper understanding of our proposed model, we compare our proposed method EUIL with several derived methods. Specifically, we explore the effects of modal diversity, text enhancement, multimodal Adapter, and time factor on UIL performance.

EUILtext: Only the text modal of the 𝒪2 platform is considered and only the original text posted by the user is taken into account, ignoring the image modal data of the 𝒪2 platform with the text of the generated image description.

Our approach leverages aligned multimodal features and temporal correlation to enhance the performance of the UIL task. By utilizing aligned multimodal features and incorporating temporal information, we have achieved significant improvements. However, despite these advancements, there remain some limitations and areas for potential improvement:

(1) Integration of Multi-Dimensional User Features: EUIL primarily focuses on studying the contribution of multi-modal user-generated content to user alignment. However, different social media platforms emphasize various types of content, which poses significant challenges to the proposed methods, potentially affecting their generalization and robustness. Beyond user-generated content, other user features on social media platforms, such as user profiles and social network structures, may provide crucial clues for enhancing the accuracy of UIL. To build more universally applicable models, future work should consider incorporating a wider range of user features into the model.

(2) Construction of Datasets with Multi-Dimensional User Features: Training and evaluating models using more diverse datasets ensures their generality and robustness. However, due to user privacy protection, publicly available datasets that meet the requirements are limited. Therefore, constructing datasets that include comprehensive features is necessary for future work, including collecting data from multiple social media platforms, and covering information from diverse user groups, thereby enhancing the representativeness and breadth of research. In this way, we can more comprehensively evaluate the model's performance in real-world scenarios, ensuring its adaptability to real-world environments and improving its recognition capabilities in complex situations.

(3) The dynamic nature of social media user behavior and preferences poses significant challenges for UIL. Capturing long-term trends and short-term changes is crucial for maintaining the accuracy and relevance of these systems. In future work, we can pursue research along the following three directions: (1) Longitudinal Tracking Mechanism: We plan to develop a mechanism capable of tracking user behavior over extended periods. This will involve collecting and analyzing historical data to identify patterns and trends in user activities. By doing so, the system can adapt to long-term changes in user behavior. (2) Dynamic Learning Algorithms: We aim to design algorithms that can continuously update user models based on incoming data. This dynamic learning capability will enable the system to quickly adapt to short-term changes in user behavior, such as shifts in interests or sudden changes in activity patterns. (3) Feedback Loop for Model Refinement: We propose implementing a feedback loop to refine user models based on real-time feedback. This could involve engaging users with the system, such as through corrections or verifications of linked identities, to improve the accuracy of the models. We believe that these enhancements will significantly improve the robustness and adaptability of our UIL system. They will enable the system to maintain its performance even as user behavior evolves.

The performance of EUIL on the TWIN dataset indicates that UIL relying solely on UGC across Twitter and Instagram platforms is feasible. Unlike user profiles, which can be disclosed or contain false information, UGC often represents genuine user behavior and is publicly available on social platforms, which makes UGC-based UIL techniques susceptible to malicious exploitation for consolidating sensitive personal information across social media platforms. Therefore, it is necessary to take measures to ensure privacy security. Research on data privacy protection in other fields is relatively advanced [30], but research on user social data privacy protection is relatively insufficient. There are specific UIL adversarial attack methods targeting user attributes, such as the DeLink [31] framework, which draws from adversarial text generation ideas to help users modify their social media usernames, thus defending against UIL. Additionally, there are adversarial attack methods targeting the structure of social networks, such as the TOAK [32] strategy, which leverages the topological structure information of networks by carefully perturbing network structures to reduce the matching accuracy of UIL models. However, there are currently adversarial methods specifically targeting UGC. To prevent UGC from being maliciously utilized, our next steps will explore how to generate adversarial UGC by introducing imperceptible perturbations without disrupting the intended expression of the user, thereby reducing the effectiveness of UGC-based UIL strategies and further protecting user data privacy.