We represent the input image as X∈RH×W×3, with H and W representing its height and width, respectively. The image X undergoes an initial preprocessing step to transform it into a one-dimensional sequence suitable for input into a Transformer encoder. For feature extraction, a convolutional layer with a kernel size of p×p and a stride of p is employed. This operation effectively partitions the image into N blocks of size p×p. The number of blocks N is determined by the formula: N=Hp×Wp. Following this, the image patches are flattened, and each vectorized patch is mapped to a latent d-dimensional embedding space (set to 1024 in this study) using a linear projection layer E, resulting in a one-dimensional sequence of image patch embeddings Sp.

To facilitate long-range modeling, encoding the spatial information of image patches is essential. Consequently, a learnable position embedding Posi with identical dimensions is incorporated into the embedding sequence S′∈R(N+1)×d. This process can be expressed by the following equation: (1)Si′=Spi+Posi

Subsequently, the embedded sequence S′ is inputted into the global Transformer encoder. The Transformer encoder is constructed with 24 stacked Transformer encoding blocks, each comprising multiple layers of MSAs and multiple layers of multi-layer perceptions (MLPs). The ultimate output produced by the global Transformer encoder is referred to as Z24, while the features produced by each stacked layer of the Transformer during the encoding process are represented as {Z1, Z2,⋯,Z24}, as depicted in Fig. 1.

The model presented in this paper adheres to the encoder-decoder framework, utilizing the Transformer architecture for feature encoding in the encoder component. However, the extracted features encompass both similar backgrounds and forged regions. To mitigate interference from features of similar backgrounds, a common approach involves dividing the Transformer features into several sets of feature blocks, computing self-correlation scores for each set, and selecting the top k feature blocks with the highest scores. This method aims to reduce interference caused by similar backgrounds [9,15]. However, this empirical approach demonstrates sensitivity to variations in image size.

Considering the inherent nature of image copy-move tampering detection, which involves identifying nearly identical forged targets based on the original source, a one-to-one matching result is preferable to a one-to-many matching outcome. To address this, the proposed model integrates a one-to-one feature matching module and utilizes a multi-scale context decoder to validate the method’s efficacy.

The feature matching module integrates the contextual information of each channel in the Transformer features through global average pooling, thereby reducing the feature map’s dimensionality from K×w×h to K×1×1. Subsequently, the channel information weights are calculated using one-dimensional convolution, and the Sigmoid activation function is applied to constrain these weights within the range of (0–1). Lastly, the feature weight information is derived through vector multiplication with the original Transformer features, yielding the aggregated information of different channel weights.

The stacked architecture of the Transformer encoder enables each image patch’s feature representation to incorporate information from other patches, allowing different encoding layers to capture forgery features at various levels of granularity. To integrate these multi-granularity forgery features, we have designed a CMFDTR-MLW decoder. As illustrated in Fig. 2, the CMFDTR-MLW decoder implements a unidirectional feature fusion strategy, facilitating information integration through a top-down pathway. This approach effectively enhances the flow of information within the Transformer encoder, thereby improving the model’s ability to perceive and synthesize multi-level features. Specifically, we divide the 24 Transformer encoder blocks evenly into four groups and extract the embedded features (Z6,Z12,Z18,Z24) from the final block of each group as input. These features are then reshaped into 3D features (Z6′,Z12′,Z18′,Z24′) with dimensions Hp×Wp×C. Subsequently, these reshaped features are processed through a feature matching module before being input into the CMFDTR-MLW decoder.

Within the MLW decoder, the high-level feature maps are combined with the feature information of the sub-high-level feature maps through element-wise addition. The resulting aggregated feature map is then added element-wise to the next lower-level feature map. The fusion process maintains a copy of the highest layer feature map Z24′ (denoted as P24), which is subsequently added to the next highest layer feature map. This result is then added to the next feature map. After each addition, the resulting feature map is preserved (i.e., P18, P12, P6). Subsequently, the four fused feature maps undergo processing through two successive 3×3 convolutions to halve the number of channels and reduce it to 3, as well as two 4x up-sampling operations. This process yields four candidate three-class tampering masks (R6, R12, R18, R24) with different hierarchical information and dimensions matching the original image. By obtaining four feature maps with distinct mixed conditions through this approach, each of the four feature maps is decoded separately to produce four decoding results. Finally, a union of the four decoding results is taken to enhance the detection accuracy of tampered regions.

In the context of image copy-move tampering detection and localization, the model proposed in this paper conducts binary classification at the pixel level and utilizes the binary cross-entropy loss function (BCELoss) for network updates. The model’s prediction mask is guided by pixel labels from a ground-truth mask, which corresponds to the original image’s dimensions. In this mask, pixels labeled as 0 are classified as original, and those labeled as 1 are classified as tampered. The calculation formula for BCELoss is presented as follows: (2)BCELoss=−(ylog(γ)+(1−y)log⁡(1−γ))where y denotes the label value of the ground-truth mask of the image.

The loss for the primary task of the proposed model is denoted as decode_loss. Furthermore, the model includes an auxiliary decoding head designed to extract outputs from various layers of the Transformer encoder. These extracted results are processed through a fusion module and a decoder, and the auxiliary loss is computed by comparing the predicted mask with the ground-truth mask, represented as au_lossi, where i denotes the i-th encoding block layer. Previous research has shown that incorporating auxiliary losses can enhance model training convergence [28]. The total loss Loss of the model is calculated by combining the main task loss decode_loss with the auxiliary loss au_lossi. The formula for computing the total loss Loss is presented as follows: (3)Loss=decode_loss+∑au_lossi

This study incorporated corresponding auxiliary decoding heads at the Z10,Z15,Z20, and Z24 layer of the Transformer encoder.

The MLW decoder ultimately generates four candidate three-category tampering masks (Z6,Z12,Z18,Z24), each containing hierarchical information and matching the original image in size, by integrating the Z6,Z12,Z18,and Z24 features from the Transformer encoder. These masks are collectively incorporated into the loss function during model training, where they undergo a weighted summation to obtain a composite cross-entropy. This approach enables more effective model training through back-propagation. The weighted summation process is expressed by the following formula: (4)decodeloss=αlossR6+βlossR12+γlossR18+δlossR24where α, β, γ, and δ are coefficients representing the proportional contribution of the four loss functions to the total decoder loss. In this study, these coefficients are assigned values of 0.1, 0.2, 0.3, and 0.4, respectively.

Datasets: The performance of our model is evaluated on four widely-recognized datasets in the field of image tamper detection: USCISI [14], CASIAv2.0 [29], DEFACTO [30], and COVERAGE [31]. These datasets not only include images but also provide corresponding ground truth masks that distinguish source, target, and background areas. The source, target, and background areas are denoted in green, red, and blue, respectively. Table 1 presents the specific characteristics of these datasets.

The USCISI dataset, introduced by Wu et al. [14], is a synthetic compilation of digital image tampering instances focusing on copy-move forgeries. It comprises 100,000 samples, each associated with a binary classification mask that distinguishes between untampered and tampered areas for CMFD. In this study’s experimental phase, 80,000 samples were randomly extracted from the USCISI dataset for training purposes, while 10,000 samples were allocated for validation, and an additional 10,000 samples were reserved for testing, adhering to an 8:1:1 division ratio.

The CASIA v2.0 dataset [29] comprises 5123 digitally manipulated images, categorized into two types of forgery: splicing and copy-move tampering. This dataset serves as a valuable resource for research in digital image forgery detection. Wu et al. [2] conducted a manual verification of 1313 copy-move forged images within the CASIA V2.0 dataset and generated corresponding binary classification masks, establishing the CASIA-CMFD dataset. In the experimental phase of this study, all samples from the CASIA-CMFD dataset are utilized as the test set to evaluate the efficacy of the proposed model.

DEFACTO [30] is a comprehensive dataset that employs public objects from a contextual database to generate semantically meaningful counterfeit images automatically. This dataset encompasses three categories of forged images: spliced forgeries, copy-paste forgeries, and repair forgeries. From this dataset, we meticulously verified and selected 7057 images containing accurately labeled copy-move tampered images. To enhance the precision of the annotations, we reprocessed the corresponding binary masks for these images, thereby establishing the DEFACTO-CMFD dataset. In the experiments conducted for this paper, the entire DEFACTO-CMFD dataset serves as the test set to evaluate the model proposed herein.

COVERAGE [31] is a dataset specifically designed for CMFD in digital images, consisting of 100 images. The dataset employs a technique of superimposing similar objects onto original authentic images, presenting a significant challenge to human visual recognition. The alterations are subtle and difficult to detect without meticulous examination, as the forged objects are seamlessly integrated. The tampered elements in the dataset encompass a diverse range of items, including merchandise, fruits, furniture, and signage. The intricacy of the forgery details poses a substantial challenge to the generalization capabilities of various copy-move tampering detection models. In this study’s experimental phase, all instances from the COVERAGE dataset serve as the test set to assess the performance of the proposed model.

Model parameters: The parameters of the Transformer encoder were configured to match those of ViT_Large, as detailed in Table 2. Additionally, batch normalization was implemented as the normalization method for each decoding head.

For pre-training, this study utilizes the weights of vit_large_patch16_384 pre-trained on ImageNet, provided by the MMSegmentation [32] project, to initialize the image preprocessing and Transformer encoder modules. The decoder module, in contrast, is randomly initialized.

Training strategy: For the training data, we implement the following standard data preprocessing techniques from MM-Segmentation: (1) Random scaling with a ratio range of 0.5 to 2.0. (2) Randomly cropped to achieve dimensions of 256 × 256 pixels. (3) Random horizontal flipping. (4) Photometric distortion. (5) Image normalization.

Training parameters: The training parameters are standardized across all models. The batch size is consistently set to 8. We utilize the SGD optimizer, setting the momentum and weight decay parameters to 0.9 and 0, respectively. The initial learning rate is established at 1e-3. For learning rate adjustment, we implement a polynomial decay strategy, setting the polynomial power to 0.9 and the minimum learning rate to 1e-4.

Test index: Choosing suitable evaluation metrics is essential for accurately gauging model performance in experimental studies. To quantify localization and other performance aspects, this study employs the most widely used evaluation metrics in the CMFD field, as established in previous literature [26]. Precision measures the ratio of correctly identified positive instances to the total instances predicted as positive. Recall represents the proportion of actual positive instances correctly identified by the model out of all actual positive instances. By merging precision and recall, the F1 score delivers an all-encompassing measure of model effectiveness.

To effectively assess the text model’s capability in distinguishing and locating source/target regions, we evaluate the precision, recall, and F-score for each model across three distinct categories: background, source, and target.

Table 3 presents the experimental results of various decoder variants of the text model on the USCISI test set. The variants include CMFDTR-MLW (multi-layer weighting scheme), CMFDTR-Naïve (one-step up-sampling), and CMFDTR-PUP (progressive up-sampling scheme). In the table, data presented in bold text indicate the best performance of the corresponding experimental indicators in the comparative experiments.

The CMFDTR-Naive decoder utilizes a single-step up-sampling process. This process involves applying a 3 × 3 convolution to the feature map Z24, followed by a 16-fold up-sampling and batch normalization. In contrast, the CMFDTR-PUP decoder employs a step-by-step up-sampling strategy. This approach processes the feature map Z24 through sequential 3 × 3 convolutions, gradual up-sampling, and batch normalization. Each up-sampling step doubles the size of the feature map from the previous step, requiring four operations to transform the feature map from size H/256×W/256 to full resolution.

The experimental findings demonstrate that the Transformer encoder exhibits adaptability to various designed decoders, achieving comparable performance in source/target distinction tasks.

Comparative experiments were performed to demonstrate that the proposed model outperforms current state-of-the-art methods in CMFD. The model was evaluated against four specialized CMFD models: BusterNet, DOA-GAN, CMSDNet, and MVSS-Net. These comparative experiments were performed using four publicly available CMFD datasets: DEFACTO CMFD, USCISI, CASIA CMFD and COVERAGE. This comprehensive approach aimed to thoroughly test and compare the performance of the proposed model across diverse datasets in the Copy-Move forgery domain.

The USCISI dataset was employed by training the model on its training set and directly evaluating it on the test set. In contrast, for the DEFACTO CMFD, CASIA CMFD, and COVERAGE datasets, all samples were utilized as the test set for evaluation without any fine-tuning. This methodology enables a more comprehensive assessment of the model’s generalization capabilities.

The pixel-wise localization performance of the compared models on the USCISI test set are shown in Table 4. In the table, data presented in bold text indicate the best performance of the corresponding experimental indicators in the comparative experiments. Owing to the substantial similarity in data distribution between the USCISI test set and the training data, each model has achieved satisfactory three-class localization performance. The results indicate that the CMFDTR proposed in this study outperforms BusterNet and CMSDNet on most metrics in the USCISI dataset, although it slightly underperforms compared to DOA-GAN and MVSS-Net. Notably, the USCISI dataset contains relatively few complex tampered samples. Most tampered samples in this dataset involve simple manipulations where source regions are scaled by a certain factor, subjected to minor random rotations, and then copied and moved to target areas. The DOA-GAN model, employing an adversarial training mechanism through the competitive process between the generator and the discriminator, captures the data’s distributional characteristics more effectively. Consequently, DOA-GAN exhibits superior detection performance on the USC-ISI dataset, where the data distribution is largely consistent. However, its performance in detecting complex tampered regions may decline, indicating limited generalization capability. MVSS-Net jointly utilizes tampering boundary artifacts and noise views of the input image to extract semantic-agnostic features, better capturing lower-level features. It enhances detection specificity through multi-scale supervision, at the cost of reduced detection sensitivity, which is compensated for through multi-view feature learning. While MVSS-Net demonstrates excellent detection capabilities on datasets with highly consistent data distribution, its generalization performance may be relatively poor when faced with data that diverges from the training data distribution. By analyzing the detection results from the CASIA CMFD, DEFACTO CMFD, and COVERAGE datasets, it is evident that the CMFDTR’s detection metrics generally surpass those of DOA-GAN and MVSS-Net, demonstrating that the proposed CMFDTR exhibits stronger generalization capabilities compared to DOA-GAN and MVSS-Net. As shown in Fig. 3, we compare the prediction results on the UISICI dataset using various methods, including BusterNet, DOA-GAN, CMSDNet, MVSS-Net, and our proposed method.

Table 5 displays the pixel-wise localization results for the models evaluated using the CASIA CMFD test set. In the table, data presented in bold text indicate the best performance of the corresponding experimental indicators in the comparative experiments. The dataset incorporates samples with Copy-Move forgeries in highly similar or semantically ambiguous background regions, as well as instances where copied and pasted regions overlap. In contrast, the USCISI dataset seldom contains such intricate tampered samples. Consequently, this disparity may result in suboptimal performance when a model trained exclusively on the USCISI training set is evaluated on the CASIA CMFD test set.

The results of the experiments confirm that the CMFDTR proposed herein surpasses other models in most performance metrics when evaluated on the CASIA CMFD dataset, surpassing the comparison models BusterNet, CMSDNet, DOA-GAN, and MVSS-Net. This indicates enhanced generalization capability and improved performance in detecting tampered regions. As shown in Fig. 4, we compare the prediction results on the CASIA CMFD test set using various methods, including BusterNet, DOA-GAN, CMSDNet, MVSS-Net, and our proposed method.

The pixel-wise localization performance for each benchmark model on the DEFACTO CMFD test set is detailed in Table 6. In the table, data presented in bold text indicate the best performance of the corresponding experimental indicators in the comparative experiments. Importantly, the DEFACTO CMFD dataset exhibits substantial differences in data distribution compared to the USCISI dataset, including more complex samples that are challenging for human visual perception, smaller target samples, and potentially misleading instances. Additionally, the DEFACTO CMFD dataset is more extensive than the CASIA CMFD dataset. The experimental results indicate that the CMFDTR proposed in this study outperforms the compared models (BusterNet, CMSDNet, DOA-GAN, and MVSS-Net) on the DEFACTO CMFD dataset for the majority of evaluation metrics. Notably, the recall is significantly higher than other comparison methods, suggesting that the proposed method, through a one-to-one matching strategy, effectively filters out incorrect matching features. This demonstrates that the CMFDTR possesses stronger generalization capabilities and superior performance in detecting tampered regions. As shown in Fig. 5, we compare the prediction results on the DEFACTO CMFD test set using various methods, including BusterNet, DOA-GAN, CMSDNet, MVSS-Net, and our proposed method.

Table 7 summarizes the pixel-wise localization performance of each comparative model evaluated on the COVERAGE test set. In the table, data presented in bold text indicate the best performance of the corresponding experimental indicators in the comparative experiments. The forged images in the COVERAGE dataset are created by overlaying similar objects onto the original real images. This dataset poses a significant challenge to human visual recognition due to its finely detailed forgeries, and it is particularly demanding for image tamper detection and localization models to perform effectively. The experimental results demonstrate that all detection metrics of the proposed CMFDTR in this study surpass those of the compared models BusterNet, CMSDNet, and MVSS-Net, but are slightly lower than the DOA-GAN model. Analysis of the visualization results suggests that the marginally lower detection performance of the method proposed in this study compared to the DOA-GAN model is attributable to the misclassification of a portion of the forged source and target regions. However, considering the comprehensive detection results across the four test sets utilized in this study, it is evident that the proposed CMFDTR exhibits stronger generalization capabilities and superior performance in detecting tampered regions for unknown tampered images. As shown in Fig. 6, we compare the prediction results on the COVERAGE test set using various methods, including BusterNet, DOA-GAN, CMSDNet, MVSS-Net, and our proposed method.