Deep learning-based image inpainting methods offer notable advantages due to their sophisticated automatic learning and contextual modeling capabilities. Unlike traditional techniques that depend on predefined rules and basic algorithms, deep learning models use advanced neural networks to automatically learn complex patterns, textures, and features from extensive datasets. This capability allows the models to capture extensive visual information without manual intervention. Moreover, these models are proficient in contextual modeling, taking into account not just the nearby area but the entire image context. This comprehensive approach results in inpainting that integrates seamlessly with the surrounding content, ensuring high visual coherence. Consequently, deep learning methods achieve more accurate and realistic inpainting results for damaged images, even in cases of irregular damage and varied textures. Their enhanced ability to deliver visually satisfying results marks a significant improvement over traditional inpainting techniques.

Initially, Pathak et al. [9] attempted to use CNN for image inpainting. Subsequently, in order to propel the progress of deep learning-based image inpainting, researchers have introduced a range of cutting-edge and inventive methodologies. For instance, Liu et al. [10] proposed a probabilistic diversity Generative Adversarial Network called PD-GAN, which introduces random noise and probabilistic sampling to generate high-quality and diverse image inpainting results. Although the introduction of noise can increase diversity, it may sometimes result in inpainting outputs with noise or blurriness, which can impact the quality and authenticity of the inpainting results.

Furthermore, certain approaches integrate attention mechanisms and context modeling into image inpainting networks, elevating their ability to restore crucial regions and intricate details within the image. In conclusion, deep learning-based image inpainting techniques offer an efficient and precise approach to image restoration tasks through the utilization of automatic learning and contextual modeling features of neural networks, along with their ability to manage multi-scale image structures [11].

The Transformer was first introduced by Vaswani et al. [12] in 2017 to address natural language processing (NLP) tasks. Building on the Transformer, Dosovitskiy et al. [13] introduced the Vision Transformer (ViT) in 2020, successfully applying it to computer vision tasks. ViT segments images into a series of patches, treats each patch as an input sequence, and uses the Transformer model for feature extraction and classification.

With the successful application of Transformer, researchers have made further improvements and extensions to it. For example, Liu et al. [14] have opened up new research directions and expanded the application scope of visual Transformer mechanisms by addressing the challenges of processing large-scale images and providing efficient computational methods. Yuan et al. [15] have improved the performance of Transformer networks by adopting more efficient training strategies that leverage the structural information reconstruction of images.

These advancements have resulted in considerable advancements in applying the Transformer mechanism to a broader spectrum of visual tasks. These studies have also demonstrated the powerful potential of the Transformer mechanism in addressing image-related problems. However, despite the Transformer’s advantage in capturing long-range dependencies and generating diverse structures. Nevertheless, the Transformer significantly increases computational complexity, making it challenging to handle high-resolution images.

Recent studies have shown significant performance improvement in computer vision tasks by combining Convolutional Neural Networks (CNNs) and Transformers. CNNs excel at image feature extraction, while Transformers have advantages in handling long-range dependencies and capturing global relationships. CNNs excel at image feature extraction, while Transformers have advantages in handling long-range dependencies and capturing global relationships. Due to the expertise of Transformers in sequence modeling and CNNs in local feature extraction, the dual-branch encoder-decoder applies to various tasks.

For example, recent studies [16–20] have also explored methods for image inpainting, where Transformers are employed to reconstruct complex coherent structures and rough textures, while CNNs enhance local texture details guided by the rough restored images. The effective fusion of global contextual information and local features during the inpainting stage has greatly enhanced the overall results. Such a combination of CNNs and Transformers has achieved significant progress in image inpainting tasks.

Models that combine Convolutional Neural Networks (CNNs) and Transformers showcase exceptional abilities in both feature extraction and sequence modeling. CNNs are effective at capturing intricate spatial details and patterns in image. In contrast, Transformers are adept at modeling sequences and understanding long-range dependencies, which is essential for applications like natural language processing and time-series analysis. By integrating CNNs with Transformers, these hybrid models benefit from the strengths of both approaches: CNNs for detailed spatial feature extraction and Transformers for handling sequential data efficiently. This combination enhances versatility, allowing these models to excel in various deep learning tasks, including the mechanism has superior capabilities to capture the local and nonlocal dependencies on face image in the face reconstruction [21]. Consequently, the fusion of CNN and Transformer architectures is becoming increasingly prevalent and valuable in advancing deep learning technologies.

The parallel dual-branch learnable Transformer network (PDT-Net) proposed by us for image inpainting tasks is illustrated in Fig. 2. In our network, the generator adopts an encoder-decoder architecture, comprising two concurrent encoders. Each encoder utilizes different feature extraction methods and model architectures. The objective of this architecture is to maximize the benefits of distinct encoders, enhancing both the expressive capability and performance of the model.

The features extracted by the dual-branch encoders are denoted as CEn{i}∈RH2(i−1)×W2(i−1)×2(i−1)C′ and TEn{i}∈RH2(i−1)×W2(i−1)×2(i−1)C′, where i represents the number of encoding layers, and i=1,⋯,5. The outputs of these two encoders are fused through the dual-branch fusion module to integrate the feature representations from both branches, resulting in the fused feature of the encoding layers, denoted as FEn{i}∈RH2(i−1)×W2(i−1)×2(i−1)C′. Then, by utilizing full-scale skip connections, we obtain rich fusion multi-scale feature information with global and local feature interactions, represented as the decoding layer feature map FDe{i}∈RH2(i−1)×W2(i−1)×2(i−1)C′. Finally, the image details and textures are restored through the decoder.

We optimize our PDT-Net using a joint loss function L, which includes multiple components: the reconstruction loss ℓ{re}, the adversarial loss ℓ{adv} [23], the perceptual loss ℓ{p} [24], and the style loss ℓ{s} [25]. These loss functions are commonly employed in various image inpainting methods [18] and are defined as shown in Eqs. (7)–(11): (7)ℓ{re}=‖Iout−Igt‖1 (8)ℓadv=EIgt[logD(Igt)]+EIoutlog[1−D(Iout)] (9)ℓp=E[Σi⁡1Ni‖ϕi(Iout)−ϕi(Igt)‖1] (10)ℓs=Ei[‖ϕi(Iout)Tϕi(Iout)−ϕi(Igt)Tϕi(Igt)‖](11)L=λreℓre+λadvℓadv+λpℓp+λsℓs

where D represents the PatchGAN discriminator [26] with spectral normalization. ϕi refers to the i-th layer activation function of the VGG19 [27] network pre-trained on the ImageNet [28] dataset. Ni represents the number of elements in ϕi. λre, λadv, λp, and λs are the weight ratios of the corresponding loss functions. We set λre = 1, λadv = 0. 1, λp = 1, and λs = 250.

As shown in Table 1, the original image dataset consists of three public datasets. Our experiments fully utilized the training and testing sets of the Paris street view dataset, which is composed of a large number of images collected from the streets, for evaluation purposes and strictly followed its original split configuration. For the CelebA and Places2 datasets, we divided the training, validation, and testing sets with a ratio of 8:1:1 for our experiments. The CelebA dataset primarily focuses on human faces and consists of 202,599 face images. We selected 50,000 images from the CelebA dataset for training and 6250 images from the testing set for evaluation, images were selected sequentially from the beginning of the dataset. The Places2 dataset covers various scenes and environments, containing millions of images. We selected twenty scene categories from the Places2 dataset, including attics, airports, arches, campuses, and more. Each category has 5000 training images, making a total of 100,000 training images. We selected 12,500 images from the testing set for evaluation, 5000 images were selected from each relevant subset.

The experiments were conducted on a system running Ubuntu 18.04, equipped with a single NVIDIA A30 Tensor Core GPU that has a memory capacity of 24 GB. For implementing the experiments, we used the PyTorch framework, which is well-suited for deep learning tasks due to its flexibility and efficiency. To optimize the model parameters, we employed the Adam optimizer [29], known for its effectiveness in handling large datasets and complex neural network structures. The Adam optimizer with β1=0.5, β2=0.9 was used to train the model, the learning rate was set to 10−4, and later it was adjusted to 10−5 to fine-tune the model. The choice of hardware and software tools was aimed at ensuring robust performance and accurate results, with the NVIDIA A30 GPU providing the necessary computational power and the PyTorch framework facilitating smooth implementation and experimentation.

To offer a clearer evaluation of our model’s performance in image inpainting, we compared it against five leading image inpainting models across three different datasets. We utilized three well-established metrics: Peak Signal-to-Noise Ratio (PSNR), Structural Similarity Index (SSIM) [30], and the ℓ1 loss function. Higher PSNR and SSIM values indicate better reconstruction quality and visual similarity, while lower ℓ1 loss values suggest better pixel-level accuracy. We ensured consistency in our experimental setup by using the same hardware and software environment for all models, and we maintained a uniform mask coverage rate throughout the experiments. This approach allowed for a fair and comprehensive assessment of our model’s performance relative to its peers.

PC [31]: Introduces a method for repairing irregular holes in images using partial convolution techniques.

The comparative experimental results presented in Table 2 clearly indicate that our model surpasses all baseline models across the three evaluation metrics. This performance improvement is further reflected in the enhanced visual coherence of the inpainted images, as shown in Fig. 6. Additionally, Fig. 6 provides a detailed visual comparison: it features the original corrupted image, the results from PC, RFR, AOT, CTSDG, T-former, our PDT-Net, and the ground truth image. This sequence of images highlights the improved fidelity and visual quality achieved by PDT-Net compared to the other methods.

We performed a series of ablation experiments on the dataset of Paris street view. By individually eliminating the crucial components of our proposed method, we reevaluated the results of inpainting. Net1 represents the removal of the linear attention module in the Transformer encoder, Net2 represents the replacement of the dual-branch fusion module with regular convolutions, and Net3 represents the replacement of the full-scale skip connections with regular skip connections. The experimental outcomes showcased in Table 3 substantiate the vital role and efficacy of the incorporated components within the method. Removing these key components noticeably deteriorated the quality of the inpainting results, further affirming their significance. The comprehensive network model, as depicted in Fig. 7, excels in reinstating intricate texture details and enhancing the coherency of the reconstructed structures.

Through extensive comparative and ablation studies, we have rigorously confirmed the advantages and effectiveness of our proposed approach for image inpainting tasks. Our rigorous analysis of the experimental data reveals that our approach consistently delivers higher quality results and enhanced performance compared to existing methods. Additionally, our method outperforms variants of other approaches where key components have been removed. The proposed PDT-Net has approximately 52.1 million parameters and requires 172.1 GFLOPs, reflecting a reasonable computational cost given its dual-branch architecture.