The forward diffusion process is defined as a Markov chain, where Gaussian noise is continuously added to successive nodes to obtain noisy samples, which in turn gradually transforms the Gaussian noise distribution into a distribution of data on which the generative model is trained. Specifically, as shown in Fig. 1, given a data sample x0∼xN, where x0 is the original image and xN is the image corresponding to the Nth moment of added noise.

The forward additive noise t-moment process is defined as: (1)xt=αtxt−1+1−αtε,αt=1−βt

where xt is the added noise data at moment t, t∈{0,1,2,⋯,T}, T is the number of times the noise is added, is the noise added at moment t, obeys Gaussian distribution, αt is the initialized value, which is the empirical value, βt increases linearly from 0.0001 to 0.002 during forward diffusion, ε is the noise, obeys Gaussian distribution. In the forward diffusion process, the later the moment, the more closely the noise data is related to the noise increased in the previous moment. According to the Markov chain, the state xt−1 is denoted as: (2)xt−1=αt−1xt−2+1−αt−1ε (3)xt=αt(αt−1xt−2+1−αt−1ε)+1−αtε

where ε∼N(0,I) can also be expressed as: (4)xt−1=αtαt−1xt−2+1−αtαt−1εfurther the relationship between xt and x0 is obtained as: (5)xt=α¯tx0+1−α¯tε.

where α¯t=∏i=1tαi.

The forward diffusion process results in a data that nearly obeys a Gaussian distribution, and the inverse diffusion process recovers the original data from Gaussian noise, generating original images for learning a parameterized posterior distribution pθ(x0|xt) through xt. Assuming that the inverse process q(xt−1|xt) is obtained, it is possible to gradually reduce an image through random noise xt. The DDPM uses the neural network pθ(x0|xt) to fit the inverse process q(xt−1|xt), with the formula: (6)q(xt−1|xt,x0)=N(xt−1|μ¯(xt,x0),β~tI)can be deduced: (7)pθ(xt−1|xt)=N(xt−1|μθ(xt,t),∑θ(xt,t))

where β~t=1−α¯t−11−α¯t⋅βt, solve the equation by Bayesian formula: (8)μ¯t(xt,x0)=αt(1−α¯t−1)1−α¯t−1xt+α¯t−1βt1−α¯tx0

αt and α¯t depend only on βt and it follows from the forward diffusion to express x0 as: (9)x0=1α¯t(xt−1−α¯tε),ε∼N(0,I)by combining Eqs. (7) and (8), a mean value, that depends only on xt, results: (10)μ¯t(xt)=1αt(xt−βt1−α¯tε)thus, a neural network εθ(xt,t) can be used to approximate ε and obtain the average of the following: (11)μ¯t(xt,t)=1αt(xt−βt1−α¯tεθ(xt,t))

where ε is the noise value predicted by the trained model. Actually, to predict the noise more accurately, a noise prediction network εθ(xt,t) is used to learn E[ε|xt] by minimizing the objective function minEt,x0,ε‖ε−εθ(xt,t)‖2, where t is uniformly distributed between 1 and T. To learn the conditional diffusion model, this paper further inputs the conditional information c into the noise prediction objective function: (12)minEt,x0,c,ε‖ε−εθ(xt,t,c)‖2

The image recovered after the noise value is obtained using the noise estimation training model in this paper, as shown in Fig. 2.

To make the model more effective, this paper conducts a systematic empirical study on its key elements and conducts ablation experiments on this paper’s dataset, evaluating the FID scores of 1 K generated samples every 5 K training iterations, and selecting the optimal effect throughthe experiments.

We investigate various strategies for integrating the long skip branch within our Transformer architecture. Specifically, we consider the main branch embedding hm and the long skip branch embedding and the long skip branch embedding hs, both of which reside in RH×W. Prior to passing these embeddings to the subsequent Transformer block, we explore five fusion approaches: (1) concatenating hm and hs followed by a linear projection (Linear (Concat(hm, hs))), (2) direct element-wise summation (hm+hs), (3) applying a linear projection to hs prior to summation (hm + Linear(hs)), (4) summing the embeddings and then applying a linear projection (Linear(hm+hs)), and (5) a baseline scenario without the long skip branch for comparative analysis. Notably, the direct summation of hm and hs (i.e., hm+hs) solely modulates the contribution of hs in a linear fashion, leaving the fundamental network architecture unaltered. In contrast, all alternative fusion strategies involving the long skip branch demonstrate enhanced performance compared to the absence of such a connection. As shown in Fig. 4a, the approach that concatenates hm and hs followed by a linear projection emerges as the most effective, suggesting that this method is particularly adept at leveraging the complementary information from both branches.

In investigating the enhancement of our model, we evaluated two methodologies for integrating an additional convolutional layer following the Transformer block: (1) The first method involved appending a 3 × 3 convolutional block following the linear projection step, which converts token embeddings into image patches, as illustrated in Fig. 4b. (2) Before linear projection, adding a 3 × 3 convolutional layer, and the one-dimensional sequence of label embeddings needs to be rearranged into two-dimensional feature map of dimensions H/P × W/P × D, with P represents the size of the patches. (3) Additionally, comparisons were made for the situation in which no additional convolutional layers were added. According to the results in Fig. 4b, the method of adding 3 × 3 convolutional layers following the linear transformation exhibits marginally superior performance to the remainingtwo alternatives.

To incorporate temporal information into the network, we evaluate two different methods for inputting the time variable t: (1) Consider it as a marker, as shown in Fig. 4c. (2) Merge the layer-normalized times into the Transformer block [15], analogous to the adaptive group normalization employed within U-Net [16]. Another method uses adaptive layer normalization (AdaLN). Use the normalization operation: AdaLN(h,y)=ysLayerNorm(h)+yb, within the Transformer block, we embed the input as h, and subsequently derive ys and yb through a linear projection of the temporal embedding. Despite the relative simplicity of the AdaLN approach, the first approach, which treats time as a marker, outperforms AdaLN as shown in Fig. 4c.

We delve into the nuances of patch embedding by examining two distinct variants. (1) The original approach employs a linear projection to transform each patch into a labeled embedding, as illustrated in Fig. 4d. (2) We employed a sequence of 3 × 3 convolutional blocks, followed by a 1 × 1 convolutional block, to map images into token embeddings. In contrast, the results indicate that the conventional patch embedding method outperforms this approach.

We delve into the realm of positional embedding variants, exploring two distinct methodologies tailored for our image restoration framework: (1) Firstly, we adopt the ubiquitous one-dimensional learnable positional embedding, which is the default setting used in this paper and proposed in the original ViT, (2) The second variant utilizes a 2-dimensional sinusoidal positional embedding, constructed by concatenating the sinusoidal embeddings of coordinates i and j for each patch located at (i, j). According to the results in Fig. 4e, the former performs better than the latter, and it is found that the model is unable to generate meaningful images after trying it without any positional embedding, which proves that positional information is crucial in image generation.

We demonstrate the scalability of our proposed model through a meticulous investigation encompassing its depth, quantified by the number of layers, its width, defined by the hidden layer size D, and the granularity of its input, characterized by the patch size. As shown in Fig. 5, the best performance was achieved at a depth of 14 in the 5 K iterations of the experiment, which shows that the depth is not positively correlated with the performance of the model, i.e., the model does not benefit from a greater depth. Analogously, augmenting the width dimension, specifically the hidden size, from 256 to 512 yields a noticeable performance enhancement. However, further escalation to 768 does not yield any discernible gains, indicating a saturation point in the model’s capacity to leverage additional width for improved performance. On the other hand, a smaller patch size consistently improves the performance. In contrast, high-level tasks (e.g., classification) may require larger patches. In practice, due to the high dimensionality of the image data, there may be an increase in cost when using smaller patch values for training, and therefore it is recommended to downscale the image data before using the model.

To better accomplish the repair task based on the diffusion model, in this paper, we refine our diffusion model’s learning process by incorporating two distinct objective functions. The primary objective function implements a straightforward denoising loss, which is computed given a reference output image x and a randomly selected time step t, the reference image with a noisy version is generated as follows: (13)x′=αtx+1−αtεtake T to be 500. We train our conditional diffusion model to faithfully reconstruct the reference image x under the influence of the conditional feature c and the time step t as follows: (14)ℒsimple=Et,x0,c,ε[‖ε−εθ(xt,t,c)‖2]based on the enhanced denoising diffusion model, we further train the network to predict the variance ∑θ(x′,c,t), which not only improves the fidelity of the reconstructed image, but also helps to improve its log-likelihood. The conditional diffusion model additionally outputs the interpolated coefficients s for each dimension and converts the output to variance as follows: (15)∑θ(x′,c,t)=exp(vlog⁡βt+(1−s)log⁡β~t)

where βt and β~ denote the upper and lower limits of the variance, respectively. The second objective function directly optimizes the KL scatter between the estimated distribution pθ(xt−1|xt,x0) and the diffusion posterior q(xt−1|xt,x0) with the following formula: (16)ℒvlb=KL(pθ(xt−1|xt,x0)‖q(xt−1|xt,x0))

The total loss function is the weighted sum of the two objective functions, formulated as follows: (17)ℒ=ηℒsimple+λℒvlb

where η and λ are the weight parameters of the balanced loss function, the improved loss function improves the performance of the network model, accelerates the convergence speed of the algorithm, which improves the efficiency of training when η=0.4, λ=0.6 is adjusted through experiments.

Due to the scarcity of cultural relics mural data set, this paper selects the training data set from the official website of Dunhuang Research Institute and the official website of Shanxi Museum to provide 4000 images of cultural relics mural with different resolution synthesized into a training dataset, through data augmentation to ultimately obtain 10,000 cultural relics images with varying resolution. We firstly manually screened 4000 images with different resolutions, eliminated images with too much single color and too much irrelevant content, and then augmented the images to generate a large dataset of 10,000 images of cultural relics.

We used 2 types of subjective and objective evaluations to validate the method. Subjective evaluation is done by observing the texture and color information of the generated image, objective methods are evaluated by peak signal-to-noise ratio and structural similarity (SSIM), peak signal-to-noise ratio (PSNR) and Fréchet Inception Distance (FID) evaluating the strengths and weaknesses of each algorithm. PSNR mainly estimates the noise fidelity of the reconstructed image, the higher the value, the better the quality of the reconstructed image. SSIM combines three factors: brightness, contrast, and structure. The mean is used as an estimate of brightness, the standard deviation as an estimate of contrast, and the covariance as an estimate of structural similarity. The value range is between [0, 1]. The closer the result is to 1, the better the reconstructed image quality is. FID calculates the similarity between advanced features of the image, and the smaller the value, the higher the degree of similarity.

For objective comparison of restoration results of image restoration methods, the comparison methods in this paper use the same input data. The following experiment is the method of this paper with the hierarchical Transformer-based image restoration method [17]; Based on generative adversarial networks to generate high quality restored images by matching and correlating background patches method Contextual Attention [18]; Shift-Net, a deep learning method that combines a priori information [12]; Global Uniform and Local Continuity (GU&LC) combining global uniformity and local continuity based on the relationship implied between linear systems and image restoration [19] and a comparison of the probabilistic diverse GAN method PD-GAN [20] for image repair on irregularly corrupted images of this dataset.