IE is formed by two main components: the ArcFace model and the Linear Mapping model. The goal of Encoder is to extract identity-specific information from an input 2D image using two components: ArcFace and Linear Mapping. First, the ArcFace model determines the level of similarity between the input image and the reference image of the same identity and then generates a feature vector to capture identity-specific information. Second, the Linear Mapping model maps this feature vector to a higher dimensional space to improve the discriminative power of the feature representation.

ArcFace: The ArcFace architecture (a model developed based on ResNet-100) [31–33] uses a shortcut to map input from previous layers to following layers, thereby avoiding the phenomenon of vanishing gradients [34]. This is important for updating model weights and enabling the model to learn the best features. At the same time, This approach is widely recognized for its effectiveness, as demonstrated by its application in several studies [33,34]. These works highlight the superiority of additive angular margin loss in achieving more discriminative features compared to traditional methods. Moreover, additional research [35,36] has further validated its advantages, particularly in improving recognition accuracy under challenging conditions. This is necessary when the input images have only one color, which makes it extremely difficult to identify the identity differences among the people in the photos. This model is trained with Glint 360k data [37] consisting of about 170 million images of 360 thousand people from 4 different continents. The study uses the knowledge learned from this model to enrich the results of the problem.

In the GD, the MICA model utilizes FLAME [38–40], which is the well-known model for transferring feature information from 2D data vectors to anthropometric morphology of faces. This is a model that has been trained on 33,000 precisely aligned 3D faces. In MICA, this model is further trained on many other datasets including LYHM [41], and Stirling [42]. This extensive training allows the Decoder component to reconstruct important features such as nose shape, size, and face thickness. As previously discussed in the preceding section, the FLAME model is a well-known model for transferring features, and it is employed within the Decoder part of the MICA model. By incorporating incomplete facial data into the transfer learning process, we can capitalize on the robust and well-optimized pre-trained MICA model. This approach is not only suitable for our study but also demonstrates its efficacy in harnessing the full potential of the pre-existing MICA model, thus enhancing the overall performance of the 3D face reconstruction process.

In [30], ArcFace architecture is expanded by a small mapping network ℳ that maps the ArcFace features to their latent space, which can then be interpreted by GD: (1)z=ℳ(ArcFace(I)),where z∈R300 and I is the RGB input image. Research by Zielonka et al. [30] used FLAME as a GD, which consists of a single linear layer: (2)𝒢3DMM(z)=B×z+A,where A∈R3N is the geometry of the average human face and B∈R3N×300 contains the principal components of the 3DMM and N is the number of vertices of the output. Then, the loss function of the MICA model is defined as follows: (3)ℒ=∑(I,𝒢)∈𝒟|kmask(𝒢3DMM(M(ArcFace(I)))−𝒢)|,where 𝒢 is the ground truth mesh, D is the data set and kmask is a region dependent weight.

The geometric loss function is a critical metric in 3D modeling and reconstruction tasks, as it measures the discrepancy between predicted and ground truth models in three-dimensional space. It is commonly defined using the Euclidean distance, which provides a straightforward and intuitive metric for evaluating differences between corresponding points in 3D space [43]. For two points p and q in 3D space, the Euclidean distance is given by: (4)d(p,q)=(p1−q1)2+(p2−q2)2+(p3−q3)2.

In this study, we use the mean of all distances of vertices to compute the geometric loss. Let P and Q be the sets of vertices for two 3D models with n vertices each. The geometric loss function is then computed as: (5)L=1n∑i=1nd(pi,qi),where pi and qi are corresponding vertices in the sets P and Q, respectively. This approach allows us to quantify the overall difference between the two 3D models by averaging the distances between corresponding vertices.

The self-supervised learning algorithm was adopted to enhance the learning ability of the pre-trained Encoder component in our model. The central idea is that training on a vast dataset, followed by introducing a small amount of new data, can significantly alter model weights, resulting in poor performance. The self-supervised learning algorithm, named Swapping Assignments Between Views (SwAV), was developed by Caron et al. [44] and is derived from the contrastive instance learning algorithm [45]. These methods are based on traditional clustering techniques [45,46]. They typically operate offline, alternating between assigning steps to each model cluster. In this approach, image features from the entire dataset are initially clustered, and the assigned clusters are used to predict different views of the image during training. However, clustering-based methods are unsuitable for online learning models due to the extensive steps required to compute the necessary image features for clustering. This self-supervised learning technique can be conceptualized as comparing different views of the same image and the clusters to which their features are assigned. Specifically, the research determines the code from an augmented version of an image sample and predicts this code from other augmented versions of the same image. Given two image features zt and zs from two distinct augmented versions of the same image, their code qt and qs is computed by matching these attributes with a set of K prototypes {c1,c2,…,cK}. Subsequently, the study constructs the swapped prediction step using the loss function as follows: (6)L(zt,zs)=ℓ(zt,qs)+ℓ(zs,qt),where the function L(zt,zs) measures the fit between the attribute zt and qs the code.

To enable the study to function online as well as offline, codes (qt, qs) are computed using only the image attributes within a batch. This approach allows the application of C prototypes across different batches. SwAV employs a clustering method to group multiple versions of the prototypes. Codes are calculated using the C prototypes in such a way that all samples in a specific image batch are evenly distributed among the prototypes. This even distribution ensures each code for different images in the batch is unique, avoiding trivial solutions where every image has the same code. Given the feature vector B consisting of Z=z1,z2,…,zB, this study aims to map them to the prototypes C=c1,c2,…,cK. The mappings are represented as Q=q1,q2,…,qK and Q is optimized to maximize the similarity between the properties and the prototypes:

(9)maxQ∈𝒬Tr⁡(Q⊤C⊤Z)+εH(Q),where H is the entropy function, H(Q)=−∑ijQijlog⁡Qij, and ε is the parameter that controls the smoothness of the mapping. It is observed that strong entropy regularization (i.e., using high ε) can often lead to a trivial solution, where all templates collapse into a unique representation and all prototypes have the same property. Therefore, ε should be fixed at a small value. To ensure equal partitioning, Asano et al. [46] constrained the matrix that belonged to the transportation polytope. To address this issue, we propose an adaptation of the solution proposed by [44], which involves constraining the transportation polytope to the minibatch when working with mini-batches as follows:

(10)𝒬={Q∈R+K×B∣Q1B=1K1K,Q⊤1K=1B1B},where 1K is a vector of ones with K dimensions. These constraints ensure that on average each prototype is selected at least B/K times in a batch. When a continuous solution for Q∗ in Eq. (10) is found, the discrete code can be calculated using the rounding process. In an online setting, we used mini-batches with discrete codes that performed worse than continuous codes. These soft codes Q∗ are the solution of Eq. (9) over the set 𝒬 and can be written as:

(11)Q∗=Diag⁡(u)exp⁡(C⊤Zε)Diag⁡(v),where u and v are the renormalization vectors in RK and RB, respectively. The vector renormalization is calculated using a small number of matrix multiplications and the iterative Sinkhorn-Knopp algorithm [47].

To achieve completion of the wound healing process and effectively aid in concealing facial imperfections, we have employed the filling extraction technique, which was initially developed and discussed in one of our prior publications [29]. This approach is specifically tailored to analyze the wound-affected area on the face, isolate it, and extract it separately for subsequent utilization in 3D printing applications.

By employing this technique, our final printable product is an amalgamation of the original, imperfect face and the separately extracted wound portion. Notably, the extracted part is displayed in a distinct color to differentiate it from the rest of the facial structure, as demonstrated in Fig. 7. The combination of these elements allows for a more comprehensive understanding of the wound area and its impact on the overall facial reconstruction process.

To effectively implement the transfer learning and self-supervised learning processes within the MICA model, we have undertaken a comprehensive data preprocessing approach, which consists of two primary steps. Each step has been carefully designed to ensure the optimization of our model’s performance and applicability in the context of reconstructing incomplete facial structures.

The first step focuses on preparing the input data in a 2D format. This decision is grounded in two main considerations. Firstly, utilizing a 2D input format is consistent with the MICA model’s architectural requirements, thus allowing for seamless integration of the data. Secondly, the adoption of 2D input data presents an alternative method for facial reconstruction, which significantly simplifies the process for medical professionals. As a result, physicians are no longer required to provide 3D MRI (Magnetic Resonance Imaging) scans. They can rely on more readily available 2D images. This adaptation seeks to create a more conducive environment that enables healthcare practitioners to efficiently engage with our model, ultimately delivering enhanced value and outcomes for patients. Additionally, patients can preview their facial appearance after treatment, providing them with a clear understanding of the expected results. The second step of the data pre-processing procedure involves employing augmentation techniques within the mathematical space to standardize the model’s output. This standardization serves as the Ground Truth for the MICA model to calculate the loss value, ensuring that the reconstructed facial structures are accurate and reliable. By integrating these augmentation techniques, we aim to improve the model’s overall performance and facilitate a more precise reconstruction of facial defects. The specific details of the input and output parameters are described in greater detail below:

The 2D input format is obtained by capturing images from all 3D data with varying lighting directions. The 2D images have a resolution of 1024×1024 pixels and are captured from 3687 3D samples within our dataset, ensuring that the angle of capture reveals any scarring present.

To prepare for the training process, the research divides the data into a 6:2:2 ratio for the training, validation, and test sets, respectively. During the data splitting procedure, the study ensures that images of different individuals are allocated to each dataset, preventing the exchange of facial information among three sets.

Runtime: The experiments were conducted on Google Colab Pro. The proposed model requires computational cost and runtime. The first epoch takes about 30 min for the training and validation process, while other epochs take only about 90 s to complete as the data is already stored.

Hyperparameters: In all experiments, this research fixed the version of hyperparameters. The optimal algorithm is Adam [47] whose initial learning rate value is 1×10−3, which is kept constant throughout the training. At the same time, a batch size of 50 images is used for each epoch. Finally, the experimental process is conducted for 50 epochs for each case as shown in Fig. 8.

Fig. 8 indicates the change in the loss function value for each epoch in two cases: 20% data and 100% data. Both cases share a similar pattern of loss function change for the first 30 epochs. The case with 100% data shows faster convergence in the following 20 epochs. This suggests that the present model is highly stable.

We used a self-supervised algorithm for model training: We conducted a total of 5 experiments. In the first experiment, the study used the pre-trained model of MICA [30] with its enriched data. In the remaining cases, we utilized the self-supervised learning technique as mentioned in Section 3 to train the Encoder component before proceeding to the supervised learning process. The present method conducted 50 epochs with self-supervised learning in two main cases. As discussed earlier, our study initially used a sample of 20% of the training data and then used a larger sample to evaluate the loss of the proposed models. Fig. 9 shows the loss of self-supervised algorithm training when sampling 20% and 100% of the data. When approaching the 50th epoch, the loss function of the method with 20% of the data sample performed better than that with 100% of the data sample. This can be explained by the fact that when dealing with larger amounts of data, self-study may have some difficulties in adapting to the new data. The strategy of training the MICA model combined with self-supervised learning is described in Algorithm 1.

During the training process, Fig. 10 shows that the loss function path of the training set of all three cases is stable and all values go below 0.5 at the 10th epoch. After that, the loss line decreased slightly and approached near zero at the 50th epoch. In all cases, the loss lines are not significantly different from each other. For the validation set, the loss lines seem to be volatile at the 10th epoch, and then become stable again as they move toward the 50th epoch.

In our recent research [29], the outcomes of the developed model were assessed using the geometric loss function. So, in this study, we explored the performance of a self-supervised learning model on a unique dataset, comparing it to the well-known transfer learning approach, specifically the Mica model. Using the geometric loss function for assessment, the results demonstrate the effectiveness of our approach in enhancing model performance. As shown in Table 1, the transfer learning method yielded a loss of 0.36, indicating a reasonable level of accuracy in reconstructing facial injuries. However, when applying our self-supervised learning model to just 20% of the training data, we observed a reduction in loss to 0.3, showing an improvement over the transfer learning method. Upon expanding the dataset to 100% of the available data, our self-supervised learning model continued to improve, reducing the loss marginally further. This consistent improvement, even with increased data, showcases the potential of our model in handling larger datasets efficiently.

These findings suggest that employing a limited amount of data in the pre-trained model using the self-supervised learning technique is a feasible approach that offers superior efficiency compared to the transfer learning method. Notably, our datasets comprise only 3687 images, accounting for less than one-tenth of the data in the purely pre-trained MICA model. As a result, our approach effectively leverages the capabilities of the trained MICA model while remaining applicable to the current research context.

Fig. 11 shows some examples illustrating the model’s ability to reconstruct the structure of human facial wounds and the deviation between the output and the reality, with scar types divided into 4 main groups including The top of the head, Forehead, Chin and Nose. The Deviation column describes the difference between the Output column and the Ground Truth column with the unit of measure calculated in units of coordinates, specifically brighter regions signify larger errors, which can be identified as wounds. As shown in Fig. 11, wounds at the forehead position are the most elusive, producing the highest errors, and wounds around the nose area give extremely good results. For a more detailed analysis of the results, we perform a statistic with the output of a total of 291 object files as input including 53 object files are The top of the head, 71 object files are Forehead, 65 object files are Chin and 102 object files are Nose as shown in Table 2. Specifically, the deviations in the coordinate units are converted to % and averaged by the number of each location, with the smallest mean error being the Nose region with 1.2% and the largest average error being the Forehead region with 7.8%, the other two regions are almost similar. This result satisfied the requirements of the Hospital cooperating in the field of plastic surgery, it supports the doctor in most stages of facial regeneration and the doctor’s intervention is only to bring completeness.

Although some errors will inevitably occur while reconstructing injured areas, the proposed model successfully reconstructed the majority of the face, leaving virtually no unaffected regions. These findings suggest that the model is proficient in addressing the challenges of reconstructing defects stemming from wounds, even when solely relying on 2D input. The results of this study have been applied to the creation of experimental products using 3D printing technology, as shown in Fig. 12, to be utilized in laboratory experiments. The plastic material used is PLA (Polylactic Acid), a versatile biopolymer. PLA is easily synthesized from abundant renewable resources and is biodegradable. It has demonstrated significant potential as a biomaterial in various healthcare applications, including tissue engineering and regenerative medicine [48]. However, it is important to note that PLA is not suitable for implantation into human skin. Our 3D printing results are intended to demonstrate its efficiency and to provide an example of its potential real-world applications. Due to the high costs associated with bioprinting technologies that can create implantable materials for humans, we hope to conduct experiments shortly using materials that are compatible with human skin grafting. The 3D printer we use features an enclosed chamber to maintain a stable temperature during the printing process. The printing procedure always begins with cleaning the hot end. We print at a lower speed of about 60 mm/s, and the PLA printing temperature is set at a high 205°C. The flow rate is increased by 5% above normal. The 3D printer is placed on a stable surface, and all components are checked to ensure they do not cause vibrations during printing. Additionally, we use an ADXL345 3-axis accelerometer to compensate for any vibrations and reduce issues with poor interlayer adhesion and rough surfaces. We monitor print quality by visually inspecting every 3D printed sample. If there is significant poor interlayer adhesion or surface roughness, the samples are reprinted. The results of this study reveal significant potential for practical applications, especially in creating models that mask flaws in the human face.

The approach in this study provides relatively high performance in the task of imperfect 3D facial reconstruction. However, the current results may not fully capture the complexities of real-world scenarios due to the significant differences in individual faces. Creating a real-world dataset for this purpose is both challenging and costly. Therefore, in this study, we have adopted an open-solution approach. Our method has been tested on synthetic data, with the expectation that it can be applied to real-world data once available, thereby enhancing the reliability and generalizability of our conclusions.