Keeping pace with the development of deep neural networks, neural style transfer is active research topic. It redefines the style of a content image by referencing to a style image to achieve an artistic appearance, thus resembling artwork with an aesthetic appeal. AST emerged with an optimization method [22] based on pretrained networks from an upstream task. However, this method requires time-consuming iterative optimization for every style transfer image. To address this problem, various methods have been devised. In [44], perceptual losses are used for real-time style transfer. In [47], a hierarchical deep convolutional neural network is proposed for fast style transfer. In [25], patch-based style transfer for fast inference is introduced. In [48], generalized style transfer is achieved without compromising the visual quality with unseen styles. An unprecedented method is proposed in [26] to transfer the style using AdaIN in a single feedforward step for real-time style transfer. AdaIN enables fast and simple style transfer and has become a common and essential operation in style transfer. Hence, various studies have been focused on solving other important problems such as view or video style transfer [16,38,39,49,50], three-dimensional style transfer [19,20,36,37], text-driven style transfer [4,7,51,52], domain generalization [28,53], and domain enhancement [29,30]. More recently, various architecture including GANs [54], attention module [55] or CLIP [56] are leveraged for the controllability [57], fast style transfer [58], applications [59].

Despite its high quality and applications, AST results may not be perceptually satisfactory in terms of user experience. To further improve AST, various methods implemented something-aware schemes to generate perceptually intuitive results. In [41], structure-aware style transfer with kernel prediction is proposed to enhance the structure style. In [42], visual fidelity is improved by using vector quantization with codebooks proposed in discrete image modeling [60,61]. For a relatively sparse shape or structure in an image, object shapes in an RGB (red-green-blue) image are considered with distance transform and patch matching modules for shape-aware style transfer. Structure guidance is used in [40] to focus on local regions. It enhances the content structure to avoid blurry or distorted alignment by AdaIN. Recently, a pattern-aware scheme to discover the sweet spot of local and global style representations with pattern repeatability has been developed [62]. Previous studies have demonstrated intuitive outcomes using something-aware strategies for AST. However, user experience faces a common limitation. Existing AST methods generally fail to deliver the intended outcome if the user transforms the input style image, while the output image is expected to reflect any changes in the input images. To explore this limitation, we perform a pre-analysis of image transformation in Section 3.

Before AST, autoencoders should be trained to perform reconstruction. As in [26], we adopt VGG-19 [23] as encoder E, which was pretrained on the MSCOCO dataset [24] and used to process the style image, and learnable decoder D with existing content and style loss functions. Unlike the method in [26], we do not use a shared encoder for the content and style images to avoid quality degradation that occurs by mapping the content and style images to the latent space using the same encoder [42].

As in [42], we first trained the content and style autoencoders for reconstruction on the MSCOCO [24] and WikiArt²2

https://www.wikiart.org/ (accessed on 24 September 2024).

After pretraining, each encoder appropriately maps the corresponding image while considering the image characteristics. Among encoder and decoder for each image, we only fix the encoder for finetuning the decoder, as detailed in Section 4.3. During inference, we fix all the parameters in the encoders and decoder, except for the user-defined parameters.

We use the encoders for inference or finetuning by freezing the encoder weights. In detail, given input image pair (Ic,Is), we generate style-transferred output image I~. Considering a transformation, each input image I∗ is transformed into I∗,J where ∗ is c or s, and J represents the transformation (i.e., rotation r, translations tx and ty, flipping along the horizontal fh and vertical fv, or zooming z) described in Section 5.1. The image is mapped onto the latent space by the encoder as follows:

(4)zc=Ec(Ic,J~),zs=Es(Is,J~),where J~ is a user-defined image transformation level, z∗∈RCf×Hf×Wf is the latent code (Cf is the number of channels, and Hf and Wf are the height and width of the encoded features), and Ec and Es represent the pretrained content and style encoders, respectively. Then, the latent code is spatially divided into patches as follows:

(5)𝒞=P(zc,Mh,Mw),𝒮=P(zc,Mh,Mw),with P(z∗,Mh,Mw) being the patching layer formulated as:(6)P(z∗,Mh,Mw)=[CH(CH(z∗,Mh,h−axis),w−axis)],where CH is a chunk function. Hence, P divides latent code z∗ into Mh×Mw patches, obtaining sets 𝒞 and 𝒮 of patches extracted from the content and style images, respectively. These sets can be represented as 𝒞={ci}i=1Mh×Mw, 𝒮={si}i=1Mh×Mw where ci, si∈RCf×HM×WM are the i-th content and style patches, respectively, HM and WM are the dimensions of the divided patch calculated as HM=Hf/Mh, WM=Wf/Mw. We assume Mh=Mw throughout the paper unless stated otherwise. AdaIN is applied to each patch with the corresponding positional patch in the other image calculated as:

(7)𝒯=AdaINpatch(𝒞,𝒮),t=AdaIN(zc,zs),where t is the AdaIN output for code blending and 𝒯 is AdaIN output set 𝒯={ti}i=1Mh×Mw calculated by AdaINpatch(𝒞,𝒮). As illustrated in Fig. 3, this operation can be formulated as:

(8)AdaINpatch(𝒞,𝒮)={ti|ti=σ(si)ci−μ(ci)σ(ci)+μ(si)}i=1M,where ci∈𝒞 and si∈𝒮. After applying AdaIN in the latent space, all the patches are aggregated and then blended using global feature t and local features 𝒯 by user-defined parameter α as a global-local weight as follows:

(9)z^α=(1−α)A(𝒯)+αt,where z^α is the final feature including user-defined factor α that sets the tradeoff between the global and local features. A(⋅) is the de-patching layer that aggregates the input patch set while ensuring that A(P(z∗,Mh,Mw))=z∗. During inference, the user can set the content image as the original one, that is, Ic,J=Ic, while transforming the style image. Finally, output image I~α is obtained as I~α=Ds(z^α).

The proposed Patchified AdaIN component can be easily used by replacing each AdaIN layer in an existing network without any other architecture modification or additional training or adaptation process. However, to enhance the regional quality of style transfer output, the decoder can be further tuned with an additional local loss using an existing loss function. To distinguish the loss functions, we refer to the existing AdaIN loss as the global loss throughout the paper. Similar to the global loss, our local loss is additionally computed from every patch of the content and style images. The local content loss is calculated as:

(10)ℒpatch_content=∑i=1M‖𝒯−P(Ec(Ds(A(𝒯)))‖2,where we omit the summation index for simplicity. Note that 𝒯 and the output of P are sets including the patchified latent feature. In the local loss term for finetuning, we exclude code blending (i.e., α=0 in 9). Thus, z^α becomes A(𝒯) without considering the global features. Therefore, the local style loss can be calculated as:

(11)ℒpatch_style=∑i=1M(∑j=1L‖μ(ϕj(Ds(A(𝒯))))−μ(ϕj(Is))‖2+‖σ(ϕj(Ds(A(𝒯))))−σ(ϕj(Is))‖2),where ϕj(⋅) represents the j-th layer in Es(⋅) and L is the depth of the layer from ϕ1(⋅) in the encoder to be used as a loss. We calculate the content loss using relu4_1 and style loss as in [26]. We add the patch loss to the existing global loss [26]. Finally, the patch and global loss functions are formulated as follows:

(12)ℒpatch=ℒpatch_content+λpatchℒpatch_style,ℒglobal=ℒglobal_content+λglobalℒglobal_style,where λ is a style loss weight. We set every λ to 10 as in [26]. ℒglobal_content and ℒglobal_style are the same as the original loss functions of [26]. The complete loss, ℒtotal, is obtained by linearly combining the patch and global loss functions as follows:

(13)ℒtotal=γℒpatch+(1−γ)ℒglobal,where γ is the patch loss scale. By changing γ, we can control the weight of the local spatial fidelity for finetuning. We set γ to 0.9. For plausible visual fidelity without explicit control, γ can be considered as a learnable parameter with an initial value of 1.0.

Additionally, we set Mh and Mw to 2 and obtain each patch by dividing the features, as shown in Fig. 4. Note that the user can control these factors (e.g., patch division strategy, number of patches), as analyzed in Section 5.5. Overall, the patch loss captures the AdaIN loss from local features si and ci to obtain local statistics. Then, with the existing AdaIN loss [26], we weight the patch loss to set the contribution of the local style features to every patch. This affects not only backpropagation of the cost function during training but also inference for resulting image with AST, as analyzed in Section 5.4.

We adopt the AdaIN network in [26] for the AST network as the baseline, but any other network relying on AdaIN can be used in our scheme by simply replacing AdaIN with the proposed Patchified AdaIN. For the encoder-decoder architecture, we use a nine-layer 3 × 3 filter convolution encoder f with up to 512 channels and decoder g with a symmetric architecture and the encoder based on [26]. The output image, I~, is calculated as I~=Ds((1−α)A(𝒯)+αt) for inference and I~=Ds(A(𝒯)) for finetuning.

To evaluate the capabilities of spatial-transformation-aware AST, we choose various geometric transformations that notably changed the regional color distribution. We applied four geometric transformations: rotation angle r, translations along the x axis, tx, and y axis, ty, flipping along the vertical, fv, and horizontal, fh, and zooming z. To prevent the global statistics of the image from excessively changing after a geometric transformation, we filled out empty areas with fictitious pixel insertion by, for example, reflecting symmetrical tiles for the rotation. Four cases were evaluated, with the transformation being given by J=(r,tx,ty,fv,fh,z), where r is a clockwise rotation angle from −π to π, tx and ty are translation factors normalized from −1 to 1, fv, and fh, are flipping indicators being true or false, and z is a scaling and cropping factor from 0 to 0.5 with respect to the original image resolution. Examples of transformations are shown in Fig. 5.

By applying predefined transformations, we performed AST using an original content image and transformed style image. For qualitative comparison, we selected AdaIN [26] as the baseline as well as LinearStyleTransfer (LST) [31], style-attentional network (SANET) [65], IEContraAST (IEST) [66], adaptive attention normalization (AdaAttN) [67], adaptive convolutions (AdaConv) [41], contrastive coherence preserving loss (CCPL) [30] as comparison networks. Because we focused on AST, we did not evaluate domain-enhanced (i.e., domain-specific) style transfer [29] and visual fidelity (whose results resemble image-to-image translation) [42]. As shown in Fig. 6, our method reflected the image transformations with varying color distributions in the output images. As expected from the pre-analysis (Section 3), no existing method could reflect the appearance change in the transformed style imaged, thus providing very similar results regardless of the applied transformation. On the other hand, the proposed method provided different results that reflected the transformations. For instance, our method generated an output image by locally reflecting the sky region (blue) in the style image in terms of spatial color distribution in the first row of Fig. 6. Similarly, the teddy bear showed varying red shades depending on the style image transformations in the fourth row, thus reflecting the variability provided by the proposed Patchified AdaIN.

The proposed Patchified AdaIN applies AdaIN to every spatially divided patch in the latent space. Nevertheless, it may generate images with a non-satisfactory appearance because the normalized latent code may be misaligned during decoding. To quantitatively evaluate this aspect, we measured the spatial color distribution according to the same patch division in the output image and used the same models as in Section 5.1 for comparison. The proposed method was finetuned using Eq. (13) for a value of a=0.0 and a=0.5 in 9. Here, we adopt two metrics: (a) spatial statistics (mean and variance), (b) Jensen-Shannon Divergence [68]. We used 1000 samples between the input style image and each output image per model. Statistical evaluation allows us to intuitively compare the distribution of pixel values. Meanwhile, JSD measure the difference of given two distributions, here image serve as a distribution. Concretely, we found that recent paper [68] leveraged Jensen-Shannon Divergence to measure the color distribution between two RGB images, we also adopt JSD metric in our quantitative evaluation. We used same images in statistical and JSD evaluation.

The quantitative results in Fig. 7 showed that our methods provided the lowest distance for the mean, indicating that our style transfer results had similar RGB mean with the style image among the evaluated models. On the other hand, the distance for the standard deviation was similar. Meanwhile, similar to statistical evaluation results (Fig. 7), JSD evaluation result showed that our method has best performance on reflecting spatial color distribution as shown Table 2. To determine whether this style statistics affected AST, we performed an evaluation considering human perception, as reported in Section 5.3.

To evaluate human perception of color-aware AST, we conducted a user study considering style transfer outputs before and after transformation. We evaluated seven content images and two style images before and after applying transformations. We then conducted a seven-question survey considering random pairs of content and style images. One question involved 14 images, and it was responded for the outputs of the proposed and six comparison AST methods, obtaining 98 responses from 30 participants. We asked which output imaged better reflected the regional color distribution of the style image after transformation. The 30 participants included a junior artist, students enrolled in the art or computer science major, graphics/vision field researchers, and an immersive content creator. Every participant selected the best image among the outputs of all the methods considering the original and transformed style images for the same content image. From the user study, the proposed method achieved the best performance in terms of regional color distribution, as shown in Fig. 8. Almost half of the participants (45.71%) selected the proposed method as the best one among the evaluated methods. Remarkably, our proposal received dominant preference (>70%) for rotation and flipping transformations. On the other hand, regarding zooming with cropping, all the methods received similar proportions of preference. This was because the color distribution of the zoomed style images was distorted before AST, as illustrated in Fig. 6. Consequently, the output images of all the methods changed regardless of the ability to preserve the color distribution, causing highly variable preferences among the participants and similar preferences across methods after zooming the style image.

Along with the evaluation results presented in Section 5.2, we can conclude that the distance in standard deviation is not as important as that in mean regarding human perception of AST. In addition, the results in Figs. 7 and 8 suggest that the proposed method achieves SOTA performance and that the distance in mean allows to suitably evaluate the style similarity regarding human perception.

In addition to the existing AdaIN [26] training loss in Eq. (13), we introduce a local patch loss in Eq. (12). We evaluated the advantage of finetuning to highlight regional details in AST. We performed style transfer with two models, namely, a pretrained encoder-decoder replacing AdaIN with Patchified AdaIN without fine-tuning, whose weights were retrieved from an AdaIN PyTorch implementation [26], and a finetuned model considering our global-local loss in Eq. (13). As shown in Fig. 9, structurally distorted results were obtained without finetuning (Fig. 9b). Specifically, the boundaries between the cat and bathtub were blurry when using the pretrained model. In contrast, the finetuned model (Fig. 9a) preserved more structural details of the content image. Hence, finetuning enhanced the output image details without notably compromising the content image. Although blurry results may be artistically preferred in some contexts, preserving structural details is often preferred for human perception.

To further evaluate the user experience, we explored runtime control strategies focusing on the Patchified AdaIN parameters of patch level and type. These controls may enhance the user experience during inference without requiring additional training.

To confirm the practicality of the proposed Patchified AdaIN, we measured the inference time according to different patch levels. The results were obtained using a computer equipped with an NVIDIA GeForce RTX 3090 GPU of 24 GB. We calculated were the average inference time over 1000 iterations.

We performed AST for patch levels from 2 to 5, as in Section 5.5. The baseline was the pretrained AdaIN model [26], and our method considered the finetuned model replacing AdaIN with Patchified AdaIN. For comparison, we normalized the AdaIN inference speed to 1. As listed in Table 3, our models had a negligible inference gap compared with the simpler baseline. Even the highest patch level provided a speed with a factor of 1.057. Hence, the proposed method incurs a small inference burden, suggesting that Patchified AdaIN can be integrated in SOTA models without notably compromising the inference speed.

As discussed in Section 5.8, Patchified AdaIN can be used in other models through simple replacement without notably increasing the inference time. We selected various SOTA AST models based on AdaIN for replacement with the proposed Patchified AdaIN. We aimed to demonstrate the extensive applicability of our method and the AST performance preservation while reflecting the regional color distribution. For evaluation, we selected CCPL [30] with α=0.0 and LinearStyleTransfer [31] with α=0.5 for code blending using (9). Patchified AdaIN was integrated into the transformation or normalization block. As the evaluated models adopted different schemes for AST, we adapted Patchified AdaIN to their structure, establishing modified Patchified AdaIN implementations that preserved the core scheme of Patchified AdaIN of explicitly dividing the features into patches and aggregating them to obtain the output image. As shown in Fig. 12, the modified Patchified AdaIN endowed the SOTA models with the ability to perform spatially color-aware AST. The original SOTA models generated highly similar output images regardless of the transformed style image. After applying the modified Patchified AdaIN, the models provided different results reflecting the transformations, thus providing diverse appearances to the output images. These results demonstrate that our method is effective and easily applicable to other AST models without compromising performance. Nonetheless, it should be carefully considered when our method will be applied into existing networks.

For real-world use cases, it is important whether method can used in general and wide use cases, that is generalizailbity. To provide this capability, we provide additional experiment with some images that have non-uniform and complex color distribution. Samples which have a high variance are chosen and we conduct style transfer under our model using those images to showcase potential generalizability in real-world use cases. As shown in Fig. 13, we observed that resulting stylized images has few visual changes even though different transformations are adopted. This issue can be the limitation of our method, but it might be addressed if user input the style images which have high contrast, vibrant color intensity. Meanwhile, our patchification strategy requires patch aggregation due to the spatial division in 2-dimensional space. Therefore, there is potential quality degradation in the edge of patch in aggregation step. Especially, if the structure of content images has semantically important information in the edge, artifacts in that region could prominently demonstrate visually non-harmonized appearance. Thus, this potential quality degradation should be carefully considered in inference step.