Model training loss and analysis

The loss function is a useful tool for network structure assessment. During the training with the ProGAN algorithm in this study, gradient optimization was realized by continuously updating each of the parameters. After training for 80 epochs, the parameters achieved an optimal state, which resulted in the maximization of the loss function. The variations in the loss and PSNR values according to epoch training are shown in Fig. 7.

Fig. 7
figure 7

PNSR and loss during network training. a PSNR change during network training. b Loss change during network training

According to the variations in the loss and PSNR values, the loss function stabilized near 6.2, and the PSNR value no longer showed an increasing tendency after increasing to approximately 29.85. At this moment, the network achieved a satisfactory convergence effect.

Influence of the dense compression unit composition on the network performance

In this study, the dense compression unit (DCU), which is the core component of the pyramid structure, was designed by referencing the residual network [26]. In our study, a 1 × 1 convolution compression layer was used as the final layer of each dense compression unit. This treatment increased the number of dense compression units but did not affect the PSNR performance, and the number of model parameters was successfully reduced. The comparison results are summarized in Table 1.

Table 1 Comparison of the composition effects of different DCU structures

The results provided in Table 1 were based on comparisons of the Set14 dataset. During operation, we tested the reconstruction effect of the two structures at 4× superresolution. Although the number of DCUs was increased in our study, this increase did not affect the PSNR value of the reconstructed images. Furthermore, the number of parameters was greatly reduced. The adoption of such a structure during training resulted in a network model that consumes less time but is more lightweight with better performance.

Superresolution reconstruction effect

To test the expressiveness of the algorithm for mural image reconstruction, several representative murals with different styles were selected for 4× superresolution processing. The results are shown in Fig. 8. The reconstructed high-resolution images retained some of the texture details of the original images, and the clarity was also satisfactory. The PSNR and SSIM values of the reconstructed image are summarized in Table 2. Based on the numerical values, the reconstructed image had satisfactory visual expressiveness.

Fig. 8
figure 8

Reconstruction of different styles of murals

Table 2 PSNR and SSIM of the mural after reconstruction

Furthermore, from the mural images of different styles, we randomly selected a total of 40 images (5 for each style), and then noise was added. After superresolution reconstruction with the proposed model, the measured average PSNR and SSIM values were 21.22 and 0.37, respectively. The reconstructed images are shown in Fig. 9, where images with more added noise (Fig. 9a and c) and those with less added noise (Fig. 9b and d) were selected for effect displays. The PSNR and SSIM values are summarized in Table 3.

Fig. 9
figure 9

Superresolution of noise murals. a and c Images with more added noise. b and d Images with less added noise

Table 3 PSNR and SSIM of the noise image after reconstruction

As shown in Fig. 9; Table 3, for images with more added noise, the noise was also proportionally amplified after superresolution handling, as a consequence of which the images remained indistinct (also attested to by the PSNR value). In contrast, the images with less added noise displayed more distinct images as well as noise after superresolution handling.

Comparative experiment of the reconstructed mural images of different types

To test the applicability of the proposed model for various mural styles, we selected eight types of mural images: animal, building, cloud, disciple, fo, people, plant and pusa with 851, 647, 732, 640, 484, 819, 588 and 610 images, respectively, for a total of 5371 images.

First, the ProGAN superresolution reconstruction model was used for 4× superresolution reconstruction of the above 5371 images. The reconstructed images are shown in Fig. 10. The average PSNR was 27.15 dB, and the average SSIM was 0.68.

Fig. 10
figure 10

Different reconstructed mural styles

Then, for comparison purposes, we selected three classical superresolution methods, i.e., bicubic interpolation (BI) superresolution reconstruction, SRGAN superresolution reconstruction, and enhanced deep residual networks, for single image superresolution (EDSR) superresolution reconstruction, as well as two other superresolution methods already applied to mural images in the literature [17] and [20]. To better reflect the performance of the algorithms on mural images of different styles, the 5371 images were divided based on their style. Six different algorithms were used for 4× superresolution reconstruction, and then the reconstruction effects were compared (Fig. 11).

Fig. 11
figure 11

Different mural reconstruction algorithms

As shown in Fig. 11, the performance effect of the traditional superresolution reconstruction method BI was not satisfactory when dealing with certain types of images, i.e., murals. The reconstructed images exhibited obvious blurs and unclear textures, as well as considerable feature loss. In addition, the reconstructed images contained artifacts and noise that did not exist in the original images. In the literature [17], the mural image superresolution reconstruction method is based on a traditional algorithm (not one based on deep learning). This method can achieve a satisfactory effect when applied to mural images with simple textures. In regard to the mural images with complex textures involved in this study, however, it created artifacts in the reconstructed image, similar to the BI algorithm, resulting in an unsatisfactory overall effect. The method used in the literature [20] improved to some extent the texture blurring effect and margin zigzagging that CNNs encounter when dealing with mural images, which can be reflected by an improved image structure after reconstruction compared with that after BI reconstruction. However, when applied to images of the “fo”, “disciple”, “people” and “pusa” classes, it failed to handle the detailed features, ultimately leading to unsatisfactory effects. Compared with the traditional BI algorithm, the deep learning-based SRGAN algorithm had a noticeably better performance due to the GAN method applied for superresolution reconstruction. The reconstructed image displayed a satisfactory reduction in texture detail while retaining the color and features of the original images. Compared with the BI algorithm, the image reconstructed by this algorithm did not have obvious colors that were too bright or dark, and the original image contrast was essentially maintained. However, this algorithm showed varying performance for the details of images with different styles.

After reconstructing images in the disciple, fo, people and pusa categories, this algorithm increased the number of artifacts in the faces of the reconstructed images, while for the other image categories, such as nature and building, this method did not satisfactorily preserve the detailed features. EDSR is a superresolution reconstruction network model that relies on an exploration period. This method showed satisfactory performance in terms of retaining texture details and colors. Compared with the original HD image and the images reconstructed by the two previous methods, the images reconstructed by this algorithm were the most similar to the real image and had a satisfactory performance for different styles of mural images. The network model proposed in this study further improved the expressiveness of the reconstructed images. It not only retained the color of the original background of the image but also considered the texture details and features of the image. Compared with the other algorithms, the proposed algorithm exhibited clear advantages. The following table summarizes the PSNR and SSIM values for the different style images after reconstruction.

As shown in Table 4, the ProGAN algorithm noticeably outperformed the other algorithms in reconstructing different styles of mural images. Compared with the BI algorithm, which had the worst performance among the considered algorithms, the ProGAN algorithm increased the average PSNR value by 6.66 dB. Although the EDSR algorithm showed a satisfactory reconstruction effect, the proposed algorithm increased the PSNR value by 0.2 dB. In summary, the ProGAN algorithm was more suitable for mural superresolution reconstruction.

Table 4 PSNR and SSIM after reconstruction of various murals with different algorithms

Comparative experiment of the reconstructed murals

Because reconstructing mural images has high requirements for local details, for the comparative experimental analysis, we selected 6 high-resolution mural images (Fig. 12) and reconstructed the image details with superresolution. Then, the reconstructed images were compared. The results are shown in Fig. 13.

Fig. 12
figure 12
Fig. 13
figure 13

Different mural reconstruction algorithms

As shown in Fig. 13, the traditional BI algorithm showed poor performance in superresolution reconstruction. The superresolution image reconstructed by this algorithm had clear distortions in color and brightness. These distortions did not occur when the method in the literature [17] was used for images with simple texture (Fig. 12b and f), and a satisfactory effect was achieved. Despite the achieved improvement, the method in the literature [17] did not achieve a satisfactory effect when applied to images with complex textures (e.g., the maid’s belt in Fig. 12d). Compared with the BI algorithm, the SRGAN algorithm showed a better performance. The reconstructed image did not have clear texture issues; however, the performance in the details of the reconstructed images was not satisfactory, especially for images with more texture details, such as the images in Fig. 12c and d. Although the method used in the literature [20] performed quite well in resolving the margin zigzagging problem and the reconstructed image exhibited clear contour detail, there was still much room to improve the color. The EDSR algorithm performed well in reconstructing the details of the images and therefore outperformed the first two algorithms for images with many details (Fig. 12c and d) or images with dull backgrounds (Fig. 12b, e and f). However, the reconstructed images did not perform well in terms of brightness, and the performance of the image features needed to be improved.

In summary, the BI algorithm, as an early classical superresolution method, did not satisfactorily reconstruct image features and texture details; therefore, the reconstructed images with this method were the worst. As the first superresolution GAN algorithm, the SRGAN showed performance improvement. However, the reconstructed images had an edge sawtooth. The EDSR algorithm restored some of the high-frequency details of the image, but the reconstructed image exhibited too much noise. In contrast, the reconstructed image with the algorithm proposed in this study had an improved texture, clarity and overall image performance.

Table 5 shows the PSNR and SSIM indices of the images reconstructed by the above four algorithms. As shown in this table, the algorithm proposed in this study had better evaluation indices than the previous algorithms. This finding indicated that this algorithm improved the mural representation effect to some extent and, therefore, might be helpful in improving the value of mural research.

Table 5 PSNR and SSIM after reconstructing murals with different algorithms

Superresolution reconstruction experiment for general images

Alongside the superresolution reconstruction experiment for mural images, we also tested the performance of the proposed algorithm on general image datasets, which included BSD100, URBAN100 and Set14. The results are summarized in Table 6.

Table 6 The PSNR (dB) results on three test datasets

Based on the PSNR value obtained in the experiment, the proposed model exhibited satisfactory performance on the public datasets. Compared with the VDSR, which was obtained based on deep convolution training, the performance of the proposed algorithm was not lower on any of the datasets. Compared with the EDSR algorithm, it even showed some advantages.

Subjective evaluation

In addition to using the PSNR and SSIM indices to evaluate the experimental results, we also selected 50 volunteers with normal vision to score the reconstructed images. The score interval was set from 0 to 5. The subjective evaluation indices included the overall impression and texture detail retention of the reconstructed images. Averages were obtained, and a high score indicated a better superresolution reconstruction effect. The final scores are shown in Fig. 14.

Fig. 14
figure 14

Comparison chart of subjective scores for the different algorithms

Based on the evaluation results and the intuitive feelings of the evaluators, the proposed algorithm outperformed the remaining algorithms in terms of reconstruction, including the overall appearance and texture detail. In the overall appearance, the images reconstructed by the proposed algorithm had a satisfactory performance in terms of brightness and smoothness. In evaluating the effect of fine superresolution handling for low-resolution murals, volunteers carefully compared the images before and after handling, and they all confirmed that the proposed algorithm had more advantages in retaining the original texture details of the image and had a better overall impression. In general, the algorithm proposed in this study successfully restored the texture of the original images while preserving the overall performance, improving the research value of the related image.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.

Disclaimer:

This article is autogenerated using RSS feeds and has not been created or edited by OA JF.

Click here for Source link (https://www.springeropen.com/)