In the investigation, protection and restoration of murals, the exact location and size of fragmentation disease can be labeled to facilitate the subsequent protection and cultural heritage of murals. However, manual labeling is time-consuming and laborious, and the results will be various due to the different experience of experts, which is not conducive to the promotion of intelligent cultural relic protection and restoration. The intelligent labeling of mural diseases through artificial intelligence can greatly improve the efficiency of mural restoration and solve these deficiencies. Therefore, an intelligent labeling method for mural fragments based on gradient-trainable Gabor and U-Net is proposed. In this paper, the disease labeling problem is transformed into the image segmentation problem for disease regions. However, due to the rich texture of the mural image and the complex edges of the fragmentation regions, a lot of detail is lost for disease labeling directly using the U-Net network. Different from previous studies, this method uses gradient-trained Gabor in the encoders to extract texture features of fragmentation disease and obtain more texture information of the disease region. In particular, res2convolution is embedded into the skip connections to narrow the semantic gap between encoder and decoder and better inject the texture information of fragmentation disease into the deep network. Finally, we proved that the method proposed in this paper can realize the intelligent labeling of fragmentation diseases accurately and efficiently through the murals of Han Tomb at Xi 'an Jiaotong University.
KEYWORDS: Data hiding, Mining, Convolution, Machine learning, Hyperspectral imaging, Feature extraction, Data modeling, Principal component analysis, Data processing, Image fusion
Implicit information exploration techniques are of great importance for the restoration and conservation of cultural relics. At present, the hyperspectral image analysis technique is one of the main methods to extract hidden information, which mainly contains two analysis methods such as principal component analysis (PCA) and minimum noise fraction rotation (MNF), both of which have achieved certain information extraction effects. In recent years, with the development of artificial intelligence, deep learning, and other technologies, nonlinear methods such as neural networks are expected to further improve the effect of implicit information mining. Therefore, this paper is oriented to the problem of extracting hidden information from pottery artifacts and tries to study and explore the hidden information mining method based on deep neural networks, expecting to obtain more stable and richer hidden information. In this paper, an auto-encoder-based implied information mining method is proposed first, and the auto-encoder (AE) framework achieves good performance in feature learning by automatically learning low-dimensional embedding and reconstructing data. However, during the experiments, it is found that some important detailed information (e.g., implicit information) is often lost in the reconstruction process because the traditional autoencoder network only focuses more on the pixel-level reconstruction loss and ignores the overall distribution. Therefore, this paper further proposes a multi-scale convolutional autoencoder network (MSCAE). It constructs a multi-scale convolutional module based on the traditional AE and designs a cyclic consistency loss in addition to the reconstruction loss, to reduce the loss of detailed information in the reconstruction process and improve the implicit information mining effect. In the experiments, we find that the proposed method can achieve effective implied information mining by extracting implied information from cocoon-shaped pots, and its visual effect has been improved compared with the traditional AE network.
Video super-resolution (VSR) aims to generate high-resolution (HR) video by exploiting temporal consistency and contextual similarity of low-resolution (LR) video sequences. The key to improving the quality of VSR lies in accurate frame alignment and the feature fusion of adjacent frames. We propose a dual channel attention deep and shallow super-resolution network, which combines with HR optical flow compensation to construct an end-to-end VSR framework HOFADS-VSR (attention deep and shallow VSR network union HR optical flow compensation). HR optical flow calculated by spatiotemporal dependency of consecutive LR frames is used to compensate adjacent frames to implement accurate frame alignment. Deep and shallow channels with attention residual block restore small-scale detail features and large-scale contour features, respectively, and strengthen the rich features of global and local regions through weight adjustment. Extensive experiments have been performed to demonstrate the effectiveness and robustness of HOFADS-VSR. Comparative results on the Vid4, SPMC-12, and Harmonic-8 datasets show that our network not only achieves good performance on peak signal-to-noise ratio and structural similarity index but also the restored structure and texture have excellent fidelity.
Generative adversarial network (GAN) for super-resolution (SR) has attracted enormous interest in recent years. It has been widely used to solve the single-image super-resolution (SISR) task and made superior performance. However, GAN is rarely used for video super-resolution (VSR). VSR aims to improve video resolution by exploiting the temporal continuity and spatial similarity of video sequence frames. We design a GAN with multi-feature discriminators and combine it with optical flow estimation compensation to construct an end-to-end VSR framework OFC-MFGAN. Optical flow estimation compensation makes use of temporal continuity and spatial similarity features of adjacent frames to provide rich detailed information for GAN. Multi-feature discriminators based on visual attention mechanism include the pixel discriminator, edge discriminator, gray discriminator, and color discriminator. GAN with multi-feature discriminators makes the data distribution and visually sensitive features (edge, texture, and color) of SR frames similar to high-resolution frames. OFC-MFGAN effectively integrates the time, space, and visually sensitive features of videos. Extensive experiments on public video datasets and surveillance videos show the effectiveness and robustness of the proposed method. Compared with several state-of-the-art VSR methods and SISR methods, the proposed method can not only recover prominent edges, clear textures, and realistic colors but also make a pleasant visual feeling and competitive perceptual index.
In the preservation and restoration of murals, labeling and recording the location and size of the paint loss disease can bring convenience to the subsequent restoration work. At present, the most common method of disease labeling is to draw the disease area manually on an orthophoto map by human-computer interaction. However, this method not only requires much time, but also leads to different labeling results due to different experts' experience. In recent years, with the development of artificial intelligence, machine learning and other technologies, it is possible to realize intelligent labeling through image processing and other methods. Therefore, this paper focuses on the mural paint loss disease and tries to explore the intelligent disease labeling method, hoping to efficiently and accurately mark the paint loss disease. In this paper, firstly, the disease labeling is transformed into an image segmentation problem, and proposes a mural paint loss disease labeling based on U-Net. However, it was experimentally found that much detailed information is often lost when the U-Net is used directly. Therefore, this paper further proposes multi-scale detail injection U-Net, including the constructed multi-scale module and the method of injecting shallow features into in-depth features, which could effectively extract more abundant edge information and improve the labeling accuracy. Furthermore, we demonstrate that the method proposed in this paper could actually achieve the intelligent labeling of the paint loss disease through the murals of the Liao Dynasty Feng Guo Temple in Yi County, Jinzhou City, China.
The sketches of painted cultural objects can be the most indicative of the style of paintings. Extraction of the sketches is an integral process used by conservators and art historians for documentation and for artists to learn historical painting styles through copying and painting. However, at present, sketch extraction is mainly manually drawn, which is not only time-consuming, but also subjective and dependent on experience. Therefore, both accuracy and efficiency need to be improved. In recent years, with the development of machine learning, a series of extraction methods based on edge detection have emerged. However, most of the existing methods can only perform successful extraction if the sketches are well preserved , but for the data with faded sketches or severe conservation issues, the extraction methods need to be improved. It is beneficial to extract the bands that accentuate the sketches while suppressing the effects of the degraded areas and the overlapping paints. We propose a sketch extraction method based on hyperspectral image and deep learning. Firstly, the hyperspectral image data is collected and the bands sensitive to the sketches are extracted by a prior knowledge of the sketches (e.g. near infrared bands will be chosen if the sketches are made of carbon ink), and a dataset including a large number of existing natural images is used to pre-train the bi-directional cascade network (BDCN). The network parameters in the model are then fine-tuned by using the images of painted cultural objects drawn by experts, so as to solve the problem of insufficient sketch dataset of painted cultural objects and enhance the generalization ability of the model. Finally, the U-net network is used to further suppress the noise, i.e. unwanted information, and make the sketch clearer. The experimental results show that the proposed method can not only effectively extract sketch from ideal data, but also extract clear sketches from data with faded sketches and even with noise interference. It is superior to the other six advanced based on edge detection methods in visual and objective comparison, and has a good application prospect. The proposed deep learning method is also compared with an unsupervised clustering method using Self-Organising Map (SOM) which is a ‘shallow learning’ method where pixels of similar spectra are grouped into clusters without the need for data labeling by experts.
The murals of the Tang Tomb are important materials for studying the social life of the Tang Dynasty, which have important protection and research value. In order to protect the tomb murals as longer as possible, it is necessary to restore the murals and accurately record the restore location. Nevertheless,the restored murals are difficult to observe directly the restore area through the human eye. This paper proposes a method to reveal the restored areas, by extracting the main components of the Multi-Hyper-spectral image of the mural with the Minimum Noise Fraction (MNF) Rotation, and the location of the restored area is clearly observed from the main component. In addition, the mural sketch reflects the main content of the mural of the Tang Tomb murals, which are of great significance to the restoration and protection of the Tang Tomb murals. In this paper, we also proposed a new method to extract the sketch of Tang Tomb mural. For the bands sensitive to the composition of the sketches, the sparsely constrained sparse non-negative matrix under- approximation method is used to decompose the optimal sketches composition, and then the sketches are automatically extracted based on the idea of layer superposition. Through the experiments on the mural paintings in the three tombs, the results demonstrated that the proposed method could effectively perceive the area of mural restoration and automatically extract the sketch accurately and clearly, while saving manpower.
Access to the requested content is limited to institutions that have purchased or subscribe to SPIE eBooks.
You are receiving this notice because your organization may not have SPIE eBooks access.*
*Shibboleth/Open Athens users─please
sign in
to access your institution's subscriptions.
To obtain this item, you may purchase the complete book in print or electronic format on
SPIE.org.
INSTITUTIONAL Select your institution to access the SPIE Digital Library.
PERSONAL Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.