Constrained by the physiology, the temporal factors associated with human behavior, irrespective of facial movement or body gesture, are described by four phases: neutral, onset, apex, and offset. Although they may benefit related recognition tasks, it is not easy to accurately detect such temporal segments. An automatic temporal segment detection framework using bilateral long short-term memory recurrent neural networks (BLSTM-RNN) to learn high-level temporal–spatial features, which synthesizes the local and global temporal–spatial information more efficiently, is presented. The framework is evaluated in detail over the face and body database (FABO). The comparison shows that the proposed framework outperforms state-of-the-art methods for solving the problem of temporal segment detection.

We present an approach for face recognition using synthesized three-dimensional (3-D) shape information together with two-dimensional (2-D) color in a deep convolutional neural network (DCNN). As 3-D facial shape is hardly affected by the extrinsic 2-D texture changes caused by illumination, make-up, and occlusions, it could provide more reliable complementary features in harmony with the 2-D color feature in face recognition. Unlike other approaches that use 3-D shape information with the help of an additional depth sensor, our approach generates a personalized 3-D face model by using only face landmarks in the 2-D input image. Using the personalized 3-D face model, we generate a frontalized 2-D color facial image as well as 3-D facial images (e.g., a depth image and a normal image). In our DCNN, we first feed 2-D and 3-D facial images into independent convolutional layers, where the low-level kernels are successfully learned according to their own characteristics. Then, we merge them and feed into higher-level layers under a single deep neural network. Our proposed approach is evaluated with labeled faces in the wild dataset and the results show that the error rate of the verification rate at false acceptance rate 1% is improved by up to 32.1% compared with the baseline where only a 2-D color image is used.

Face recognition is a challenging task in computer vision. Numerous efforts have been made to design low-level hand-crafted features for face recognition. Low-level hand-crafted features highly depend on prior knowledge, which is difficult to obtain without learning new domain knowledge. Recently, ConvNets have generated great attention for their ability of feature learning and achieved state-of-the-art results on many computer vision tasks. However, typical ConvNets are trained by a gradient descent method in supervised mode, which results in high computational complexity. To solve this problem, an efficient unsupervised deep learning network is proposed for face recognition in this paper, which combines both 2-D Gabor filters and (2D)2 PCA to learn the multistage convolutional filters. To speed up the calculation, the learned high-dimensional features are further encoded using short binary hashes. Finally, the obtained output features are trained using LinearSVM. Extensive experimental results on several facial benchmark databases show that the proposed network can obtain competitive performance and robust distortion-tolerance for face recognition.

This paper focuses on compressive sensing (CS)-based hyperspectral imaging (HSI). A band-by-band reconstruction approach, namely prior image constrained compressive sensing (PICCS)-based HSI, is proposed. Furthermore, a more effective PICCS model is built in this paper. Each hyperspectral band is reconstructed based on the previous one, which utilizes not only the sparsity of each hyperspectral band in a certain basis but also the similarity between two consecutive bands. Moreover, compared with the algorithms which reconstruct all the hyperspectral bands simultaneously, PICCS-based HSI reduces the requirements for computational ability and computational memory of the receivers. In addition, compared with the independent band-by-band reconstruction algorithms and tensor-SL0-based HSI, PICCS-based HSI significantly reduces the number of measurements with similar or better reconstruction quality. The convergence of the two algorithms is proved and some simulations are provided to illustrate their effectiveness.

The robust design and adaptation of multimedia networks relies on the study of the influence of potential network impairments on the perceived quality. Video quality may be affected by network impairments, such as delay, jitter, packet loss, and bandwidth, and the perceptual impact of these impairments may vary according to the video content. The effects of packet loss and encoding artifacts on the perceived quality have been widely addressed in the literature. However, the relationship between video content and network impairments on the perceived video quality has not been deeply investigated. A detailed analysis of ReTRiEVED test video dataset, designed by considering a set of potential network impairments, is presented, and the effects of transmission impairments on perceived quality are analyzed. Furthermore, the impact on the perceived quality of the video content in the presence of transmission impairments is studied by using video content descriptors. Finally, the performances of well-known quality metrics are tested on the proposed dataset.

The recently proposed robust principal component analysis (RPCA) theory and its derived methods have attracted much attention in many computer vision and machine intelligence applications. From a wide view of these methods, independent motion objects are modeled as pixel-wised sparse or structurally sparse outliers from a highly correlated background signal, and all these methods are implemented under an ℓ1 -penalized optimization. Real data experiments reveal that even if ℓ1-penalty is convex, the optimization sometimes cannot be satisfactorily solved, especially when the signal-to-noise ratio is relatively high. In addition, the unexpected background motion (e.g., periodic or stochastic motion) may also be included. We propose a moving object detection method based on a proximal RPCA along with saliency detection. Convex penalties including low-rank and sparse regularizations are substituted with proximal norms to achieve robust regression. After the foreground candidates have been extracted, a motion saliency map using spatiotemporal filtering is constructed. The foreground objects are then filtered out by dynamically adjusting the penalty parameter according to the corresponding saliency values. Evaluations on challenging video clips and qualitative and quantitative comparisons with several state-of-the-art methods demonstrate that the proposed approach works efficiently and robustly.

Gabor filters are widely utilized to detect iris texture information in several state-of-the-art iris recognition systems. However, the proper Gabor kernels and the generative pattern of iris Gabor features need to be predetermined in application. The traditional empirical Gabor filters and shallow iris encoding ways are incapable of dealing with such complex variations in iris imaging including illumination, aging, deformation, and device variations. Thereby, an adaptive Gabor filter selection strategy and deep learning architecture are presented. We first employ particle swarm optimization approach and its binary version to define a set of data-driven Gabor kernels for fitting the most informative filtering bands, and then capture complex pattern from the optimal Gabor filtered coefficients by a trained deep belief network. A succession of comparative experiments validate that our optimal Gabor filters may produce more distinctive Gabor coefficients and our iris deep representations be more robust and stable than traditional iris Gabor codes. Furthermore, the depth and scales of the deep learning architecture are also discussed.

Recently, binary descriptors have attracted significant attention due to their speed and low memory consumption; however, using intensity differences to calculate the binary descriptive vector is not efficient enough. We propose an approach to binary description called POLAR_MOBIL, in which we perform binary tests between geometrical and statistical information using moments in the patch instead of the classical intensity binary test. In addition, we introduce a learning technique used to select an optimized set of binary tests with low correlation and high variance. This approach offers high distinctiveness against affine transformations and appearance changes. An extensive evaluation on well-known benchmark datasets reveals the robustness and the effectiveness of the proposed descriptor, as well as its good performance in terms of low computation complexity when compared with state-of-the-art real-time local descriptors.

Inspired by unsupervised feature learning (UFL) within the self-taught learning framework, we propose a method based on UFL, convolution representation, and part-based dimensionality reduction to handle facial age and gender classification, which are two challenging problems under unconstrained circumstances. First, UFL is introduced to learn selective receptive fields (filters) automatically by applying whitening transformation and spherical k-means on random patches collected from unlabeled data. The learning process is fast and has no hyperparameters to tune. Then, the input image is convolved with these filters to obtain filtering responses on which local contrast normalization is applied. Average pooling and feature concatenation are then used to form global face representation. Finally, linear discriminant analysis with part-based strategy is presented to reduce the dimensions of the global representation and to improve classification performances further. Experiments on three challenging databases, namely, Labeled faces in the wild, Gallagher group photos, and Adience, demonstrate the effectiveness of the proposed method relative to that of state-of-the-art approaches.

This paper presents a hybrid sparse-representation-based approach to single-image super-resolution reconstruction. Our main contribution is threefold: (1) jointly utilize nonlocal similarity of intensity image and low-rank property of gradient image under the framework of sparse representation; (2) incorporate both the high-resolution (HR) and low-resolution dictionaries into the reconstruction process; and (3) incorporate both the unknown HR image and the sparse coefficients into a single objective function. By alternatively minimizing the objective function with respect to the unknown HR image and the sparse coefficients, we get an estimate of the target HR image. Extensive experiments validate that compared with many state-of-the-art algorithms the proposed method yields comparable results for noiseless images and achieves much better results for noisy images.

The formulation and application of an algorithm based on partial differential equations for processing underwater images are presented. The proposed algorithm performs simultaneous smoothing and enhancement operations on the image and yields better contrast enhancement, color correction, and rendition compared to conventional algorithms. Further modification of the proposed algorithm and its combination with the powerful contrast-limited adaptive histogram equalization (CLAHE) method using an adaptive computation of the clip limit enhances the local enhancement results while mitigating the color distortion and intrinsic noise enhancement observed in the CLAHE algorithm. Ultimately, an optimized version of the algorithm based on image information metric is developed for best possible results for all images. The method is compared with existing algorithms from the literature using subjective and objective measures, and results indicate considerable improvement over several well-known algorithms.

A large corpus of ceramic sherds dating from the High Middle Ages has been extracted in Saran (France). The sherds have an engraved frieze made by the potter with a carved wooden wheel. These relief patterns can be used to date the sherds in order to study the diffusion of ceramic production. The aim of the ARCADIA project was to develop an automatic classification of this archaeological heritage. The sherds were scanned using a three-dimensional (3-D) laser scanner. After projecting the 3-D point cloud onto a depth map, the local variance highlighted the shallow relief patterns. The saliency region focused on the motif was extracted by a density-based spatial clustering of FAST points. An adaptive thresholding was then applied to the depth to obtain a binary pattern close to manual sampling. The five most representative types of motif were classified by training an SVM model with a pyramid histogram of visual words descriptor. Compared with other state-of-the-art methods, the proposed approach succeeded in classifying up to 84% of the binary patterns on a dataset of 377 scanned sherds. The automatic method is extremely time-saving compared to manual stamping.

We propose a local texture descriptor, local intensity area descriptor (LIAD), which is applied for human facial recognition in ideal and noisy conditions. Each facial image is divided into small regions from which LIAD histograms are extracted and concatenated into a single feature vector to represent the facial image. The recognition is performed using a nearest neighbor classifier with histogram intersection and chi-square statistics as dissimilarity measures. Experiments were conducted with LIAD using the ORL database of faces (Olivetti Research Laboratory, Cambridge), the Face94 face database, the Georgia Tech face database, and the FERET database. The results demonstrated the improvement in accuracy of our proposed descriptor compared to conventional descriptors [local binary pattern (LBP), uniform LBP, local ternary pattern, histogram of oriented gradients, and local directional pattern]. Moreover, the proposed descriptor was less sensitive to noise and had low histogram dimensionality. Thus, it is expected to be a powerful texture descriptor that can be used for various computer vision problems.

The use of geographical tagging in social-media images is becoming a part of image metadata and a great interest for geographical information science. It is well recognized that geographical topic learning is crucial for geographical annotation. Existing methods usually exploit geographical characteristics using image preprocessing, pixel-based classification, and feature recognition. How to effectively exploit the high-level semantic feature and underlying correlation among different types of contents is a crucial task for geographical topic learning. Deep learning (DL) has recently demonstrated robust capabilities for image tagging and has been introduced into geoscience. It extracts high-level features computed from a whole image component, where the cluttered background may dominate spatial features in the deep representation. Therefore, a method of spatial-attentional DL for geographical topic learning is provided and we can regard it as a special case of DL combined with various deep networks and tuning tricks. Results demonstrated that the method is discriminative for different types of geographical topic learning. In addition, it outperforms other sequential processing models in a tagging task for a geographical image dataset.

Digital imaging has become a very important technique in the conservation of cultural art relics because it can nondestructively acquire the color and spectral image of cultural art relics for different applications. Imaging accuracy is one of the key factors in digital protection of cultural art relics. In order to improve the color and spectral accuracy for digital imaging of cultural art relics, the idea of making the specific color charts for different kinds of artworks is presented. Taking ancient Chinese Dunhuang murals as the specific object of study, a prototype pigments color chart of the Dunhuang murals (DCC), containing a six-step grayscale and 30 colored pigment samples, is made to investigate its pigment types and painting techniques. Under the premise of considering the difference in the number of samples in color charts, the DCC is tested and compared with the classic and widely used standard Macbeth colorchecker (CC) in two aspects: color correction for RGB imaging and spectral reconstruction for spectral imaging. The results show that the prototype pigments color chart is more effective and exhibits superior performance to the CC in both aspects for digital conservation of the Dunhuang murals.

An image resolution enhancement approach based on discrete wavelet transform (DWT) and new edge-directed interpolation (NEDI) for degraded satellite images by geometric distortion to correct the errors in image geometry and recover the edge details of directional high-frequency subbands is proposed. The observed image is decomposed into four frequency subbands through DWT, and then the three high-frequency subbands and the observed image are processed with NEDI. To better preserve the edges and remove potential noise in the estimated high-frequency subbands, an adaptive threshold is applied to process the estimated wavelet coefficients. Finally, the enhanced image is reconstructed by applying inverse DWT. Four criteria are introduced, aiming to better assess the overall performance of the proposed approach for different types of satellite images. A public satellite images data set is selected for the validation purpose. The visual and quantitative results show the superiority of the proposed approach over the conventional and state-of-the-art image resolution enhancement techniques.

This paper presents a real-time foveation technique proposed as an alternative method for image obfuscation while simultaneously preserving privacy in face deidentification. Relevance of the proposed technique is discussed through a comparative study of the most common distortions methods in face images and an assessment on performance and effectiveness of privacy protection. All the different techniques presented here are evaluated when they go through a face recognition software. Evaluating the data utility preservation was carried out under gender and facial expression classification. Results on quantifying the tradeoff between privacy protection and image information preservation at different obfuscation levels are presented. Comparative results using the facial expression subset of the FERET database show that the technique achieves a good tradeoff between privacy and awareness with 30% of recognition rate and a classification accuracy as high as 88% obtained from the common figures of merit using the privacy-awareness map.

The concept of radar imaging based on orbital angular momentum (OAM) modulation, which has the ability of azimuthal resolution without relative motion, has recently been proposed. We investigate this imaging technique further in greater detail. We first analyze the principle of the technique, accounting for its resolving ability physically. The phase and intensity distributions of the OAM-carrying fields produced by phased uniform circular array antenna, which have significant effects on the imaging results, are investigated. The imaging model shows that the received signal has the form of inverse discrete Fourier transform with the use of OAM and frequency diversities. The two-dimensional Fourier transform is employed to reconstruct the target images in the case of large and small elevation angles. Due to the peculiar phase and intensity characteristics, the small elevation is more suitable for practical application than the large one. The minimum elevation angle is then obtained given the array parameters. The imaging capability is analyzed by means of the point spread function. All results are verified through numerical simulations. The proposed staring imaging technique can achieve extremely high azimuthal resolution with the use of plentiful OAM modes.

We propose an approach to reconstruct a three-dimensional model from a stream of RGB-D images. Compared with existing methods, our strategy performs well in challenging cases, involving quick movements of the camera and missing depth values. The key idea underlying our approach is to combine registration with patch segmentation based on RGB information. We use these segments and the patch significance correspondence algorithm to transform a global restriction into a number of local restrictions during camera moving. Furthermore, we propose a method that improves the low precision of geometric registration by aligning corresponding patches instead of entire point clouds. In light of this consideration of spatial relations, we also propose a fusion strategy to extract the correct transformations in spite of poor RGB-D sequences. The results of tests on RGB-D benchmark sequences and comparisons with KinectFusion system showed that the proposed approach substantially increases the accuracy of the reconstructed models.

Convolutional neural network (CNN)-based approaches have received state-of-the-art results in scene classification. Features from the output of fully connected (FC) layers express one-dimensional semantic information but lose the detailed information of objects and the spatial information of scene categories. On the contrary, deep convolutional features have been proved to be more suitable for describing an object itself and the spatial relations among objects in an image. In addition, the feature map from each layer is max-pooled within local neighborhoods, which weakens the invariance of global consistency and is unfavorable to scenes with highly complicated variation. To cope with the above issues, an orderless multi-channel mid-level image representation on pre-trained CNN features is proposed to improve the classification performance. The mid-level image representation of two channels from the FC layer and the deep convolutional layer are integrated at multi-scale levels. A sum pooling approach is also employed to aggregate multi-scale mid-level image representation to highlight the importance of the descriptors beneficial for scene classification. Extensive experiments on SUN397 and MIT 67 indoor datasets demonstrate that the proposed method achieves promising classification performance.

We propose a method for blotch detection in archived videos by modeling their spatiotemporal properties. We introduce an adaptive spatiotemporal segmentation to extract candidate regions that can be classified as blotches. Then, the similarity between the preselected regions and their corresponding motion-compensated regions in the adjacent frames is assessed by means of motion trajectory estimation and textural information analysis. Perceived ground truth based on just noticeable contrast is employed for the evaluation of our approach against the state-of-the-art, and the reported results show a better performance for our approach.

In real applications, a pinhole camera capturing objects through a planar parallel transparent plate is frequently employed. Due to the refractive effects of the plate, such an imaging system does not comply with the conventional pinhole camera model. Although the system is ubiquitous, it has not been thoroughly studied. This paper aims at presenting a simple virtual camera model, called a plate refractive camera model, which has a form similar to a pinhole camera model and can efficiently model refractions through a plate. The key idea is to employ a pixel-wise viewpoint concept to encode the refraction effects into a pixel-wise pinhole camera model. The proposed camera model realizes an efficient forward projection computation method and has some advantages in applications. First, the model can help to compute the caustic surface to represent the changes of the camera viewpoints. Second, the model has strengths in analyzing and rectifying the image caustic distortion caused by the plate refraction effects. Third, the model can be used to calibrate the camera’s intrinsic parameters without removing the plate. Last but not least, the model contributes to putting forward the plate refractive triangulation methods in order to solve the plate refractive triangulation problem easily in multiviews. We verify our theory in both synthetic and real experiments.

Quaternion polar complex exponential transform (QPCET) moments and their invariants are widely used as powerful tools in many image processing and pattern recognition applications. However, the accuracy of the conventional approximated method for computing QPCET moments suffers from geometric and numerical errors. This approximated method is very time-consuming. Moreover, computing the high orders of approximated QPCET moments suffers from numerical instability. Computational methods are proposed for fast and accurate computation of the QPCET moments for color images in two coordinate systems. In the first method, the Gaussian quadrature method is applied to compute higher-order moments of QPCET in the Cartesian coordinates. On the other side, an exact kernel-based method is employed to compute the higher-order moments of QPCET in the polar coordinates. A set of numerical experiments is conducted and the obtained results clearly show that the conventional approximated method is unstable, where the numerical instability encountered with moment order ≥10, while the first proposed method is unstable for moment order ≥60. On the other side, the second proposed method is stable for all orders. The comparison clearly shows the superiority of the second proposed method in terms of image reconstruction capability, numerical stability, fast computation, rotation invariances, and robustness to different kinds of noises.

Detection of visually salient image regions is extensively applied in computer vision and computer graphics, such as object detection, adaptive compression, and object recognition, but any single model always has its limitations to various images, so in our work, we establish a method based on multimodel saliency maps to detect the object, which intelligently absorbs the merits of various individual saliency detection models to achieve promising results. The method can be roughly divided into three steps: in the first step, we propose a decision-making system to evaluate saliency maps obtained by seven competitive methods and merely select the three most valuable saliency maps; in the second step, we introduce heterogeneous PCNN algorithm to obtain three prime foregrounds; and then a self-designed nonlinear fusion method is proposed to merge these saliency maps; at last, the adaptive improved and simplified PCNN model is used to detect the object. Our proposed method can constitute an object detection system for different occasions, which requires no training, is simple, and highly efficient. The proposed saliency fusion technique shows better performance over a broad range of images and enriches the applicability range by fusing different individual saliency models, this proposed system is worthy enough to be called a strong model. Moreover, the proposed adaptive improved SPCNN model is stemmed from the Eckhorn’s neuron model, which is skilled in image segmentation because of its biological background, and in which all the parameters are adaptive to image information. We extensively appraise our algorithm on classical salient object detection database, and the experimental results demonstrate that the aggregation of saliency maps outperforms the best saliency model in all cases, yielding highest precision of 89.90%, better recall rates of 98.20%, greatest F-measure of 91.20%, and lowest mean absolute error value of 0.057, the value of proposed saliency evaluation EHA reaches to 215.287. We deem our method can be wielded to diverse applications in the future.

An approach for impulse noise detection and removal in color images based on Moran’s I (MI) statistic is proposed. The proposed method consists of detection and removal components and is called Moran’s I vector median filter (MIVMF). The detection module is able to determine if a pixel is noise or noise-free. If it is a noise pixel, the vector median filter (VMF) will be used to remove the noise. This detection capability meets the so-called “switching” mechanism, which only selects noisy pixels for denoising. Hence, this proposed filter will expedite the processing time with the reduced number of vector calculations in the VMF due to this detection function. This type of detection is achieved with MI index and the indication of one-dimensional Laplacian kernels. We compare the proposed MIVMF with other well-developed vector-type median filters in the literature. Our experimental results show that the proposed filter is not only faster in the filtering process but also efficient in removing random impulse noise with different noise levels in color images. The MIVMF demonstrates a promising denoising result based on the criteria of peak signal-to-noise ratio and structural similarity index metric. With the visualization of processed images, the MIVMF can avoid image blurring, preserve the edge details, and achieve superior noise reduction.

A single-image super-resolution (SR) method is proposed. The proposed method uses a generated dictionary from pairs of high resolution (HR) images and their corresponding low resolution (LR) representations. First, HR images and the corresponding LR ones are divided into patches of HR and LR, respectively, and then they are collected into separate dictionaries. Afterward, when performing SR, the distance between every patch of the input LR image and those of available LR patches in the LR dictionary is calculated. The minimum distance between the input LR patch and those in the LR dictionary is taken, and its counterpart from the HR dictionary is passed through an illumination enhancement process. By this technique, the noticeable change of illumination between neighbor patches in the super-resolved image is significantly reduced. The enhanced HR patch represents the HR patch of the super-resolved image. Finally, to remove the blocking effect caused by merging the patches, an average of the obtained HR image and the interpolated image obtained using bicubic interpolation is calculated. The quantitative and qualitative analyses show the superiority of the proposed technique over the conventional and state-of-art methods.

The objective of this study is to compare several change detection methods for a monostatic camera and identify the best method for different complex environments and backgrounds in indoor and outdoor scenes. To this end, we used the CDnet video dataset as a benchmark that consists of many challenging problems, ranging from basic simple scenes to complex scenes affected by bad weather and dynamic backgrounds. Twelve change detection methods, ranging from simple temporal differencing to more sophisticated methods, were tested and several performance metrics were used to precisely evaluate the results. Because most of the considered methods have not previously been evaluated on this recent large scale dataset, this work compares these methods to fill a lack in the literature, and thus this evaluation joins as complementary compared with the previous comparative evaluations. Our experimental results show that there is no perfect method for all challenging cases; each method performs well in certain cases and fails in others. However, this study enables the user to identify the most suitable method for his or her needs.

Object tracking is an interdisciplinary research topic in image processing, pattern recognition, and computer vision which has theoretical and practical application value in video surveillance, virtual reality, and automatic navigation. Compressive tracking (CT) has many advantages, such as efficiency and accuracy. However, when there are object occlusion, abrupt motion and blur, similar objects, and scale changing, the CT has the problem of tracking drift. We propose the Markov object location prediction to get the initial position of the object. Then CT is used to locate the object accurately, and the classifier parameter adaptive updating strategy is given based on the confidence map. At the same time according to the object location, extract the scale features, which is able to deal with object scale variations effectively. Experimental results show that the proposed algorithm has better tracking accuracy and robustness than current advanced algorithms and achieves real-time performance.