A shallow yet effective convolutional neural network (CNN)-based detector was proposed for automatic detection of random-valued impulse noise (RVIN) from images. We guided the proposed CNN-based detector to automatically extract the implicit statistics and learn the detection mechanism with a large number of patches and their corresponding noise labels regarding center pixels. Compared with the reference RVIN detectors, the proposed CNN-based one takes advantage of the prior knowledge obtained in the training phase and shows impressive detection accuracy across different noise ratios.

This editorial summarizes the JEI Special Section on Image and Video Analysis, Detection and Recognition.

Chinese calligraphy is often used to perform the beauty of characters in Chinese culture and is quite suitable in the study of shape representation. The skeleton of a digital line pattern can be treated as the shape descriptor. However, the skeleton-biased and reconstruction-incomplete phenomena often exist in a skeletonization method, which results in the difficulty of using the skeleton to perform the beauty of Chinese calligraphy characters. To overcome this difficulty, skeletal line information derived from the skeletal points and indexed boundary points is defined, and its transformation is implemented by a procedure of two-phase skeletal line placement (SLP). Based on the SLP, an effective algorithm including the SLP-stroke for strokes, SLP-fork for forks, and SLP-end for the end parts of strokes is developed for constructing the skeletal line-based shape descriptor. Four indices of measurement of skeleton deviation, number of distorted forks, number of spurious strokes, and measurement of reconstructability are used to evaluate the performance of the proposed approach. Experimental results show that the skeleton-biased phenomenon can be greatly reduced and the pattern reconstructability close to 100% is achieved, thus confirming that the proposed skeletonization approach is suitable for the Chinese calligraphy character representation and reconstruction.

Vehicle color recognition is easily affected by subtle environmental changes. The existing recognition methods cannot achieve an accurate result. A high-accuracy vehicle color recognition method using a hierarchical fine-tuning strategy for urban surveillance videos is proposed. Different from the conventional convolutional neural networks-based methods, which usually obtain a single classification model, the proposed method combines pretraining and hierarchical fine-tunings to obtain different classification models that can adapt to the change of illumination conditions. First, the GoogLeNet is pretrained using the ILSVRC-2012 dataset to obtain the initial weight parameters of the network. During the first stage of fine-tuning, the whole vehicle color dataset is used to fine-tune the pretrained results to get the initial classification model. Then, an image quality assessment method is proposed to evaluate the illumination conditions of the image. The whole vehicle color dataset is divided into some subdatasets according to the evaluation results. The second stage of fine-tuning is performed on the initial classification model using each subdataset. Thus, the final classification models for the subdatasets are obtained. The experimental results on different databases demonstrate that the recognition accuracy of the proposed method can achieve superior performance over the state-of-the-art methods.

Recent studies in saliency detection have exploited contrast value as a main feature and background prior as a secondary feature. To apply the background prior, most approaches are based on soft- or hard-segmentation mechanisms, and a significant improvement is seen. However, because of contrast feature usage, the soft-segmentation (SS)-wise models have many technical challenges when a high interobject dissimilarity exists. Although hard-segmentation-wise saliency models intuitively use the background prior without usage of the contrast feature, this model suffers from local noises due to undesirable discontinuous artifacts. By analyzing the drawbacks of the existing models, a combination saliency model, reflecting both soft- and hard-segmentation techniques is shown. The proposed model consists of the following three phases: SS-wise saliency, hard-segmentation-wise saliency, and a final saliency combination. In particular, we proposed an iterative reweighting processing for which an influence of outlier segmentation maps is decreased to improve the hard-segmentation-wise saliency. As shown in the experimental results, the proposed model outperforms the state-of-the-art models on various benchmark datasets, which consist of single, multiple, and complex object images.

Eye and mouth state analysis is an important step in fatigue detection. An algorithm that analyzes the state of the eye and mouth by extracting contour features is proposed. First, the face area is detected in the acquired image database. Then, the eyes are located by an EyeMap algorithm through a clustering method to extract the sclera-fitting eye contour and calculate the contour aspect ratio. In addition, an effective algorithm is proposed to solve the problem of contour fitting when the human eye is affected by strabismus. Meanwhile, the value of chromatism s is defined in the RGB space, and the mouth is accurately located through lip segmentation. Based on the color difference of the lip, skin, and internal mouth, the internal mouth contour can be fitted to analyze the opening state of mouth; at the same time, another unique and effective yawning judgment mechanism is considered to determine whether the driver is tired. This paper is based on the three different databases to evaluate the performance of the proposed algorithm, and it does not need training with high calculation efficiency.

One of the most challenging tasks in computer vision is to emulate human cognitive ability to extract the salient object in a scene. We tackle the task of unsupervised salient video object segmentation using boundary connectedness and space-time salient regions. First, boundary prior measure is used to separate salient regions detected in both space and time. Then, background-foreground regions connectedness is computed and combined with appearance model via an iterative energy minimization framework to segment the salient moving object. For temporal consistency, the segmentation result of the current frame is used in addition to the optical flow and the boundary prior to segmenting the next frame. The experiments show a good performance of our algorithm for salient video object segmentation on benchmark datasets even in the presence of different challenges.

Characteristics of an image, such as smoothness, edge, and texture, can be better preserved using the nonlocal differential operator in image processing. We establish an L¹-based nonlocal total variational (NLTVL¹) model based on Retinex theory that can be solved by a fast computational algorithm via the alternating direction method of multipliers. Experiential results demonstrate that our NLTVL¹ method has a good performance on enhancing contrast, eliminating the influence of nonuniform illumination, and suppressing noise. Furthermore, compared with previous works, including traditional Retinex methods and variational Retinex methods, our proposed approach achieves superior performance on edge and texture preservation and needs fewer iterations on recovering the reflectance image, which is illustrated by examples and statistics.

An effective method to develop anatomically real numerical breast phantoms for T1-weighted MR images of different tissue densities is presented. The dielectric properties for breast tissues are calculated and analyzed using different dispersion models (i.e., one- and two-pole cole–cole and Debye models). The method presented in this paper propounds significant improvements in comparison with existing MRI-based numerical phantoms in terms of denoising of images, tissue segmentation, nonlinear mapping of dielectric properties with realistic shapes using all the dispersion models and densitywise classification of phantoms. This method is a multistep approach in which each MRI voxel is mapped with the appropriate dielectric properties according to different dispersion models. The MRI data was collected and interpolated according to the size of the uniform grid for finite difference time domain computations followed by the preprocessing of MR images to enhance them. Thereafter, the voxel intensities were segregated into two groups as adipose and fibro glandular tissues. These tissue intensities were assigned the corresponding dielectric properties. Three-dimensional (3-D) numerical phantoms were created according to all the dispersion models. After the comparison among the models, it has been found that, along with frequency, the dielectric properties vary according to the variation of the dispersion model parameters. It was also observed that the dielectric properties calculated from one-pole cole–cole and two-pole Debye models are more close to the real properties of breast tissues than other models. A generalized method has been defined for developing the 3-D phantoms for all classes of breast according to the inhomogeneity of fibroglandular tissues using the dispersion models. The frequency-dependent and dispersion model parameters-dependent dielectric properties have been assigned to the phantoms. These real-like phantoms after 3-D printing would help researchers working in the field of breast cancer detection studies.

A practical two-level robust hand detection system is developed using the proposed sparse texture features and color–texture features. Traditional dense features such as a histogram of oriented gradients, Gabor, segmentation-based fractal texture analysis, and histogram features are dense in general, with high time-complexity, limiting their use in practical hand detection systems. However, if only the prominent edge or texture parameters of an image are processed, the time complexity of the system can be significantly reduced, along with an increase in performance. Performance of any practical system is most affected by the presence of spurious objects in the background. Proposed approaches use four efficient filtering techniques to extract salient regions of an image retaining significant object-related information, followed by extraction of texture features, as mentioned above. Therefore, in the first stage, 10 sparse variants of these existing texture features are extracted and assessed using five classification models, namely Naïve Bayes, Real AdaBoost, Gentle AdaBoost, Modest AdaBoost, and SVM. In the second phase of the system, a two-level hand detection model is developed using the proposed texture and color–texture features using a sliding window-based SVM classification model. Experimental analysis shows that proposed sparse-based features are not only time efficient but also shows an improvement of 23.4% in the practical two-level detection system over the existing motion and skin filtering-based hand detection system.

Many approaches for background subtraction and people detection have been developed so far. However, the best state-of-the-art methods do not yet give satisfactory results in real transportation environments. Indeed, these latter configurations imply several difficulties such as fast brightness changes, noise, shadows, scrolling background, etc., and a single approach cannot deal with all these. We propose an approach for people segmentation and tracking in videos that is suited for real-world conditions. Our strategy combines several state-of-the-art methods for people detection, silhouette appearance modeling and tracking. Each process also uses its own frame preprocessing pipeline. The optimal combination of the people classifiers used, as well as the optimal parameters of each of the combined methods, being too difficult to be determined altogether, a genetic algorithm is used to determine the optimal classifier parameters and their combination weights. The output of the latter is used as an initialization for a multiframe graph-cut operating on superpixel graphs. Our proposed approach is evaluated on the BOSS European project database that was acquired in moving trains and contains typical scientific locks encountered in real transportation systems.

We present texture-based active contours method for two-phase image segmentation in a statistical framework. The proposed method first combines color, texture, and saliency weight to form an augmented image and introduces the joint distribution of these features into the image likelihood term in the energy function. Second, we use the local probability distribution to obtain a smooth label that can reduce the fragmentation in the initialization and evolution of segmentation contours. Finally, we propose a simple and efficient geometric prior based directly on the level sets and introduce the related spatial constraints into the Bayes inference to estimate the smooth probabilistic label. Therefore, the image is represented by high-dimensional features but segmented in low-dimensional space. Furthermore, evolving of the level-set function and updating of the smooth probabilistic label are run alternately in a fast manner. We experimentally compare our texture-based method with others on complicated natural images and demonstrate its good performance in practice.

Both color and depth information may be deployed to seek by content through RGB-D imagery. Previous works dealing with global descriptors for RGB-D images advocate a decision level merger in which color and depth representations, independently computed, are juxtaposed to pursue a search for similarities. Differently, we propose a “learning-to-rank” paradigm aimed at weighting the two information channels according to the specific traits of the task and data at hand, thereby effortlessly addressing the potential diversity across applications. In particular, we propose a method, referred to as “kNN-rank,” which can learn the regularities among the outputs yielded by similarity-based queries. Another contribution concerns the “HyperRGBD” framework, a set of tools conceived to enable seamless aggregation of existing RGB-D datasets to obtain data featuring desired peculiarities and cardinality.

Histopathologic images are time consuming for both specialist and machine learning methods with their complex structure and huge dimensions. In these cases, delays in the diagnosis of disease occur, as well as the treatment of fewer patients. When the histopathological images are examined at low resolution for shortening the examination time, it is almost impossible to identify the cancerous regions. When examining high-resolution images, it takes a long time to inspect because the image is divided into patches. Despite the fact that fairly fast machine learning methods are offered for the shortening of the analysis period, the number of patches to be examined has a negative effect on the decision time. For this reason, the area under examination needs to be reduced. For this, first of all, the destruction of cell-free areas and then the destruction of areas containing noncancerous cells must be eliminated. An effective and fast method of area reduction is presented for faster analysis and real-time use of histopathological images by machine learning algorithms. A two-step approach is used in the proposed method. In the first step, 3  ×  3 texture properties of images are obtained and discrete wavelet transform is applied. Then, the image is cleaned with simple morphological processes. In the second step, the cleaned image is subjected to a discrete wavelet transform. Thus, the changes in cell-containing regions are captured, and regions that may be dangerous are identified. The proposed method reduced the areas to be examined by 98.5% to 99.5% with 95.33% accuracy.

Unlike the conventional feedforward neural network, an emergent learning technique—which we call extreme learning machine (ELM)—provides a generalized performance of neural network with less user intervention and comparatively faster training. We study ELM with five different activation functions, sigmoidal, sine, hard limiter, triangular basis, and radial basis, for handwritten Indic script identification in multiscript documents. To describe scripts, both script dependent and independent features are computed. For validation, a dataset of 3300 handwritten line-level document images (300 samples per script) of 11 official Indic scripts is used. In our study, we observe that the sigmoidal activation function performs the best regardless of the number of scripts used, i.e., script identification cases: biscript, triscript, and multiscript.

Person reidentification is the process of matching individuals from images taken of them at different times and often with different cameras. To perform matching, most methods extract features from the entire image; however, this gives no consideration to the spatial context of the information present in the image. We propose using a convolutional neural network approach based on ResNet-50 to predict the foreground of an image: the parts with the head, torso, and limbs of a person. With this information, we use the LOMO and salient color name feature descriptors to extract features primarily from the foreground areas. In addition, we use a distance metric learning technique (XQDA), to calculate optimally weighted distances between the relevant features. We evaluate on the VIPeR, QMUL GRID, and CUHK03 data sets and compare our results against a linear foreground estimation method, and show competitive or better overall matching performance.

Character recognition is a well-known area because of its wide applications, including text-searchable documents, digital storage and support, and manual work automation. Recognizing characters is challenging, and, on top of that, writer’s writing style habits make handwritten character recognition even more challenging. This article addresses a set of features and classifiers that are less complex to implement and give significant recognition rates for Hindi language characters. The proposed work is based on two shape descriptors, such as (1) histogram of oriented gradients and (2) geometric moments. These descriptors, when used as features, reflect properties of characters that minimize intraclass variations and maximize interclass variations. The generated feature set is tested using two supervised classifiers, namely support vector machine (SVM) and multilayer perceptron (MLP). A thorough investigation into various evaluation parameters has been presented. Experimental results show that high recognition results are obtained when both shape descriptors are used in combination with an SVM classifier. This technique is evaluated on four publicly available databases, and it achieves a recognition rate of 96.8% on one of the datasets. A comparative analysis of the proposed method with relevant and recent works of this field proves its superiority. This work introduces a much less complex and promising approach toward isolated handwritten character recognition.

Detecting changes of the same scene taken at different time instances is crucial and demanding for medical, remote sensing, infrastructure, agriculture, and planogram compliance applications. We propose a statistical-based approach by exploiting the linear relationship. Initially, region of interest is identified using a graph-cut-based technique followed by geometrical alignment via area-based registration. To perform statistical correlation, we adopt features such as block-wise average coefficient value of the first level of the discrete wavelet transform (DWT-LL1) and the map obtained using hybrid saliency approaches. In the former approach, Pearson’s correlation measure is calculated for the DWT-LL1, and in the latter, PCC has been calculated using the saliency value. Change has been detected using optimal PCC value while minimizing the error rate. Experimental results on datasets reveal that saliency feature and DWT-LL1 perform better for normal and noise corrupted images, respectively. The efficiency of the proposed method is validated by user study with average mean opinion score of 70%. Hybrid saliency-based change detection gives 92.9% of correct classification and hence useful for the vision-based applications like damage detection in a car.

Human action recognition has been widely used in various fields of computer vision, pattern recognition, and human–computer interaction and has attracted substantial attention. Combining deep learning and depth information, this paper proposed a method of human action recognition based on improved convolutional neural networks (CNN). First, we use the depth motion maps to extract the depth sequence features and obtain three projected maps corresponding to front, side, and the top views. On this basis, an improved CNN is constructed to realize the recognition of human action, which uses three-dimensional (3-D) input and two-dimensional process identification to speed up the computation and reduce the complexity of recognition process. We evaluate our approach on two public 3-D action datasets: MSR Action3D dataset and UT-Kinect dataset, and our private CTP Action3D dataset built using Kinect to collect data. The experimental results show that the proposed methods of human action recognition achieve higher average recognition rate of 91.3% on MSR Action3D dataset, 97.98% on UT-Kinect dataset, and the average recognition rate is 93.8% on our CTP Action3D dataset. Furthermore, the trained model on one depth video sequence dataset can be easily generalized to different datasets without changing network parameters.

Digital filters with separable realizations and steerable responses are ideal for processing multidimensional signals, e.g., two-dimensional (2-D) images, in high-throughput sensor systems. Banks of band-pass differentiators with perfectly shaped frequency responses at the dc limit, for impulse responses with vanishing moments—e.g., Savitzky–Golay, Butterworth derivatives, and other maximally flat filters—are appealing because they support a bivariate polynomial interpretation (in Cartesian coordinates) of the input signal; however, for nonpolynomial inputs, the behavior of these directional filters changes with steering angle (i.e., they are anisotropic). Filter banks designed from Gaussian derivatives have almost perfect isotropy; however, they have nonvanishing moments. A procedure for the design of highly isotropic separable filters with steerable responses, vanishing moments, and configurable scale is described in this paper. It may be used to develop both finite-impulse response and low-complexity infinite-impulse response designs with linear-phase noncasual realizations.

Traditional ear recognition methods yield good results in constrained environments. However, the recognition rates fall considerably with the introduction of unconstrained environmental conditions, such as pose, illumination, background clutter, and occlusion. On the other hand, soft biometrics is by definition quite suitable for unconstrained environments and has been exploited in some previous ear recognition studies, but it has focused either on constrained acoustic soft biometrics or body-based soft biometrics. Therefore, instead of dealing with variations caused by unconstrained environmental conditions individually, we investigate the utility of ear-based soft biometrics extracted from unconstrained ear images. The proposed system is based on the fusion of traditional ear recognition and ear-based soft biometric traits using Bayesian fusion. The traditional ear recognition system is based on two texture-based feature extraction methods and support vector machine classification. Whereas the soft biometrics system is based on skin color, hair color, and mole location extracted from unconstrained ear images. The experiments conducted on an unconstrained ear database, Annotated Web Ears, show an average 5.8% improvement in the recognition results when ear-based soft biometric traits are fused with traditional ear recognition.

We focus on saliency estimation in digital images. We describe why it is important to adopt a data-driven model for such an illposed problem, allowing for a universal concept of “saliency” to naturally emerge from data that are typically annotated with drastically heterogeneous criteria. Our learning-based method also involves an explicit analysis of the input at multiple scales, in order to take into account images of different resolutions, depicting subjects of different sizes. Furthermore, despite training our model on binary ground truths only, we are able to output a continuous-valued confidence map, which represents the probability of each image pixel being salient. Every contribution of our method for saliency estimation is singularly tested according to a standard evaluation benchmark, and our final proposal proves to be very effective in a comparison with the state-of-the-art.

Vehicle classification is vital to an intelligent transport system. To obtain a high accuracy, it is the most crucial process to extract reliable and distinguishable features of vehicles. A feature extraction method using a lightweight convolutional network for vehicle classification is proposed. The main contributions are threefold: (1) a lightweight network named LWNet with two convolution layers is proposed to extract the features of the vehicles; (2) Hu moment is integrated with spatial location information to improve its own describing and distinguishing ability; and (3) histogram of oriented gradient (HOG) feature is extracted from the complete image, and then the above two kinds of features are combined with HOG to form the vector. And then, a support vector machine is trained to obtain the classification model. Vehicles are classified into six categories, i.e., large bus, car, motorcycle, minibus, truck, and van. The experimental results have demonstrated that the classification accuracy can achieve 97.39%, which is 3.81% higher than that obtained from the conventional methods. In addition, for this vehicle classification task, the proposed lightweight convolutional network can achieve comparable or even higher performance compared to the deep convolutional neural networks, while the proposed method does not need the support of a graphics processing unit and has much lower complexity without the training process.

The deep convolutional neural network (CNN) has recently shown state-of-the-art performance in many image processing tasks. We examine the use of deep CNN for semantic image segmentation, which separates an input image into multiple regions corresponding to predefined object classes. We follow the most successful deep CNN-based semantic segmentation in recent years and focus on the study of the contextual aspects. To examine the context-awareness, we manually modify the context of the input images and study the effects on the segmentation results. The experiments through systematic context changes show that the model is sensitive to some context changes. We then propose context-changing data augmentation to train context-insensitive models using images solely from the original context. We experimentally validate the effectiveness of the proposed method and summarize its limitations. Finally, we discuss the need of training context-free single-class semantic segmentation models and suggest approaches for it.

To avoid grabbing the unintentional user motion in a video sequence, video stabilization techniques are used to obtain better-looking video for the final user. We present a low power rototranslational solution, extending our previous work specifically addressed for translational motion only. The proposed technique achieves a high degree of robustness with respect to common difficult conditions like noise perturbations, illumination changes, and motion blurring. Moreover, it is also able to cope with regular patterns, moving objects and it is very precise, reaching about 7% of improvement in jitter attenuation, compared to previous results. Overall performances are competitive also in terms of computational cost: it runs at more than 30  frames  /  s with VGA sequences, with a CPU ARM926EJ-S at just 100 MHz clock frequency.

We describe a system for vehicle make and model recognition (MMR) that automatically detects and classifies the make and model of a car from a live camera mounted above the highway. Vehicles are detected using a histogram of oriented gradient detector and then classified by a convolutional neural network (CNN) incorporating the frontal view of the car. We propose a semiautomatic data-selection approach for the vehicle detector and the classifier, by using an automatic number plate recognition engine to minimize human effort. The resulting classification has a top-1 accuracy of 97.3% for 500 vehicle models. This paper presents a more extensive in-depth evaluation. We evaluate the effect of occlusion and have found that the most informative vehicle region is the grill at the front. Recognition remains accurate when the left or right part of vehicles is occluded. The small fraction of misclassifications mainly originates from errors in the dataset, or from insufficient visual information for specific vehicle models. Comparison of state-of-the-art CNN architectures shows similar performance for the MMR problem, supporting our findings that the classification performance is dominated by the dataset quality.

The inherent memory effect of the Grunwald–Letnikov fractional derivatives is combined with the effectiveness of information sets for detecting edges in digital images. Fractional derivatives are utilized in the computation of fractional gradients, which are further processed using information sets for the proposed edge detector to extract more edges than possible with the traditional edge detectors. In the proposed approach, first the gradient operators of the Sobel mask are converted into the fractional form and convolved with the given gray image. In the next step, the histogram of the fractional gradients is fuzzified using the Gaussian membership function and the sigmoidal membership function. The optimal parameters for these membership functions are selected through a grid search method based on the information set-based edge strength factor ESf. In the final step, defuzzification is performed to obtain the final edge maps. By plotting the edge maps, analyzing the boundary information, and Pratt’s figure of merit scores for images in the Berkeley segmentation dataset, it is observed that the resulting edge maps of the proposed edge detector contain more quantitative and qualitative information than that of traditional edge detectors even in the presence of noise.

Recently, a number of graph-based approaches have been proposed to detect salient regions in images. Although the graph is essential for these approaches, the graph construction method has not been studied in much detail. We propose a graph construction method that makes better use of multiple Gestalt principles. Specifically, spatial proximity, color similarity, and texture similarity between image regions are employed to choose different edges of the graph. Furthermore, the confliction among multiple Gestalt principles is solved using a primal rank support vector machines algorithm to compute the edge weights. Our experimental results on various salient region detection databases with comparisons to representative approaches demonstrate that the proposed graph construction method helps to detect salient objects in images.

Many approaches have been suggested for automatic pedestrian and car detection to cope with the large variability regarding size, occlusion, background variability, and aspect. Current deep learning-based frameworks rely either on a proposal generation mechanism (e.g., “faster R-CNN”) or on inspection of image quadrants/octants (e.g., “YOLO”), which are then further processed with deep convolutional neural networks (CNN). We analyze the discriminative generalized Hough transform (DGHT), which operates on edge images, for pedestrian and car detection. The analysis motivates one to use the DGHT as an efficient proposal generation mechanism, followed by a proposal (bounding box) refinement and proposal acceptance or rejection based on a deep CNN. We analyze in detail the different components of our pipeline. Due to the low false negative rate and the low number of candidates of the DGHT as well as the high accuracy of the CNN, we obtain competitive performance to the state of the art in pedestrian and car detection on the IAIR database with much less generated proposals than other proposal-generating algorithms, being outperformed only by YOLOv2 fine-tuned to IAIR cars. By evaluations on further databases (without retraining or adaptation), we show the generalization capability of our pipeline.

We developed a technique, using fractal complexity analysis of optical flow in two-dimensional (2-D) videos, to characterize information content in observed motion. Several lines of evidence demonstrate that visually available properties of motion can characterize the state of a system. This paper will describe the method used and will present a test case regarding the accuracy of the method. An analytical comparison of simple human movement (arranging items on a table) and American Sign Language (ASL) will be given as an example application. The normalized spectral density in the range of 0.1 to 15 Hz indicated significantly higher fractal complexity in the optical flow of ASL video data, indicating that information content in 2-D video data can be characterized using complexity analysis of optical flow. The technique used for quantification of information content in visual motion data is likely to be applicable for distinguishing biological versus nonbiological motion in 2-D video data, making inferences about the states of biological objects from the dynamics of optical flow, and in assessing likelihood of information content in a video stream.

We present GRAPHJ, a forensic tool for handwriting analysis. The proposed tool has been designed to implement the real forensic protocol adopted by the “Reparto Investigazioni Scientifiche” of Carabinieri, Italy. GRAPHJ allows the examiner to (1) automatically detect text lines and words in the document, (2) search for a specific character and detect its occurrences in the handwritten text, (3) measure different quantities related to the detected elements (e.g., heights and widths of characters), and (4) generate a report containing measurements, statistics, and the values of all parameters used during the analysis. The generation of the report helps to improve the repeatability of the whole process. The experiments performed on a set of handwritten documents show that GRAPHJ allows one to extract quantitative measures comparable to those acquired manually by an expert examiner. We also report a study on the use of the relative position of the superscript dot of the “i” characters as a parameter to infer the identity of the writer. The study has been performed using GRAPHJ and illustrates its value as a forensic tool.

Deep learning is revolutionizing the already rapidly developing field of computer vision. The convolutional neural network (CNN) is a state-of-the-art deep learning tool that learns high level features directly from a huge dataset of labeled images. In document image processing, ink analysis allows for determination of ink age and forgery and identification of pen or writer. The spectral information of inks in hyperspectral document images provides valuable information about the underlying material and thus helps in identification and discrimination of inks based on their unique spectral signatures even if they have the same color. Ink mismatch detection is a key step in document forgery detection. Although various ink mismatch detection techniques are available in the recent literature, there is a constant need for development of more accurate and effective methods to empower automated document forgery detection. A state-of-the-art deep learning method for ink mismatch detection in hyperspectral document images is proposed. The spectral responses of ink pixels are extracted from a hyperspectral document image, reshaped to a CNN-friendly image format and fed to the CNN for classification. The proposed method effectively identifies different ink types in a hyperspectral document image for forgery detection and achieves an overall accuracy of 98.2% for blue and 88% for black inks, which is the highest accuracy among the latest techniques of ink mismatch detection on the UWA Writing Ink Hyperspectral Images (WIHSI) database and differentiates between the highest number of inks mixed in unbalanced proportions in a hyperspectral document image. Furthermore, a detailed discussion on selection of appropriate CNN architecture and classification results are presented in this paper along with comparison with the former methods of ink mismatch detection. This research opens a new window for research on automated forgery detection in hyperspectral document images using deep learning.

The state-of-the-art approaches of copy image retrieval are based on the bag-of-visual-words model, which represents an image with a set of visual words obtained by quantizing local features. However, the quantization process reduces local features’ discriminative power and thus causes many false matches of local features between images. As a consequence, this brings down the effectiveness of copy image retrieval in large-scale image dataset. In order to handle this problem, postverification methods have been proposed to reject false matches. Previous works of the postverification method focused mainly on geometric relationship consistency among matches of local feature between query image and its candidate for rejecting false candidates. The variation consistency of local feature’s attributes is proposed to verify if two pairs of matches are consistent. The matching reliability of local features can be measured by a voting-based method, which is based on the number of consistent matches between two images. This method can easily integrate more attributes of local feature, such as dominant orientation, position, and scale, rather than position of local feature. Experiments on the large-scale datasets demonstrate the effectiveness and efficiency of the proposed approach and show it outperforms the state-of-the-art postverification approaches.

The saliency object detection is a hot topic of computer vision. Traditional saliency detection methods are overly dependent on handcrafted low-level features. The saliency detection methods based on deep learning can effectively solve the problem, which extracts high-level features automatically. However, there are some noises in the extracted high-level features that affect the detection performance. We propose a deep learning framework for saliency detection based on global prior and local context. First, we use feature maps generated by combining some middle-level features as the input of global-prior-based deep learning model, which can reduce the interference of distracting feature information for the saliency detection. Then, two deep learning models use respectively local contexts of color image and depth map as input, which combine global prior to generate the initial saliency map. Finally, the optimized saliency map can be obtained based on spatial consistence and appearance similarity. Experiments on two publicly available datasets show that the proposed method performs better than other nine state-of-the-art approaches.

Deep learning-based methods have shown great advantage in image super resolution (SR) compared to conventional methods. We propose a deep learning-based method named multichannel densely connected convolutional network (MDesNet) for single image SR. The proposed MDesNet first decomposes the input image into intrinsic mode functions (IMFs) and residue based on bidimensional empirical mode decomposition. Then, some densely connected convolutional subnetworks are used to exploit deep features from IMFs and residue. Since low- and high-frequency components have been decomposed into different IMFs, we can obtain accurate features from IMFs and residue by applying relatively shallow subnetwork so as to avoid vanishing gradient problem. All the features are fused in feature fusion subnetwork and output the reconstructed high-resolution image. We have compared our method with some recently proposed learning-based methods. Experimental results show that our proposed method gains the best overall quality.

Existing algorithms for road object detection currently exhibit three major drawbacks: slow detection speed, difficulty recognizing small objects, and low detection precision. To address these problems, we propose an algorithm called fast and accurate road object detection (FAROD) based on the faster region-based convolutional neural networks (R-CNN). The FAROD has two parts: the accurate object proposal network (AOPN) and object attribute learning network (OALN). The AOPN accurately generates road object-like regions in real time, and the OALN detects the corresponding classifications and locations of road object-like regions. To promote greater accuracy for small object detection, we introduce a deconvolution structure and loss functions for the AOPN and OALN. We improve computation speed by alternately optimizing and jointly training the networks. Our experimental results show that FAROD offers significant improvement over existing road object detection algorithms in dealing with difficult images, especially road images within small objects. The test results of the mean average precision improved by 4.5%, and the detection time reduced by one-third than that of the state-of-the-art object detection algorithm faster R-CNN.

Image deblurring with impulse noise is a typical ill-conditioned problem that requires regularization techniques to guarantee stable and high-quality imaging. According to the statistical properties of impulse noise, an L1-norm data fidelity term and a total variation (TV) regularizer have been combined to contribute a popular regularization model. However, traditional TV-regularized variational models usually suffer from staircase-like artifacts in homogenous regions resulting in visual quality degradation. To eliminate undesirable artifacts, we propose a high-order variational model by replacing the TV with a detail-preserving total generalized variation (TGV) regularizer. To further enhance imaging performance, the spatially adaptive regularization parameters are automatically selected, based on local image features to promote the high-order TGV-regularized variational model. The resulting nonsmooth optimization problem is effectively handled using the alternating direction method of multipliers-based numerical method. The proposed variational model has the capacity to remove blurring and impulse noise effects while maintaining fine image details. Comprehensive experiments were conducted on both gray and color images to compare our proposed method with several state-of-the-art image restoration methods. Experimental results have demonstrated its superior performance in terms of quantitative and qualitative image quality evaluations.

Visual simultaneous localization and mapping (SLAM) based on RGB-D data has been extensively researched in the past few years and has many robotic applications. Most of the state-of-the-art approaches assume static environments. However, the static assumption is not usually true in real world environments, dynamic objects can severely degrade the SLAM performance. In order to reduce the influence of dynamic objects on camera pose estimation, this paper proposes an approach that uses static point and line features. Static weights of point and line features indicating the likelihood of features being part of static environment are estimated. According to the calculated static weights, the data associated with dynamic objects are filtered out. The remaining static point and line features are considered inputs for refined pose estimation. Experiments are conducted with challenging dynamic sequences from TUM RGB-D dataset. The results demonstrate that the proposed approach is able to effectively improve the accuracy of RGB-D SLAM in dynamic environments.

This research work proposes a robust image watermarking technique for achieving robustness against desynchronization attacks using a geometric distortion correction (GDC) approach. The GDC is performed using support vector regression (SVR). The proposed algorithm uses the combined domain of lifting wavelet transform as well as singular value decomposition method. Incorporation of a support vector machine (SVM) has been done during watermark extraction to achieve enhanced robustness against diverse attack conditions. The present results show that the proposed technique achieves robustness against geometric and nongeometric attacks blindly by integrating SVR-based GDC technique with SVM-based binary watermark extraction. Again, the SVR-based blind GDC technique can estimate the geometric distortion of any arbitrary attacked image irrespective of the content present in it. Finally, comparative analysis with different state-of-the-art techniques suggests that the proposed technique provides improved performance in most of the cases.

Group-based sparse representation modeling of natural images codes each patch in a group as a sparse linear combination from an over-complete dictionary and assumes that the coding coefficients of each patch have a common set of nonzero support. This model has shown great success in image restoration (IR) applications; however, current group-based sparse models are simple extensions of traditional L0 or L1 sparse models and lack spatial adaption and principled fashion. We propose a group-based sparse model named as simultaneously sparse coding with Laplacian scale mixture (LSM) prior through probabilistic approach. In this model, the representation coefficients of each patch in group are characterized by the same LSM prior, which is dominated by the shared scale variables. By imposing a hyperprior distribution over the scale variables, the sparse coefficients and shared scale variables can be jointly estimated via alternating optimization. Furthermore, we generalize this model to process more general IR problems under the plug-and-play alternating direction method of multipliers framework. Extensive experiments on image denoising, deblurring, and single-image super-resolution demonstrate that the proposed method achieves notable objective and subjective improvements over many state-of-the-art restored methods.

To solve the problems of occlusion and fast motion of small targets in unmanned aerial vehicle target tracking, an adaptive algorithm that fuses the improved color histogram tracking response and the correlation filter tracking response based on multichannel histogram of oriented gradient features is proposed to realize small target tracking with high accuracy. The state judgment index is used to determine whether the target is in a fast motion or an occlusion state. In the fast motion state, the search area is enlarged, and the color optimal model that suppresses the suspected area is used for rough detection. Then, redetection in the location of multiple peaks in the rough detection response is carried out using the correlation filter to accurately locate the target. In an occlusion state, the model stops updating, the search area is expanded, and the current color model is used for rough detection. Then, redetection in the place of multiple peaks in the rough detection response is carried out using the correlation filter to accurately locate the target. Experimental results show that the proposed method can track small targets accurately. The frame rate of the proposed method is 40.23 frames/s, indicating usable real-time performance.

False cracks, such as split joints and scratches, have macroscopic geometry that is similar to real cracks, which can influence the crack detection efficiency for a concrete bridge. To solve this problem, a crack detection algorithm based on the mesoscale geometric features of cracks is proposed. Through the mesoscale analysis of concrete crack formation and propagation mechanisms, it is found that a concrete crack propagates at the interface between aggregates and mortar and usually has a meandering path, whereas a false crack’s path is usually smooth or even straight. Thus the path smoothness of a crack candidate is chosen as the detection basis. The algorithm extracts a crack candidate with conventional methods, and also its skeleton for representing the path. In addition, the feature parameters are designed to quantify the path smoothness. Moreover, a back propagation neural network (BPNN) and a support vector machine (SVM) for the classification of crack candidates are trained using the proposed feature parameters as the input. Experimental results show that the classification rate of the BPNN trained by new features is 91.7%, which is better than the BPNNs trained by conventional features. The classification rate of the SVM is 93.3%, which is more suitable for engineering in small size samples.

With video-assisted surgery devices becoming more common in all fields of diagnostics and therapy, the question of how well such systems are able to reproduce surface colors of organic tissue arises, especially in cases where a proper distinction of different kinds of tissue—not only through their texture but also through their color—might be crucial for the success of a surgery. Since modern devices are usually made of a highly efficient, multispectral LED light source in combination with some light-guiding structures and a digital camera system, an approach of optimizing these systems’ color reproduction properties based on the estimation of in-situ spectral reflectances is proposed. Following the International Organization for Standardization (ISO) standard procedure of colorimetric characterization, an initial color correction matrix was determined first by solving a linear least-mean-squares optimization problem for a small set of artificial color samples mapping the corresponding camera responses onto the samples’ tristimulus values. This initial matrix was then used as a starting point for a second, nonlinear optimization, which makes use of the estimated reflectance spectra in order to minimize the average squared color difference between the color corrected and the actually perceived tristimulus values of the individual organic tissue samples. Compared to the ISO standard, which only knows simple color patches to be used for the nonlinear optimization, a significant enhancement in the color reproduction of organic tissue could be confirmed with the newly proposed method being applied instead.

This paper analyzes how an experimenter can balance errors in subjective video quality tests between the statistical power of finding an effect if it is there and not claiming that an effect is there if the effect is not there, i.e., balancing Type I and Type II errors. The risk of committing Type I errors increases with the number of comparisons that are performed in statistical tests. We will show that when controlling for this and at the same time keeping the power of the experiment at a reasonably high level, it is unlikely that the number of test subjects that are normally used and recommended by the International Telecommunication Union (ITU), i.e., 15 is sufficient but the number used by the Video Quality Experts Group (VQEG), i.e., 24 is more likely to be sufficient. Examples will also be given for the influence of Type I error on the statistical significance of comparing objective metrics by correlation. We also present a comparison between parametric and nonparametric statistics. The comparison targets the question whether we would reach different conclusions on the statistical difference between the video quality ratings of different video clips in a subjective test, based on the comparison between the student T-test and the Mann–Whitney U-test. We found that there was hardly a difference when few comparisons are compensated for, i.e., then almost the same conclusions are reached. When the number of comparisons is increased, then larger and larger differences between the two methods are revealed. In these cases, the parametric T-test gives clearly more significant cases, than the nonparametric test, which makes it more important to investigate whether the assumptions are met for performing a certain test.

In microvision, the efficiency of image analysis is reduced by low signal-to-noise ratios, large amounts of data, and target over-concentration. Processing speeds can be optimized by introducing a biomimetic visual attention mechanism; however, the most mature attention mechanisms are not fully applicable to microvision environments. They rely on pixels, and this reliance significantly limits processing speeds. Therefore, we propose a fast attention mechanism based on the visual-entropy value and a method of evaluating the difference-entropy convergence of the target contour. We have performed a theoretical analysis, a large quantity of data sampling, and comparisons of multiple experiments. Given a fixed field of view, the visual-entropy attention mechanism can quickly locate the target area, and thus is highly suitable to a microvision environment. The difference-entropy convergence evaluation method can effectively distinguish a special, damaged target.

We address the problem of matching stereo-omnidirectional images acquired by a catadioptric stereo system. The proposed approach deals with omnidirectional stereo image sequences, which allows taking advantage of the temporal information in both rectification and stereo matching steps. First, spherical edge points detection is performed on the spherical images in order to compute the disparity map only on significant features, i.e., edge points. Second, a spherical rectification is performed on the three omnidirectional pairs that are involved in the disparity map computation at the current frame, such as the current pair, the current left image, and its preceding one, and the current right image and its preceding one. Third, dynamic programming-based stereo matching is applied in each of the three stereo pairs. The disparity maps computed at the preceding frame are exploited to deduce the disparity range that will be used by the dynamic programming. Fourth, the three disparity maps provided by the last step are combined together into one 360-deg disparity map. The proposed method has been tested on a high-resolution omnidirectional images captured with a stereo pair of catadioptric cameras, and the obtained results are satisfactory.

Depth images captured from modern depth cameras generally suffer from low spatial resolution, noise, and missing regions. These kinds of images cannot be used directly in applications related to depth images, e.g., robot navigation, 3DTV, and augmented reality, which basically need high-resolution input images with no noise o missing regions to function properly. To address the problem of low spatial resolution, noise degradation, and missing regions in depth images, we propose methods based on a guidance color image for depth reconstruction (DR) from sparse depth inputs and depth image super-resolution (SR). We also suggest a scenario wherein these problems can be integrated and addressed simultaneously. Further, we also demonstrate applications of the proposed approach for depth image denoising and depth image inpainting. In our approach, the guidance color image is used for obtaining the segment cues by applying mean-shift (MS) or simple linear iterative clustering (SLIC) segmentation on it. These strong segment cues help in aiding the DR and SR problems by considering the corresponding segments in the input depth image, and estimate the unknown pixels by either plane fitting or median filling approaches. Furthermore, we explore both direct and pyramidal (hierarchical) approaches for SR and DR-SR for higher upsampling factor. As such, our approaches are relatively simpler than some of the contemporary methods, yet the experimental results of the proposed methods show superior performance as compared with some other state-of-the-art DR and SR methods.

A content-based image retrieval system is proposed using an optimized weighted feature voting technique with multifeatures and multidistance measures. The proposed system consists of two phases: enrollment phase and querying phase. In enrollment phase, six features have been extracted from each enrollment image. These features are color histogram, edge histogram, edge direction histogram, hierarchical annular histogram, Gabor filter, and co-occurrence matrix. The extracted features are weighted, and feature weights are optimized using the four optimization techniques. Particle swarm optimization, ant lion optimizer, bird swarm algorithm, and whale optimization algorithm are applied and compared to decide the suitable technique. In querying phase, the same features are extracted from the query image and comparing these features to the extracted features from the database images through the matching measures. Three distance measures have been used and compared for the matching process: histogram intersection, Euclidean distance, and cosine distance. Based on these distances and the feature weights, the class of the query image is identified using the weighted feature voting mechanism. Finally, the related images are retrieved based on the feature that returned the maximum number of images in the identified class. For the validation and performance evaluations of the proposed system; Wang, Caltech101, and UW datasets are used. The experimental results show that the proposed method achieves improved precision in comparison with existing methods with 93% for Wang database, 92% for UW database, and 94% for Caltech101 database.

During the tracking, kernelized correlation filters may fail as the target is occluded seriously and goes out of view. To solve this problem, a long-term visual tracking algorithm based on adaptive correlation filters is proposed. First, we learn two correlation filters to locate the target and estimate the target scale, respectively. Meanwhile, we learn an independent target appearance correlation filter conservatively updated to know the occlusion degree of the target. Second, we combine the Kalman filter to predict and the support vector machine detector to redetect when tracking failure occurs, caused by the target undergoing severe occlusion or disappearing in the camera view. Third, to solve model drifts due to serious appearance changes of the target, we apply an adaptive model updating strategy to update the correlation filters and classifier. Extensive experimental results on the OTB2013 benchmark dataset demonstrate that our proposed method achieves the excellent overall performance against the nine state-of-the-art methods while running efficiently in real time.

The ability for computers to target and track objects within video sequences has attracted increasing amounts of attention in recent years. Many tracking algorithms adopt correlation filtering-based algorithms, in which targeting of features in the previous video frame is essential to predicting the target’s position in the next frame. A serious problem in tracking is drift. If drift occurs in the present frame, it becomes increasingly difficult to correct this error in later frames, which can lead to loss of the tracked target and greatly reduced tracking accuracy and stability. To solve this problem and improve tracking performance, we propose an algorithm that incorporates deep features. A sparse representation method is used to filter deep features and improve their complexity. Then, a Siamese network is used to judge the similarities between the target and template and determine a confidence level. Next, a single shot multibox detector (SSD) is introduced to support tracking, which is activated when the confidence level falls below a certain threshold. The correct target can be found by scanning the present frame using the detector and then updating the template. The confidence level is then reduced and the SSD continues working to resolve the drift. Tracking stability and accuracy are greatly improved after the redetection process. We use large-scale benchmark datasets to demonstrate that the proposed approach performs better than existing state-of-the-art approaches.

Objective video quality metrics are designed to be as reliable as the subjective quality assessments on which they are calibrated and validated. However, existing standard methodologies for subjective video quality assessment provide low reliable results for some conditions. We investigate whether an extension of the quality ruler experimental methodology, originally defined for images and shown to be more reliable than, e.g., standard single stimulus (SS) methods, can be adapted to reliably assess the quality of videos. The video quality ruler methodology allows subjects to assess video quality using a set of reference anchor images (the ruler), spanning a wide range of quality altogether, but closely spaced in function of quality one from the other. Subjects are asked to compare the quality of the displayed test video with the quality of these anchor images, displayed on a tablet, and indicate which of the reference images matches in quality the test video. As a result, the video quality assessment task is reduced to a set of visual comparisons between video and reference image quality. We describe how to adapt the original quality ruler methodology to video quality assessment, and we compare the proposed methodology with two other, widely used experimental methodologies: the single stimulus (SS) and the double stimulus (DS) method. Our results show that video quality ruler is a reliable method to assess video quality according to a multitude of criteria.

Noodle is a type of pasta, mainly composed of wheat flour (WF), widely consumed due to its easy preparation. Recently, there has been a growing concern in the food industry about nutritionally enriched processed wheat products, and the analytical methods used to characterize these products. We implemented a computer vision system (CVS) using image analysis and prediction algorithms, to predict three different components in pasta: hydrolyzed soy protein (HSP), fructo-oligosaccharide (FOS), and WF. Pasta samples used in the experiments were produced with 12 different combinations of these components, varying the amounts of HSP, FOS, and WF. Microscopy images of samples were acquired, preprocessed, and segmented to extract image features. We investigated 56 image features from four types (color, intensity, texture, and border) along with four machine learning algorithms (gradient boost machine, multilayer perceptron artificial neural network, support vector machine, and random forest) and partial least-squares to predict the quantity of noodle components. Accurate results were obtained for HSP and WF, with coefficient of regression (R2) of 0.82 and 0.75, and root mean square error (RMSE) of 0.12 and 0.15, respectively. On the other hand, FOS was not accurately identified (R2  =  0.39, RMSE  =  0.21). The results support the potential application of CVS in the processing industry for noodle production.

Resulting from nonuniform illumination and finger shift, current phalangeal joint locating methods cannot segment adequate vein pattern information and locate phalangeal joint reliably. Hence, we propose a robust gradient-based approach to detect phalangeal joint accurately and achieve sufficient valuable information. The detection accuracy is 98.23% and 99.32% in two datasets consisting of abnormal finger-vein images, respectively. Furthermore, the captured image contains noise and irregular shading inevitably, which can seriously affect further comprehensive study of finger-vein characteristics. Thus, we propose a fast and self-adaptive recovery model, which is built on atmospheric scattering model and finger-vein imaging principle, to extract finger-vein patterns reliably. Finally, a more reliable template matching method is used for finger-vein patterns matching. The resulting equal error rate is 2.95% and 2.83% in two databases, which consist of 1260 and 384 finger-vein images, respectively.

Gender classification, a two-class problem (male or female), has been the subject of extensive research recently and gained a lot of attention due to its varied set of applications. The proposed work relies on individual facial features to train a convolutional neural network (CNN) for gender classification. In contrast with previously reported results that assume the facial features are independent, we consider the facial features as correlated features by training a single CNN that jointly learns from all facial features. In terms of accuracy, our results either outperform, or are on par with, other gender classification techniques applied to three different datasets namely specs on faces, groups, and face recognition technology. In terms of performance, the proposed CNN has significantly fewer parameters as compared with other techniques reported in the literature. Our learnable parameters are fewer than those required in techniques reported in recent work, which enables them to make the network less sensitive to over-fitting and easier to train than techniques that use different CNNs for each facial feature as reported in the literature.

Over the past couple of years, deep learning in convolutional neural networks (CNNs) has proven to be extremely powerful in image processing tasks including image denoising. We deal with digital image denoising based on deep learning. In this regard, dilated CNNs have been successful in achieving impressive denoising quality by obtaining a considerably effective trade-off between receptive field size and network depth, and thus reducing the computational intensity of the CNN architecture. Such networks in general use rectifier linear units as the activation function for introducing nonlinearity into the network. We propose a dilated CNN with exponential linear units for image denoising. Our aim is to enhance the performance of dilated networks in image denoising and develop an effective denoising CNN architecture by optimizing computational intensity. We experiment with a (conventional) 10-layer network, as well as a 5-layer version of the proposed model. It is observed that the proposed 5-layer network succeeds to achieve state-of-the-art denoised image quality as well as high computational efficiency. Our experimental results prove the efficiency of the proposed models in image denoising as well as complexity reduction.

Latent Dirichlet allocation is the prevalent topic model and performs well for image classification. However, it ignores visual word spatial information, which affects topic assignment accuracy. This paper proposes an effective topic model framework based on spatial pyramids including visual word regional information: spatial topic pyramid model (STPM). STPM divides the images into different scale regions and uses the regional topic distributions to represent the images. The regional topic distributions effectively represent image characteristics, because they include global information (regarding the image as a single region) and the regional relationships of visual words in different scale regions. Since the pyramid layers are independent, different topic models and parameters can be used for different scale layers. It makes STPM flexible and easily extensible.

Many factors lead to spatially varying blur kernels in a blurred image, such as camera shake, moving objects, and scene depth variation. The traditional camera shake removal methods either ignore the influence of varying depth values or object motion in dynamic scenes, while the methods not limited to removing camera shake always make simple assumptions about camera motion trajectory. We consider these factors in a unified framework, with the aid of an alternate-exposure capture strategy and simultaneously recorded inertial sensor readings. The inertial measurements relate the long-exposed blurred image to preceding and succeeding short-exposed noisy images. The special exposure arrangement effectively addresses the problem inherent in reconstructing camera motion from inertial measurements. In addition, the noisy image pair bracketing the blurred image is used for motion detection and initial depth map estimation, making the proposed method free of user interaction and additional expensive devices. Contrary to previous methods that individually parametrize the motion blur of the moving foreground layer and the static background layer, we exploit the fact that camera shake has a global influence to decompose the motion of the foreground layer such that a more tight constraint between the motion of layers is established. Given the motion and image data, we propose a single-energy model and minimize it using alternating optimization to estimate the spatially varying motion blur and the latent sharp image. Comparative experimental results demonstrate that our method outperforms conventional camera motion deblurring and object deblurring methods on both synthetic and real scenes.

Most of existing approaches in keyword spotting are system-oriented, which did not take into consideration the user’s needs. However, a user may want to find words, sentences, or texts that match his target image in his mind. The challenge here is how to formulate one’s mental image to reach what he is looking for. The key idea is to design and build a model that properly adapts the human reasoning in information searching through an interactive process. We propose a mental model for handwritten keyword spotting based on relevance feedback, feature weighting, and optimization. This model meets simultaneously the user’s needs, the system behavior, and the user–system relationship. In an appropriate feature space, the query is progressively built from user-supplied keywords, old queries, and spotted images. This dynamic process not only converges toward the desired word images, but also helps the hesitant user to clarify progressively what he is looking for. The proposed model was showcased via a user-friendly interface, which we tested including real users on three well-known handwritten datasets; Institute for Communications, Braunschweig University, Germany/École Nationale d’Ingénieurs de Tunis, Tunisia, Institut für informatik und Angewandte Mathematik, and George Washington. The experimental results show that the proposed method provides promising scores with a reasonable number of refinements.

Widespread deployment of biometric systems has made researchers focus on its vulnerability to even the simplest attempts to breach security through presentation attacks, which involve presenting an artefact (fake sample) to the biometric sensor. We present an approach for presentation attack detection that enables a palm-vein sensor to provide effective countermeasures against these attacks. Our method is based on analysis of noise residual computed from the acquired image. The palm-vein image acquired by the sensor is denoised through median filtering, a well-known nonlinear technique for noise reduction. Subsequently, a noise residual image is obtained by subtracting the denoised image from the acquired image. The local texture features extracted from the noise residual image are then used to detect the presentation attack by means of a trained binary support vector machine classifier. We have performed evaluations on a publicly available palm-vein dataset consisting of 4000 bona fide and fake images collected from 50 subjects in two different sessions. Our approach consistently achieves a perfect average classification error rate of 0.0%. The results also suggest that the proposed approach is more effective than state-of-the-art methods in palm-vein antispoofing.

A screen content coding (SCC) extension to high-efficiency video coding has been developed to improve the coding efficiency for videos with computer-generated text and graphics. It employs additional coding tools such as intrablock copy and palette modes for intraprediction. Although these modes can provide high coding efficiency for screen content videos, they cause an increase in computational complexity. This paper proposes a fast intraprediction algorithm for SCC by content analysis and dynamic thresholding. To skip unnecessary modes for a coding unit (CU), a rough CU classification is performed as a preprocessing step. Then, two early mode decision techniques are proposed by performing a fine-granular CU classification and deriving a content-dependent rule with adaptive thresholding based on the background color. To make early termination of CU partitions, another content-dependent rule with adaptive thresholding according to rate-distortion cost is derived to skip unnecessary partitions. In addition, a scene change detection method is adopted to update all content-dependent thresholds for different scenes in a sequence. Experimental results show that the proposed algorithm achieves 35.95% computational complexity reduction on average with 1.20% Bjøntegaard delta bitrate loss under all intraconfiguration, which outperforms all state-of-the-art algorithms in the literature.

Conventional learning-based methods for super-resolution (SR) have proven efficient for their potential of recovering the local details on low-resolution (LR) sharp images. Adaptive Gaussian mixture models (AGMMs) specialized for SR are developed based on the assumption that the corresponding high- and low-resolution image patches can be jointly modeled by GMMs. As a regularization term, the AGMMs are then embedded into the whole image SR models. The embedded AGMMs (EAGMMs) outperform the simple whole image SR models greatly. In addition, EAGMMs can also serve as a common framework for SR on LR blurred images. We have evaluated both AGMMs and EAGMMs on a variety of test images, and obtained very promising and competitive performance. In most cases, AGMMs and EAGMMs generate better results than the state-of-the-art SR methods.

In a biometric verification system, leakage of biometric data leads to permanent identity loss since original biometric data are inherently linked to a user. Further, various types of attacks on a biometric system may reveal the original template and utilize in other applications. To address these security and privacy concerns, cancelable biometric has been introduced. Cancelable biometric derives a protected template from the original biometric template using transformation functions and performs comparison between templates in the transformed domain. Recent approaches toward cancelable fingerprint generation either rely on aligning minutiae points with respect to singular points (core/delta) or utilize the absolute coordinate positions of minutiae points. We propose a noninvertible ridge feature transformation method to protect the original fingerprint template information. The proposed method partitions the fingerprint region into a number of sectors with reference to each minutia point employing a ridge-based co-ordinate system. The nearest-neighbor minutiae points in each sector are identified, and ridge-based features are computed. Further, a cancelable template is generated by applying the Cantor pairing function followed by random projection. We have evaluated our method with FVC2002, FVC2004, and FVC2006 databases. It is evident from the experimental results that the proposed method outperforms existing methods in the literature. Moreover, the security analysis demonstrates that the proposed method fulfills the necessary requirements of noninvertibility, revocability, and diversity with a minor performance degradation caused due to cancelable transformation.

Nowadays, important scientific works are oriented to develop optimal palmprint representations, especially used in the palmprint recognition field. These representations must be discriminant, robust, compact, and easy-to-implement in order to ensure the best performance in terms of accuracy, computation cost, and storage requirement. In this way, our paper presents a multiscale analysis framework for efficient palmprint representation called discriminant Gabor Lpq spatial pyramid histogram, which significantly relies on Gabor wavelet, local phase quantization (LPQ) descriptor, and spatial pyramid histogram (SPH) method. First, the Gabor wavelet function with two scales and four orientations is used to capture the local structure in palmprint images. Second, the LPQ operator is applied to the response images of Gabor filter to get label LPQ images in order to fully explore the blur invariant property and the texture information. This is conducted in multiscale and multidirection space. Third, for each of them, the SPH of vertical decomposition is applied to obtain local palmprint feature descriptors. Next, the obtained histograms are normalized. Then the global representation of the palmprint image is obtained by concatenating all the local feature descriptors. After that, the discriminant representation of the palmprint image is constructed using whitened linear discriminant analysis. Finally, the K-nearest neighbor classifier is used for identification. Experiments, conducted on three palmprint databases (PolyU2D, PolyU 2D/3D, and IITD), show that the proposed method provides a significant performance improvement compared to the state-of-the-art and recently proposed methods in terms of accuracy.

Most of the current semantic segmentation approaches have achieved state-of-the-art performance relying on fully convolutional networks. However, the consecutive operations such as pooling or convolution striding lead to spatially disjointed object boundaries. We present a dense boundary regression architecture (DBRS²), which aims to use boundary cues to aid high-level semantic segmentation task. Specifically, we first propose a multilevel guided low-level boundary (MG-LB) learning method, where we exploit multilevel convolutional features as guidance for low-level boundary detection. The predicted MG-LB boundaries are used to enable consistent spatial grouping and enhance precise adherence to segment boundaries. Then, we present a significant global energy model based on boundary penalty and appearance penalty, which are respectively defined on the predicted boundaries and coarse segmentations obtained by the DeepLabv3 network. Finally, the refined segmentations are regressed by minimizing the global energy model. Extensive experiments over PASCAL VOC 2012, ADE20K, CamVid, and BSD500 datasets demonstrate that the proposed approach can obtain state-of-the-art performance on both semantic segmentation and boundary detection tasks.

We propose a synthetic aperture radar (SAR) automatic target recognition (ATR) method by jointly classifying three complementary features based on multitask compressive sensing (MtCS). The principal component analysis features, elliptical Fourier descriptors and the azimuthal sensitivity image, are extracted or constructed to describe the intensity distribution, target shape, and electromagnetic characteristics of the original SAR images, respectively. The three features describe the original SAR image from different aspects, thus their joint use can provide more discrimination for distinguishing different classes of targets. Afterward, the three features are jointly classified based on MtCS, which can properly represent individual tasks, and also exploit their inner correlations. Therefore, it is promising that the discriminability of different features can be better exploited to improve the ATR performance. Extensive experiments are conducted on the moving and stationary target acquisition and recognition dataset under both the standard operating condition and several typical extended operating conditions, i.e., configuration variance, large depression angle variance, noise corruption, and partial occlusion. The results demonstrate the effectiveness and robustness of the proposed method in comparison with several state-of-the-art SAR ATR methods.

Advances in imaging and display engineering have given rise to new and improved image and video applications that aim to maximize visual quality under given resource constraints (e.g., power, bandwidth). Because the human visual system is an imperfect sensor, the images/videos can be represented in a mathematically lossy fashion but with enough fidelity that the losses are visually imperceptible—commonly termed “visually lossless.” Although a great deal of research has focused on gaining a better understanding of the limits of human vision when viewing natural images/video, a universally or even largely accepted definition of visually lossless remains elusive. Differences in testing methodologies, research objectives, and target applications have led to multiple ad-hoc definitions that are often difficult to compare to or otherwise employ in other settings. We present a compendium of technical experiments relating to both vision science and visual quality testing that together explore the research and business perspectives of visually lossless image quality, as well as review recent scientific advances. Together, the studies presented in this paper suggest that a single definition of visually lossless quality might not be appropriate; rather, a better goal would be to establish varying levels of visually lossless quality that can be quantified in terms of the testing paradigm.

We propose a residual network with dense block (RNDB) to obtain a higher performance over the original residual network (ResNet). In the designed RNDB, a small dense block in the basic architecture of the residual block is introduced to enhance the feature mapping. As such, the application of the dense block helps the underlying ResNet adequately use the feature maps with dense connections. Thus, the optimization ability of the RNDBs is significantly promoted. In addition, the dense block is adopted in the variants of ResNet such as the residual networks of residual networks (RoR) and wide residual network (WRN) to further improve their performances. Finally, the experiments are utilized to demonstrate the validity of the employment of the dense block to the ResNets, RoRs, and WRNs, where the state-of-the-art result is effectively achieved on CIFAR-10 compared to the original ResNets, RoRs, and WRNs, respectively.

This paper performs cryptanalysis of a recently proposed image encryption scheme. The original encryption scheme uses plain image parameter to modify secret keys as a security measure and is based on improved one-dimensional (1-D) chaotic map with larger chaotic range than existing 1-D chaotic maps. 1-D chaotic maps have simple structure and have lower computation cost than higher dimensional chaotic maps. However, it has been found that the encryption scheme is vulnerable toward chosen plaintext attack with computational complexity of attack less than complexity of brute force attack. In addition, this paper presents an enhanced encryption scheme to remedy the flaws of original scheme. The security of the proposed encryption scheme has been analyzed through experimental results.

Fingerprint recognition systems are widely used for authentication purposes in security systems. However, fingerprint recognition systems can easily be spoofed by imitations of fingerprints using various spoof materials. A compact and discriminative set of features is needed to discriminate between live and spoof fingerprints. We explore combined Shepard magnitude and orientation for live fingerprint detection using independent quantization of global and local features extracted in spatial and frequency domain. The spatial domain features that are extracted comprise of the magnitude of perceived spatial stimuli that is computed from the net variation of perceived edge information. Rotation invariance is achieved by extracting local features based on phase information of significant frequency components in the frequency domain. The concatenated feature vector associated with a fingerprint image is represented as a two-dimensional histogram. The support vector machine classifier is used to classify the fingerprint as either live or spoof. Experiments are performed on three databases, i.e., the fingerprint liveness detection (LivDet) competition databases of 2011, 2013, and 2015. Results showed a reduction in average error rate to 5.8, 2.2, and 5.3 on LivDet 2011, 2013, and 2015, respectively.

To effectively explore image content for image fusion and generate a full end-to-end image fusion pipeline, a parameter adaption algorithm based on gradient transfer and total variation (TV) minimization is proposed, and the fusion problem is transformed into a human visual system guided TV minimization problem, where the data fidelity term maintains the structure edge features of the image, and the regularization term preserves the gradient variation texture detail information. Then, the fusion problem is cast into the optimization of minimum energy functional model. To facilitate choice of regularization parameter, a modified fixed-point iterative method is employed to solve the optimization. Extensive experiments show that this model can obtain both rich detail and spectral information; adjusting the parameters of the proposed adaptive selection strategy can effectively improve the fusion effect while making the image fusion process work as an end-to-end pipeline. Compared to eight state-of-the-art fusion methods, the experimental results demonstrate the effectiveness of the proposed method.

Recently, sparse representation has been widely introduced into tracking methods to improve their performance. However, these methods only focus on reconstructing the candidate samples while ignoring the discriminative information of the background, which greatly limits their performance, especially when the target undergoes heavy occlusion. To tackle this issue, we propose a discriminative collaborative representation-based tracker. We first propose an appearance model based on collaborative representation, in which the appearance of a candidate is represented as a linear combination of the dictionary with a discriminative constraint in the training stage. This constraint can enlarge the margins of reconstruction coefficients that correspond to the positive and negative templates of dictionary. To further enhance the discriminability of the tracker, we introduce this constraint to a state estimation model in the decision stage, which utilizes the reconstruction coefficients to search the optimal candidate as the tracking result. In addition, we use a dictionary update strategy based on collaborative representation, which can promote the adaptability of the tracker. This strategy not only facilitates the dictionary to preserve the historical appearance of the tracking object but also prevents the seriously occluded target from being introduced into dictionary. The experimental results on several challenging sequences demonstrate the robustness of our tracker.

In recent years, large-scale person re-identification has attracted a lot of attention from video surveillance. Usual approaches addressing this task either learn the effective feature embeddings or design the learning architectures to obtain discriminative metrics. Most of them only focus on improving the accuracy of recognition but neglect retrieval efficiency. To improve the accuracy and efficiency of person re-identification simultaneously, an accurate and fast method is proposed based on the bag of visual words (BoVW) model, which has widely been applied in image retrieval. A bag of local features is developed to simplify feature representation for person re-identification. Cross-view dictionary learning is used to eliminate the redundancy of these local features. These local features consist of scale invariant feature transform and local maximal occurrence representation (LOMO) that are invariant in scale and color, respectively. Finally, integrated BoVW histograms are obtained, which encode the images by k-means clustering. Experiments conducted on the CUHK03, Market1501, and MARS datasets show that the proposed method performs favorably against existing approaches.

Distributed compressive video sensing (DCVS) is a paradigm for low-complexity video compression. To improve the rate-distortion (RD) performance of DCVS, the existing works generally separately improve either sampling efficiency or reconstruction performance. We propose an RD efficient DCVS codec that takes advantage of the side information (SI) at the decoder side to jointly improve both sampling efficiency and reconstruction performance. Specifically, on the one hand, by making use of the statistics of the SI, a general content-dependent sampling (CDS) scheme is developed to enhance the sampling efficiency. The proposed CDS scheme can make the important frequency components of each video frame more efficiently sampled to facilitate their accurate reconstruction. On the other hand, by exploiting the residual between each nonkey frame and its SI, a joint residual sparsity and Tikhonov regularization approach is proposed to regularize the multihypothesis (MH) prediction optimization problem to obtain a more accurate MH prediction of the nonkey frame. Meanwhile, an efficient algorithm based on the split Bregman iteration technique is designed to solve the joint regularized optimization problem. The experimental results show that our DCVS codec significantly outperforms the existing DCVS codecs in terms of the RD performance while being computationally efficient.

Traditional trackers are easily affected by uncertain changes in tracking targets, such as occlusion, deformation, and background clutter. To solve these problems, we propose a tracking method, namely location-matching tracking under a convolutional neural network (CNN), which consists of a process of localization, recognition, and model updating. In the location subprocess, the target’s locations of the previous (first) frame and the current frame are utilized to estimate a series of specific regions by the average displacement, and these locations are proven to be useful to improve the probability of a successful tracking. In the recognition subprocess, a CNN is adopted to classify the estimated regions, and we calculate the confidence score maps of these regions to estimate the final target region. To improve the accuracy of the tracking, we propose an optimal similarity matching to verify the final target region and make a confidence decision to update the network. Compared with the state-of-the-art trackers on challenging object tracking benchmark benchmarks, the proposed method can achieve the same or even higher tracking accuracy.

Dynamic texture (DT) and scene descriptions are challenging problems in video understanding that play crucial role in different important applications of computer vision. An efficient framework, called completed and statistical adaptive patterns on three orthogonal planes (CSAP-TOP), for representation of dynamic textures and scenes is addressed in this work. It utilizes adaptive thresholding applied in a completed schema on high-order moment images of three orthogonal planes. Some beneficial properties from its complementary features are inherited: taking into account information from local variations of magnitudes, robustness against noise and near-uniform regions, exploiting more useful and stable textural information at local and regional scales. In addition, we also consider the impact of high-order filtered images and adaptive thresholding in volume statistical adaptive patterns (VSAP) to investigate their influence and efficiency on describing DT and scene sequences. Evaluations of recognition on different datasets of dynamic textures and scenes (UCLA, DynTex, YUPENN, YUP++) show that our proposal, without utilizing sophisticated learning techniques, significantly outperforms recent results in the state-of-the-art.

An adaptive weighted kriging interpolation filter for removal of high-density salt-and-pepper noise (SPN) in images is proposed. The proposed scheme introduces a method for computing the estimation value of a noisy pixel in the center of the processing window. Different from the existing adaptive decision based on kriging interpolation algorithm, the proposed kriging interpolation for different adaptive windows cares about the action of not only the Euclidean distance between nonnoisy pixels but also the size of the current processing window. Therefore, the corrupted pixel is replaced by the inverse filtering-radius weighted sum of kriging interpolations for different adaptive windows. The final processing window is required to include at least three noncorrupted pixels. The proposed algorithm is extensively evaluated on a variety of benchmark images and the experimental results show that it outperforms several standard and popular algorithms in terms of visual quality and quantitative results.

Visual tracking based on deep learning technique is a very attractive research topic recently in the computer vision field. Deep convolutional neural networks (CNNs) are inherently limited to low spatial resolution, due to the max pooling process in the modules, and they are constrained by the high computation burden. We present a pretrained deep learning network architecture to the task of visual tracking, by introducing a wavelet representation in the network and a two-stage fine-tuning for learning appearance features, which improves the original deep learning tracker. Moreover, a loss layer based on Bayesian theorem is adopted to compute maximum classifier score, instead of the softmax loss layer, which can enhance the success rate. In addition, the idea of wavelet pooling helps perform feature dimension reduction. In addition, wavelet representation helps to reduce the computation time greatly. Compared with the original algorithm and other state-of-the-art methods, the proposed tracking method shows excellent performances on test baseline dataset. As our optimized spectrum CNN can extract a compact and efficient representation of objects, it can be further applied to multiple objects tracking.

This study concentrates on consistent object contour extraction method for stereo image segmentation after the object regions in the left image have been obtained. By taking advantage of the epipolar geometry, our approach introduces an energy optimization framework that incorporates both the stereo correspondence term and patch-based object contour probability term. The contour map is generated by integrating the terms of stereo correspondence and patch-based object contour probability; then, the optimal contours are obtained using geodesic distance technology. The core of the proposed method is to build upon an energy optimization framework with two key contributions: first, it incorporates the patch-based object contour probability term that introduces two search strategies to efficiently find the joint nearest neighbor patch pairs for the stereo image pair. The patch-based object contour probability term provides consistent and reliable priors for the contour extraction. Second, previous methods encounter missing pixels in the extracted contour in the occluded regions. Our approach overcomes this limit by introducing the geodesic distance technology to search the optimal contours. Experimental evaluation on Middlebury dataset and Adobe open dataset indicates that the results of our stereo image segmentation are comparable with or of higher quality than state-of-the-art methods.

Smile detection plays an important role in human emotion analysis and has wide applications. However, there is still a gap between the performance of the current smile detection algorithms and real-world applications, due to variations of head pose and environment noise. We propose a robust framework based on convolutional neural networks (CNNs) for smile detection. To alleviate the influence of head pose variations and improve performance, the proposed framework customizes two-feature learning layers such as (1) smile feature extraction layer is constructed by hidden factor analysis for learning head pose-insensitive smile features; (2) smile feature discrimination layer is constructed by marginal Fisher analysis, and it is used to learn discriminative features for further enhancing the discrimination between smile and nonsmile. The two layers both work as fully connected layers, and they are connected layer by layer to a backbone CNN network. Experiments have been performed on two publicly available datasets, and the results show that the proposed framework delivers promising performance (95.45% on GENKI4K and 93.62% on labeled faces in the wild attribute) and outperforms the state of the art.

Action recognition in realistic scenes is a challenging task in the field of computer vision. Although trajectory-based methods have demonstrated promising performance, background trajectories cannot be filtered out effectively, which leads to a reduction in the ratio of valid trajectories. To address this issue, we propose a saliency-based sampling strategy named foreground trajectories on multiscale hybrid masks (HM-FTs). First, the motion boundary images of each frame are calculated to derive the initial masks. According to the characteristics of action videos, image priors and the synchronous updating mechanism based on cellular automata are exploited to generate an optimized weak saliency map, which will be integrated with a strong saliency map obtained via the multiple kernels boosting algorithm. Then, multiscale hybrid masks are achieved through the collaborative optimization strategy and masks intersection. The compensation schemes are designed to extract a set of foreground trajectories that are closely related to human actions. Finally, a hybrid fusion framework for combining trajectory features and pose features is constructed to enhance the recognition performance. The experimental results on two benchmark datasets demonstrate that the proposed method is effective and improves upon most of the state-of-the-art algorithms.

We propose a face recognition method using distinctive triangle encoding and nonlocal constraint-based sparse representation (TENCSR). With TENCSR, first the pixel values of all images are used as the low-level features. Next, a clustering method is proposed by considering the density distribution of target data, named shared weights support vector data description (SW-SVDD), which makes the obtained decision boundary closer to the optimal. On the basis of SW-SVDD, a more distinctive triangle encoding (MDTE) method is introduced by considering the cluster center information and the size information of cluster, which makes the encoded features more distinctive. Then the high-level features are obtained by encoding those low-level features using MDTE. Meanwhile, a nonlocal constraint-based sparse representation classifier (NC-SRC) is proposed by the biological discovery that dissimilar inputs have dissimilar codes. Finally, those high-level features are classified by the proposed NC-SRC. Experimental results on Georgia Tech, CVL, IMM, FRGC, and AR databases show that our TENCSR outperforms some state-of-the-art algorithms.

This erratum corrects information in a table.