2.1.

Multichannel Conversion to a Grayscale Image

In the RGB space, the red, green, and blue components of a vector are commonly related to the pixels of an RGB image of size $M\times N$. They can be represented by $C(x,y)$, as in the following equation:

Typically, a compound gradient image $g^c(x,y)$ is determined by^18,19

In general, the modulus of the gradient of the color planes $g^c$ is computed using the Euclidean distance,²⁰ as follows:

Conventionally, the partial derivative along the $x$-axis of a two-dimensional function $C(x,y,i)$ can be numerically approximated as $\frac{\partial C(x,y,i)}{\partial x}\approx C(x+1,y,i)-C(x,y,i)$. Likewise, the partial derivative along the $Y$-axis is given by $\frac{\partial C(x,y,i)}{\partial y}\approx C(x,y+1,i)-C(x,y,i)$.

In color image processing, the gradient is commonly used as a procedure of color edge detection. Therefore, the modulus of the gradient of the color planes is a sharp image, which can be computed using the equation

In addition to the RGB space, color images are also processed in the hue, saturation, and intensity (HSI) color space, because it is a suitable model for color description and analysis. The HSI space is modeled as a double cone where hue represents the dominant color, saturation represents the purity of the color, and intensity represents the brightness, respectively. As stated by Gonzalez and Woods,²⁶ this model decouples the intensity component from the color-carrying information (hue-saturation) in a color image. The intensity channel is an essential descriptor of monochromatic images, and it is classically used for multichannel conversion to a grayscale image. The difference measurement when working in the HSI color space is modified as established by Koschan and Abidi.¹⁸

In this work, the multichannel conversion to a grayscale image has been done by means of the MGC operator as is shown in Fig. 1. As the MGC operator is a color edge detection technique for digital images, the MGC(RGB) and MGC(HSI) matrices show the high spatial frequency content of the input color images. Thereby, this makes it suitable for finding focused regions.

Fig. 1

Multichannel conversion to grayscale images by means of the MGC operator from (a) defocused and (b) focused color images. The MGC(RGB) and MGC(HSI) images show sharp regions for the case of focused images, unlike the dark uniform intensity distributions for the case of defocused images. As can be seen, the MGC operator is sensitive to variations in intensity but not very sensitive to variations in hue and saturation.

2.2.

Autofocus Methods

In the literature, there exist some comparisons about the performance of AF algorithms.^4,6,8,9,12 Each algorithm is capable of producing a figure of merit (FM) that is analyzed by taking into account the global or local variance in the image intensity values $f(x,y)$. Customarily, the AF algorithms can be classified into five groups according to their mathematical nature: derivative-based algorithms,^27,28 statistical algorithms,¹⁰ histogram-based algorithms,^6,12 intuitive algorithms,⁹ and image transformations-based algorithms.³ Throughout this paper 15 AF algorithms, which have been widely reported in the literature, are tested and compared using the MGC images. This task was carried out to improve the performance of AF algorithms. Table 1 summarizes the definitions of the most typical AF metrics defined in the new approach, namely the MGC transformation. The output of an ideal AF algorithm is commonly defined as having a maximum value in relation to the best focused image position. Moreover, this value clearly decreases as defocus increases. As noted by Tian,³ the fundamental requirements for an FM are unimodality and monotonicity, which ensure that the FM has only one extreme value and is monotonic on each side of its peak or valley. Furthermore, Redondo et al.⁴ defined the number $\eta$ of local maxima, the width of the focus curve $\alpha /\beta$ given by the width of the focus curve at 80% and 40%, and noise/illumination invariance as important features of the autofocus curve. Another complementary characteristic of the AF algorithms is their accuracy and fast response. To evaluate the AF performance (AP) of each AF algorithm, the following score is proposed:

Table 1

AF algorithms rewritten in terms of the modulus of the color gradient operator gc(x,y).

2.3.

Multifocus Image Fusion

As mentioned previously, any microscopic imaging system can only focus the field of view (FOV) of the sample that is inside the DOF of the objective lens. This means that only certain axial planes of the sample are in-focus. A current solution to this drawback is a multifocus image fusion to reconstruct an all-in-focus image of the complete FOV for a particular specimen. This can be done by capturing images of the sample on different focal axial planes. In this section, a color image fusion scheme based on the MGC method is proposed as shown in Fig. 2.

Fig. 2

DOF extension on GPU. (a) Source color images $C_z(x,y,i)$, (b) modulus of the gradient of color planes $g_z^c(x,y)$, (c) fusion rule, and (d) 3-D visualization of an in-focus image.

Let $C_z(x,y,i)$ be a set of input images, where $z=\mathrm{1,2},\dots ,Z$. The index $i=\mathrm{1,2},3$ is related with the channel/band used. For each axial plane, Eq. (2) is computed to create a compound gradient image and then for each pixel $(x,y)$ the maximum value is selected using $\mathrm{sap}(x,y)={\max }_z\{g_1^c(x,y),\cdots g_Z^c(x,y)\}$. In other words, the $\mathrm{sap}(x,y)$ matrix denotes the slice axial position of in-focus pixels along the $z$-axis. A postprocessing stage involves a spatial consistency algorithm.¹⁷ This postprocessing is carried out by means of a low pass filtered $\widetilde{\mathrm{sap}}(x,y)$ matrix using a $p\times q$ median filter. This algorithm ensures that the majority of the intensity pixels in a $p\times q$ neighborhood of $\widetilde{\mathrm{sap}}(x,y)$, come from the same $z$-slice or from the closest one. For example, the spatial consistency of the $\widetilde{\mathrm{sap}}(x,y)$ matrix is shown in Fig. 3. Figure 3(a) contains three $p\times q$ neighborhoods, where the value of the slice axial position is higher than its neighbors. Figure 3(b) shows these values adjusted to match the values of the $p\times q$ neighborhood of $\widetilde{\mathrm{sap}}(x,y)$ to conserve the continuity of the surface of the sample.

Fig. 3

(a) Slice axial position $\mathrm{sap}(x,y)$ of in-focus pixels $(x,y)$ along the $z$-axis, (b) postprocessing $\widetilde{\mathrm{sap}}(x,y)$ matrix by means of a low-pass filter to reach spatial consistency.

To avoid the introduction of artificial information, the fused image $\mathrm{\varphi }(x,y,i)$ is composed from the (multichannel) pixels that are present in the original input data $C_z(x,y,i)$, only if the slice position fulfills the condition $z\in \widetilde{\mathrm{sap}}(x,y)$ for each pixel $(x,y)$. Therefore, a multifocus image fusion algorithm can be defined as follows:

The resulting fused images are evaluated with a nonreference image quality metric based on measuring the anisotropy of the images. The anisotropic quality index (AQI) of an image $\mathrm{\varphi }(x,y,i)$ is given by²⁹

4.1.

Focusing Results

To obtain a performance evaluation of the 15 AF techniques on the MGC images, six $z$-stacks of 21 multichannel images are recorded using two histological samples. Each stack has a particular color that is highly dominant as shown in Fig. 4. This allows us to evaluate the MGC method for different color distributions and amplifications inside of the digital image.

Fig. 5

Focus measure for a $z$-stack of images amplified at $2.5\times$, which have been processed on the MGC(RGB) of color images.

Fig. 6

Focus measure for a $z$-stack of images amplified at $40\times$, which have been processed on the MGC(HSI) of color images.

Fig. 7

Elastic cartilage sample, magnified using four microscope objectives of (a) $2.5\times$, (b) $10\times$, (c) $40\times$, and (d) $100\times$ (oil immersion).

Fig. 8

Focus measure from a $z$-stack of images, which are acquired using different objectives. All the FM present monotic behavior even up to an amplification of $100\times$, and the new color distribution inside of the images acquired from the amplified sample under inspection.

Some research⁴ has reported that beyond a magnification of $63\times$, the performance of the various AF metrics is drastically impaired. According to results shown in the graphs of Fig. 8, when using the MGC approach all the FM curves present monotonic behavior, even when the magnification is increased to $100\times$ (oil immersion). This last experimental result supports the advantage of a color-to-MGC space transformation. Nevertheless, a problem arises when images of a sample of thickness $t\ge \mathrm{DOF}$ are acquired at a magnification of $100\times$. There exist portions of the image partly in focus. In the graphs of Fig. 8(d), two regions in focus located at $z=0$ and $z=6\mu \mathrm{m}$ can be seen. According to the results given in Tables 4 and 5, all the AF measures realized in the MGC space are accurate in spite of the different magnifications, unlike some typically used channels for focus measure.

Table 4

Autofocusing Performance AP of all metrics in different grayscale channels and MGC images. The mean and standard deviation of AP is given in bold.

Table 5

Autofocusing Performance AP of all metrics in different grayscale channels and MGC images. The mean and standard deviation of AP is given in bold.

Another advantage of the MGC method is its computational simplicity and inherent parallelism. Figure 9 shows the computational cost of the MGC(RGB) method in a $z$-stack of digital images of $2584\times 1936\text{pixels}$, when they run on Intel^© X^©(R) 2.10 GHz, 16 GB RAM, NVIDIA Quadro K4000. The parallelized MGC method on the GPU is one order of magnitude faster than the same application implemented in CPU.

Fig. 9

Execution time of the MGC(RGB) method in a $z$-stack of digital images.

4.2.

Multifocus Image Fusion Results

It is well known that the digital images of thick microscopic objects provided by an optical widefield microscope device are strongly blurred for the portion of the object that lies outside of the DOF of the objective lens. We can seek those regions of the FOV, which are conveniently located in-focus. The present subsection will describe the results of a method to merge multifocus frames based on the MGC approach.

Our experiment starts with the acquisition of a digital image $z$-stack from a histological sample. This set of $z$-images are obtained by moving the microscope stage along the optical axis. For this, the axial extension $t$ of the sample is defined and then the axial stage with the sample is moving to cover this extension. The interplanar distance $\mathrm{\Delta }Z$ between different optical sections is less than the axial resolution of the microscope, defined as the DOF in Table 3. From this table, it is evident that $\mathrm{\Delta }Z$ is determined by the NA of the objective lens.

Digital images of a beetle shell are acquired with amplification of $10\times$ and interplanar distance $\mathrm{\Delta }Z=3\mu \mathrm{m}$. The given $z$-stack is composed of 42 images with $1024\times 768\text{pixels}$. Figure 10(a) shows the in-focus image obtained with the software package EDoF plug-in³¹ based on a complex wavelets algorithm for EDoF.¹⁷ The fusion process takes an average execution time of 52.2 s, whereas the fused image of Fig. 10(b) is based on the proposed MGC fusion method. In this technique, the resulted slice axial position matrix $\mathrm{sap}(x,y)$ is low-pass filtered using a $p\times q$ median filter with $p=q=3,15,35$. The total execution time is 32.1 s. The 3-D visualization of the resulting in-focus image is sketched in Fig. 10(c). Finally, the nonreference image quality metric of Eq. (7) is computed for the in-focus image quality assessment. The results are shown in Fig. 10(d).

Fig. 10

Image fusion results using (a) the software package EDoF plug-in and (b) the MGC fusion method. (c) 3-D visualization of (b). (d) Fusion evaluation of in-focus images. The readjustment of in-focus pixels of $\widetilde{\mathrm{sap}}(x,y)$ along the $z$-axis avoid false edge detection.

Another example is the case of an umbilical cord that is imaged at the amplification of $10\times$ and interplanar distance $\mathrm{\Delta }Z=3\mu \mathrm{m}$. The given $z$-stack is composed of 39 images with $2584\times 1936\text{pixels}$. Again, Fig. 11(a) shows the in-focus image obtained with the EDoF plug-in.^17,31 It takes an average execution time of 477.43 s. The fused image of Fig. 11(b) is based on the proposed MGC fusion method, where the resulted slice axial position matrix $\mathrm{sap}(x,y)$ is again low-pass filtered using a $p\times q$ median filter with $p=q=\mathrm{3,15},35$. The total execution time is 238.9 s, and the fusion evaluation is shown in Fig. 11(d). As we can see, the proposed fusion method reveals a high-quality image independent of faulty illumination during the image acquisition.

Fig. 11

Image fusion result using (a) the software package EDoF plug-in and (b) the MGC fusion method. (c) 3-D visualization of (b). (d) Image fusion evaluation. As it can be seen, the proposed fusion method reveals a high quality image independently of faulty illumination during image acquisition.