Averaged subtracted polarization imaging for endoscopic diagnostics of surface microstructures on translucent mucosae

Katsuhiro Kanamori

doi:10.1117/1.JBO.21.7.071105

24 December 2015 Averaged subtracted polarization imaging for endoscopic diagnostics of surface microstructures on translucent mucosae

Katsuhiro Kanamori

Author Affiliations +

Journal of Biomedical Optics, Vol. 21, Issue 7, 071105 (December 2015). https://doi.org/10.1117/1.JBO.21.7.071105

Abstract

An endoscopic image processing technique for enhancing the appearance of microstructures on translucent mucosae is described. This technique employs two pairs of co- and cross-polarization images under two different linearly polarized lights, from which the averaged subtracted polarization image (AVSPI) is calculated. Experiments were then conducted using an acrylic phantom and excised porcine stomach tissue using a manual experimental setup with ring-type lighting, two rotating polarizers, and a color camera; better results were achieved with the proposed method than with conventional color intensity image processing. An objective evaluation method that uses texture analysis was developed and used to evaluate the enhanced microstructure images. This paper introduces two types of online, rigid-type, polarimetric endoscopic implementations using a polarized ring-shaped LED and a polarimetric camera. The first type uses a beam-splitter-type color polarimetric camera, and the second uses a single-chip monochrome polarimetric camera. Microstructures on the mucosa surface were enhanced robustly with these online endoscopes regardless of the difference in the extinction ratio of each device. These results show that polarimetric endoscopy using AVSPI is both effective and practical for hardware implementation.

1. Introduction

The key to endoscopic inspections during the early stages of gastric or colon cancer is a close inspection of the microstructure on the surface of the mucosa. Here, “microstructure” means two-dimensional (2-D) gastric areas surrounded by gastric pits in the gastric mucosa whose regular or irregular texture patterns are influenced by the presence of early-stage cancer. It is difficult to observe such microstructures under ordinary white light; therefore, invasive methods have been used. Spraying blue indigo-carmine fluid allows doctors to observe the puddle of blue fluid in surface microgrooves rather than relying on white light to inspect the mucosa’s microstructure. This approach, however, has drawbacks: the mucosa may get damaged and bleed due to the fluid spraying, and this leads to its rapid discoloration hampering further observation. Thus, there is a great interest in developing noninvasive digital image processing. For example, a “pseudo-indigo-carmine-sprayed image” (PISI)¹ is created from color intensity images (CIIs) obtained by a conventional endoscope; nonlinear edge detection is executed on such images with the assumption that concave grooves will be darker (due to the generation of shadows) than the neighboring planar region, thus enhancing the blue component of the color in the detected area. However, the detection and enhancement accuracy of such images do not yet match that of traditional indigo-carmine-sprayed images. Therefore, the use of polarimetric imaging (rather than conventional intensity imaging) to extract surface geometric information of the mucosa has been studied.²^–⁴

Many conventional studies have been conducted in the field of polarimetric biomedical imaging. The simple subtracted polarimetric technique uses two images (called co- and cross-polarized images) and calculates the degree of polarization (DOP) between the two images by means of subtraction. This technique is frequently used for dermatology: either a typical, affordable, color, dermatological camera may be covered with a fixed analyzer (polarizer) and used with polarized lighting in two different polarimetric statuses⁵ or a professional dermatological camera may be used with fixed polarized lighting and two CCDs with different polarimetric capturing.⁶^,⁷ The main objective for dermatological purposes is to separate light reflected from surface and deep layers and collect clear images of tattoos, moles, and melanoma. This type of simple polarimetric imaging can also be used to get a microcirculation image⁸ or optical anisotropy of collagen tissue.⁹

On the other hand, Mueller polarimetric imaging¹⁰^–¹² uses a more complicated polarization state generator (PSG) for lighting and a polarization state analyzer (PSA) as the camera; the polarization status of both PSG and PSA can be changed in a $\sim 4 \times 4$ combination. The main focus of Mueller imaging is the observation of cancerous zones in deeper layers of tissue using polarimetric characteristics such as birefringence.

There are several recent studies of polarimetric endoscope prototypes. Since it is quite challenging to implement PSA and PSG on practical endoscopic systems,¹³^–¹⁶ the typical polarimetric endoscope has two cross-polarization lightings,¹⁷ fixed polarized lightings,¹⁸ or stereo-type cameras.¹⁹ The objective of these systems is either the contrast enhancement of blood vessels in deep tissue layers or specular highlight removal. Only one prototype endoscope implementation²⁰ uses two pairs of polarized lighting and capturing in order to collect the optical anisotropy of the tissue.

Unlike conventional polarization imaging and polarimetric endoscopy studies, the objective of this paper is the image enhancement of mucosal surface microstructures using the depolarization properties of the surface groove regions. This new technique employs two pairs of co- and cross-polarization images captured under different linearly polarized illumination, and image processing specially designed for enhancement of microstructures.

Section 2 of this paper details the theoretical background for contrast enhancement of the microstructure. Section 3 describes the basic hardware and software used for the experiments. Section 4 describes an experiment that uses an acrylic phantom and a porcine gastric mucosa and discusses the objective evaluations that were conducted on polarimetric images compared with conventional indigo-carmine-sprayed images. Section 5 presents two types of online prototype polarimetric endoscopes. Section 6 concludes the paper and describes future work.

2. Theoretical Background

2.1.

Groove Contrast Enhancement Using Polarized Light

The anatomy of a gastric mucosa is shown in Fig. 1(a). It has numerous grooves called “gastric pits” that are about 0.2-mm wide and deep. These grooves create 2- to 3-mm wide microstructures, called “gastric areas,” on a 2-D planar surface.

Fig. 1

Cross section of the gastric mucosa: (a) anatomy model; (b) white nonpolarized lighting and capturing; and (c) polarized lighting and capturing.

Figure 1(b) shows a schematic diagram of a cross section of the gastric mucosa, which has plane and groove regions running perpendicular to the cross-section plane. When nonpolarized white light is used, the microstructures on the mucosal surface are difficult to discern for two reasons. First, the mucosa is a translucent medium; incident light scatters in that medium and diffuses uniformly so that concave regions show similar intensity to planar regions, with no shadow generation. Some of the light that enters the mucosa reflects from the shallow layers within the mucosa and other light penetrates and reflects from the deeper layers; these two light components are added together by the observer (the camera), greatly reducing groove contrast.

Second, the endoscope has a coaxial illumination in which the objective and lighting lenses are virtually parallel so that the light is unable to illuminate the object from a slant, which means that shadows caused by surface unevenness cannot be clearly seen.

Figure 1(c) shows the linearly polarized light from polarizer 1 vertically entering the organ’s surface mucosa from above. The reflected light that returns to the camera can be divided into the following five types.

1. Strong specular reflection light or glare at the surface that maintains the incident polarization state. This light cannot be distinguished from other polarized light and inevitably remains in the subtracted polarization image during the surface microstructure enhancement process, and capturing this light should be avoided as much as possible (SP).
2. Backscattered light from the groove region that maintains the incident polarization state (SG).
3. Backscattered light from the groove region that is depolarized by multiscattering from the groove region (DG).
4. Backscattered light from the superficial layer of the mucosa that retains its incident polarization state (SR).
5. Backscattered light that has penetrated to deeper layers in the mucosa and returns in a depolarized state due to multiscattering ( $D$ ).

Here, we assume that the light in the groove region is multiscattered and partially depolarized with no optical anisotropy. This simple model is derived from the experimental results, which indicate that the subtracted polarization image quality is almost the same regardless of the direction of the microstructure’s grooves because the grooves which comprise biological tissue are optically incomplete and cause random polarization.

As Fig. 1(c) shows, the light that is reflected from the microstructure is observed through the analyzer (polarizer 2). There are two statuses: copolarized ( $⃫$ ) and cross-polarized ( $⊥$ ), in which the polarization axes of polarizers 1 and 2 are parallel and orthogonal, respectively.

Table 1 shows how the microstructure images can be enhanced using this polarimetric processing under the assumption that light absorption in the mucosa can be neglected.

Table 1

Contrast enhancement between groove and plane regions.

	Groove region			Plane region			Contrast (IP/IG)	Contrast IP/IG (d=1/2, dg=3/4)
	SG	DG	IG=SG+DG	SR	D	IP=SR+D	Contrast (IP/IG)	Contrast IP/IG (d=1/2, dg=3/4)
$I (⃫)$	$1 - d g$	$d g / 2$	$1 - d g / 2$	$1 - d$	$d / 2$	$1 - d / 2$	$(2 - d) / (2 - d g)$	$6 / 5$
$I (⊥)$	0	$d g / 2$	$d g / 2$	0	$d / 2$	$d / 2$	$d / d g$	$2 / 3$
$I (⃫) - I (⊥)$	$1 - d g$	0	$1 - d g$	$1 - d$	0	$1 - d$	$(1 - d) / (1 - d g)$	2
$I (⃫) + I (⊥)$	$1 - d g$	$d g$	1	$1 - d$	$d$	1	1	1

As the total intensity of reflected light in the groove region IG and plane region IP is partially depolarized, we assume the depolarized ratio $d g$ and $d$ ( $0 < d < d g < 1$ ). This means that the partially depolarized ratio is higher in IG than IP.

The contrast between the groove and the plane region (IP/IG) becomes significant over 1 for the subtracted polarization image $I (⃫) - I (⊥)$ , meaning that the contrast between planes and grooves, which are the main elements of the microstructure, can be amplified significantly using this polarimetric processing. On the other hand, the contrast between the groove and the plane region (IP/IG) is 1 for the intensity image $I (⃫) + I (⊥)$ , which means that the groove and the plane regions have the same intensity. This explains the invisibility of the surface microstructure under nonpolarized white light. Here, since the true values of $d$ and $d g$ cannot be measured, we can assume $1 / 2$ and $3 / 4$ , respectively, in the rightmost row of Table 1, and we get the contrast as 2 or the subtracted polarization image can double the image contrast of the groove region.

2.2.

Averaged Subtracted Polarization Images

Since we have two different polarization statuses, both the lighting (polarizer status) and the camera (analyzer status), the polarization axes of incident light L0 and L90 denote the polarization axes of 0 deg and 90 deg and the two different polarization statuses of the cameras, denoted as C0 and C90, indicate polarized axes of 0 deg and 90 deg. The combination of light and camera polarization status is expressed, e.g., as L0C0 or L0C90, which is expressed as HH or HV in the conventional study.¹⁴ There are four types of subtracted polarization processing as follows:

Eq. (1)

C 0 - D i f = I [L 0 C 0] (⃫) - I [L 90 C 0] (⊥),

Eq. (2)

C 90 - D i f = I [L 90 C 90] (⃫) - I [L 0 C 90] (⊥),

Eq. (3)

L 0 - D i f = I [L 0 C 0] (⃫) - I [L 0 C 90] (⊥),

Eq. (4)

L 90 - D i f = I [L 90 C 90] (⃫) - I [L 90 C 0] (⊥) .

Averaged subtracted polarization images (AVSPIs) are defined by the subtraction of the averaged cross-polarized images from copolarized images, as expressed in the following equation:³^,⁴

Eq. (5)

AVSPI = {I [L 0 C 0] + I [L 90 C 90]} / 2 - {I [L 0 C 90] + I [L 90 C 0]} / 2 = (L 0 - D i f + L 90 - D i f) / 2 = (C 0 - D i f + C 90 - D i f) / 2 .

Theoretically, AVSPI can be derived as a Mueller matrix element (M22),¹⁴ but Eq. (5) gives a new important definition of AVSPIs. This new concept has three points.

First, AVSPI can compensate for the image quality difference due to slant-angle lighting. When the incident light enters the surface at a slant angle, L0 and L90 lighting serve as an $S$ -wave and a $P$ -wave, respectively, and AVSPI can average L0-Dif and L90-Dif.

Second, AVSPI can cancel out the difference in the lightings, optics, and sensors during co- and cross-polarized observations and creates high-quality and robust subtracted polarized images.

Third, although AVSPI is a time-sequential method, direct image subtraction between frames is not used, but two pairs of mean values of the sequentially captured images are subtracted each other. As this procedure is time-domain low-pass filter (LPF), AVSPI is not too sensitive for the movement of the endoscopes.

3. Hardware and Image Processing

3.1.

Lighting and Camera with Manually Rotated Polarizer Setup

Figure 2(a) shows the manual polarimetric imaging experimental setup with ring-type illumination with a white LED light fiber source and flexible fiber light guide, a rotatable polarizer (polarizer 1) with a hole in the center to apply linearly polarized illumination, and a rotatable analyzer (polarizer 2) to select the reflected light from the object in the specified direction of polarization. Both polarizers (1 and 2) have polymer thin films with glass plates. They can be rotated both manually and independently and set to arbitrary polarization direction angles, including the L0C0, L0C90, L90C90, and L90C0 positions. Here, rotation angle $ψ_{I}$ is defined by the camera coordinate system, as shown in Fig. 2(b). In this system, the 0 deg and 90 deg angles denote the horizontal ( $X$ ) and vertical ( $Y$ ) directions of the image frame, respectively.

Fig. 2

Manually rotated polarizer setup.²^–⁴ (a) Experimental arrangement schematic and appearance. (b) Definition of polarizer angle in the camera coordinate system.

3.2.

Image Processing to Generate Enhanced Groove Images

Figure 3 shows an overall illustration of image processing for groove detection and color enhancement processing. The image processing described above is executed for both AVSPI for polarimetric imaging and conventional intensity image processing. Since the manually rotated polarizer setup cannot capture nonpolarized CIIs, the nonpolarized CII is generated by image processing according to the following equation:

Eq. (6)

CII = {I [L 0 C 0] + I [L 90 C 90] + I [L 0 C 90] + I [L 90 C 0]} / 4 .

Fig. 3

Image processing schematic: (a) imaging processing flow and (b) $5 \times 5$ space-differential mask.

First, $1024 \times 768 pixel$ color images with red, green, and blue components of 8 bits each are acquired using the color camera; these are then converted into $512 \times 384 pixel$ color images, which are closer to the typical image size of today’s endoscopes. In this process, space LPF ( $5 \times 5$ Gaussian type) is performed to suppress undesirable aliasing noise, which can be amplified at the differential processing stage before $1 / 2 \times 1 / 2 pixel$ sampling occurs.

The space differential filter shown in Fig. 3(b) is designed to extract a mesh structure that continuously covers the mucosal surface. The following procedure is executed in areas of $5 \times 5$ to $7 \times 7 pixels$ and is executed only for the green component of the color image because the green plane image has less noise than other plane images and because the penetration depth of green light into the medium is roughly the median of all R, G, and B component images.

1. In each mask area, the pixel value of the center position $R_{i j}$ is compared with those of the end pixel positions in four directions: horizontal (filled square), vertical (filed star), upper right (filled triangle), and lower right (filled circle).
2. The center pixel is recorded as a candidate for the concave area if the center value is less than two values in any one of the four directions.
3. The maximum intensity drop value in the four directions is estimated as the $Δ$ -value, which is output as a groove gray image (GGI) after being multiplied by 50 to generate an 8-bit gray image.

To generate a groove-enhanced image (GEI), the blue component is enhanced using GGI and the input color image. The procedure is executed by retaining the blue component and subtracting by GGI value from the red and green components¹

Eq. (7)

{\begin{cases} GEI (R) = R - GGI \\ GEI (G) = G - GGI \\ GEI (B) = B \end{cases} .

4. Experimental Results

4.1.

Evaluation Using an Acrylic Phantom

First, polarimetric imaging is tested with an artificial phantom that uses an acrylic lenticular plate with parallel grooves on the surface. Figure 4(a) shows the cross section of the phantom, which has a layered structure like real mucosa; its translucent lenticular plate has a 2-mm thickness and many parallel cylindrical grooves with 0.7-mm pitch and 0.14-mm depth (Nihon Tokushu Kogaku Jushi Co., Ltd.). The phantom also has a transparency sheet printed with vessel patterns and a perfect diffuser plate.

Fig. 4

The basic experiment using phantom: (a) schematic diagram of the cross section; (b) top view of the phantom and reflection status; (c) GEI using CII; and (d) GEI using AVSPI.

Figure 4(b) shows a 2-D view of the phantom top. Two types of lenticular plates having the same size and structure but different transparencies are attached side by side without any gap. The parallel grooves run horizontally across the entire image. The lenticular plate of the upper area (“opal 422”) is more transparent than that of the lower area (“opal 432”) and has the same white color; therefore, the vessel patterns are clearer in the upper area than in the lower area. If the phantom is illuminated from straight above, the specular reflection at the groove top covers the entire view of the field; however, the phantom is slightly tilted to the lighting and capturing axis, thus dividing the entire field into a right-side specular reflection area and a left-side diffuse reflection area. Although grooves are visible in the right-side specular reflection area, we can hardly recognize them in the left-side diffuse reflection area—especially in the area denoted by the rounded rectangle (where the diffuse reflection from translucent “opal 432” appears). Incident light scattering is dominant in this area, which hides the shading of grooves completely and this may be the same phenomenon by which the microstructure on the translucent mucosae is invisible.

Figures 4(c) and 4(d) show the GEI using CII and AVSPI. The L45 (45 deg) and L135 (135 deg) light polarization angles are used for the groove, which are set to 0 deg (horizontal direction) to avoid strong optical anisotropy due to groove direction. In Fig. 4(c), GEI using CII is noisy and deformed because of the influence of the vessel pattern under the lenticular plate and the difference in reflection statuses. On the other hand, Fig. 4(d) shows GEI using AVSPI, where grooves can be detected with almost the same spacing regardless of the plate transparency and specular/diffuse reflection statuses. These results show that polarimetric imaging performs better than intensity imaging in detecting grooves on the surface of translucent mediums.

4.2.

Enhancement of the Porcine Gastric Mucosal Image

Samples of fresh gastric mucosa from slaughtered porcine were excised and stretched on a cork board. Polarimetric capture and imaging were conducted using the same experimental setup described in Sec. 3. After all image-acquisition experiments were complete, indigo-carmine fluid was sprayed onto the surface of mucosa to obtain the microstructure image as the ground truth image.

Figures 5(a)–5(d) show four polarized images captured using the manually rotated polarizer setup. Figures 5(a) (L0C0) and 5(b) (L90C90) are copolarized images with strong specular reflections, and Figs. 5(c) (L0C90) and 5(d) (L90C0) are cross-polarized images that show reddish, deeper-layer tissues without specular reflection. The arrow signs at the top left of each image denote the polarization axis direction of the lighting.

Fig. 5

Captured polarized images and enhanced microstructure images of the porcine gastric mucosa: (a) L0C0, (b) L90C90, (c) L0C90, (d) L90C0, (e) L0-Dif, (f) L90-Dif, (g) AVSPI, (h) DOP, (i) CII, (j) Indigo-carmine-sprayed color intensity image, (k) GEI using CII, and (l) GEI using AVSPI.

Figures 5(e)–5(g) show subtracted polarization images. There are no significant differences in the direction that surface grooves travel in the 2-D plane, which indicates that the incident light is partially depolarized in the groove regions and thus shows no anisotropy. However, the surface microstructures in Fig. 5(e) (L0-Dif) are clearer than those in Fig. 5(f) (L90-Dif). This difference can be explained as follows. The mucosal surface slopes slightly from the lower part to the upper part of Fig. 5, as can be seen from the big puddle of indigo-carmine fluid in Fig. 5(j). Therefore, in this setup, the incident lights at L0 and L90 behave as the $S$ - and $P$ -waves to the surface, respectively, and a greater amount of incident light reflects near the mucosal surface for L0-Dif (the $S$ -wave). As a result, more superficial structures can be seen with L0-Dif in Fig. 5(e) than with L90-Dif in Fig. 5(f).

Figure 5(g) shows AVSPI, in which L0-Dif and L90-Dif images are averaged as shown in Eq. (5), showing that it is possible to obtain more stable subtracted polarization images.

It is interesting that these subtracted polarization images are close to being monochromatic gray images; this reason is that the superficial layer of the gastric mucosa appears to have no blood vessels, showing as almost gray with no color.

Figure 5(h) shows the DOP image, which was obtained by Eq. (8) using the images in Figs. 5(a) and 8(c). The DOP image is more greenish than the AVSPI images and this is quite similar to the results in the previous study¹⁹

Eq. (8)

DOP = \frac{I [L 0 C 0] - I [L 0 C 90]}{I [L 0 C 0] + I [L 0 C 90]} .

Figures 5(i)–5(l) compare the resulting AVSPI images with images obtained by other methods. Figure 5(i) shows a CII created from the images shown in Figs. 5(a)–5(d) using Eq. (6); in this image, it is hard to make out any surface microstructures. Figure 5(j) shows the same CII after it has been sprayed with indigo-carmine fluid; note that the microstructures are now apparent. This image can be regarded as the “ground truth” in this experiment. Figure 5(k) shows the GEI results of CII. Although macroshapes, such as folds in the mucosa, are contrast-enhanced, the surface microstructures shown there are somewhat different from those shown in Fig. 5(j).

GEI using AVSPI [Fig. 5(l)] shows enhanced contrasts of surface microstructures and closely resembles the indigo-carmine-sprayed image selected as the ground truth, which is shown in Fig. 5(j). Thus, we see a clear improvement over the enhanced result we obtained in Fig. 5(k).

4.3.

Texture Feature Vector Using a Gray-Level Co-Occurrence Matrix

This section discusses the objective evaluation used to determine which processing strategy is really closer to the ground truth (indigo-carmine-sprayed image): GEI using CII or AVSPI. The conventional evaluation method (looking at pixel-to-pixel differences between two images) cannot be used here since the indigo-carmine-sprayed images have slight pixel misregistration between CII and AVSPI. Touching or moving the sample and deformation of the surface due to moisture inevitably occurs during the indigo-carmine-fluid spraying process.

To counteract this problem, the texture feature vectors extracted from the indigo-carmine-sprayed image (ground truth) and the image that results from the proposed method are calculated. The distance between these two vectors is used as the evaluation measure. The procedure comprises several steps, which are listed as follows:

1. Binary image of the microstructure.
The extracted GGI that have gray-level value information is binarized to remove instability due to the gray value distribution. The minimum threshold values are used in this binarization to make the groove binary images as thick as possible.
2. Multidimensional GLCM feature vector.
GLCM²¹^,²² creates a matrix $P (i, j)$ that shows the probability that two image pixel pairs separated by displacement $D$ have the values of $i$ and $j$ . In this case, $i$ and $j$ have the values of 0 and 1, respectively. As Fig. 6(a) shows, displacement $D$ has four directions: 0 deg (horizontal), 90 deg (vertical), and 45 deg, and 135 deg (diagonal). A GLCM is created for each direction and then the four are averaged together. Thus, $P (i, j)$ is created for each displacement $D$ (1 to 16).
3. Four texture measures.
$P (i, j)$ itself cannot be used as the feature vector; four typical features are calculated from $P (i, j)$ . These four features are correlation, contrast, energy, and homogeneity. Their definitions and meanings are listed in Table 2. Here, $μ$ and $σ$ are the means and standard deviations, respectively. Figure 6(b) shows an example of these four texture vectors with 16-dimensions that correspond to displacement $D$ (1 to 16).
4. Final distance measure.
The distance between two textures can thus be quantified as the Euclidean distance between each texture measure vector. The feature vector is treated as 4-, 6-, 8-, and 16-dimension vectors in the following calculation; there were no significant differences between those numbers. We used one scalar distance value by adding the four different distances to create the final texture difference.
If the value is high, the texture from the image processing resembles the ground truth to a lesser extent. We defined the total difference in texture as the sum of these four differences.
Eq. (9)
$Distance of Texture = (Distance in Correlation) + (Distance in Contrast) + (Distance in Energy) + (Distance in Homogeneity) .$

Fig. 6

GLCM texture analysis: (a) definition of co-occurrence matrix and (b) four texture vectors.

Table 2

Four texture measures.

Texture measures derived from GLCM	Definition
1. Correlation	$\sum_{i, j} \frac{(i - μ_{i}) (j - μ_{j}) P (i, j)}{σ_{i} σ_{j}}$
2. Contrast	$\sum_{i, j} {\| i - j \|}^{2} P (i, j)$
3. Energy	$\sum_{i, j} P {(i, j)}^{2}$
4. Homogeneity	$\sum_{i, j} \frac{P (i, j)}{1 + \| i - j \|}$

4.4.

Evaluation Experiments Using Subimage Division

The entire binary image shown in Fig. 5 is divided into $4 \times 4 = 16$ subimages, which are sized as $128 \times 96 pixels$ . The texture features are calculated for each subimage. The texture differences are evaluated by each subimage that has no accurate pixel registration. Figure 7(a) shows 16 subimages (regions A–P) for the indigo-carmine-sprayed image. The binary images of these subimages are generated accordingly using the procedure discussed in Sec. 4.3. Figure 7(b) shows an example binary subimage (region J). Here, Distance_P denotes the texture feature distance between the indigo-carmine-sprayed subimage (ground truth) and the AVSPI subimage, and Distance_I denotes the distance between the indigo-carmine-sprayed subimage (ground truth) and the CII subimage.

Fig. 7

Quality evaluation of microstructure enhancement using subimage division. (a) 16 subimages (regions A–P). (b) Binary GGI in region (J) and the distance between texture vectors. (c) Objective evaluation result. (d) Subjective evaluation result.

Figure 7(c) shows the objective evaluation result for the above 16 subimages. Only the positive subimages appear; the negative subimages are masked with blue squares. Here, the positive subimage represents the situation where Distance_P is smaller than Distance_I, which means that the AVSPI result more closely resembles the ground truth image than does the CII. The negative subimage represents the opposite situation, in which the CII resembles the ground truth to a greater extent than does the AVSPI image. A subjective evaluation was also executed using 11 people in order to confirm the results of the objective evaluation. These 11 volunteers were asked which image is more similar to the ground truth: the AVSPI or CII. Figure 7(d) shows the result of that subjective evaluation.

Table 3 shows the final positive/negative score comparison of the objective and subjective evaluations—in other words, the numbers of subimages that were assigned positive and negative scores. The objective evaluation results show that the AVSPI images performed better than the CIIs by 11 to 5 for all subimages and by 7 to 1 for the lower eight subimages. Since these lower eight images ( $I - P$ ) are not influenced with a puddle of indigo-carmine fluid or specular reflections, the statistics for these areas are likely to be more accurate. Subjective evaluation results show that the AVSPI images performed better than did the CIIs by 9 to 7 for all subimages and by 6 to 2 for the lower eight. From these results, we conclude that polarimetric processing using AVSPI can enhance microstructures and bring the image closer to the indigo-carmine-sprayed image than can CII processing.

Table 3

Positive/negative scores in the evaluation results.

Positive/negative score from objective evaluation		Positive/negative score from subjective evaluation
All 16 subimages	Lower 8 subimages	All 16 subimages	Lower 8 subimages
$11 / 5$	$7 / 1$	$9 / 7$	$6 / 2$

5. Online Polarimetric Rigid Endoscope Using Averaged Subtracted Polarization Image

5.1.

Prototype Endoscope Using a Beam-Splitter-Type Color Polarization Camera

Currently, no flexible endoscopes have yet been designed; thus, a rigid endoscope with polarimetric cameras is used instead. As shown in Fig. 8(a), the beam splitter (BS)-type endoscope hardware³^,⁴ consists of three parts: a special color-polarization camera, a rigid scope, and a polarized LED light. The FD-1665 3CCD camera (Fluxdata, Inc.)²³ uses a three-way prism BS that splits incoming white light into three components with identical spectra and three single-chip color CCD image sensors mounted with a $0 \deg / 45 \deg / 90 \deg$ polarizer. Only two ( $0 \deg / 90 \deg$ ) CCDs are active in this prototype, and C0 and C90 are captured simultaneously in real time.

Fig. 8

Prototype endoscope using a BS-type color polarization camera.³^,⁴^,²³ (a) Prototype schematic of a BS-type polarimetric endoscope; (b) Advantage of AVSPI which can cancel differences in two lightings and optical characteristics.

A specially designed ring-shaped LED light is mounted at the tip of the borescope. It consists of eight sub-LED lamps with two channels for four lamps. Channels 1 and 2 have a 0 deg and 90 deg polarizer, respectively, on their lamps. This system has two operation modes: polarized and nonpolarized imaging. Nonpolarized intensity imaging can be realized by averaging the C0 and C90 images when all LED-polarized lights are on.

Figure 8(b) shows the effectiveness of AVSPI using this BS-type polarimetric endoscope. The prototype endoscope has two types of illumination and two polarizers and CCDs with BS which have two optical paths. These lighting and optical characteristics are not completely the same, and there is a little difference. Therefore, four types of subtracted polarization images; L0-Dif, L90-Dif, C0-Dif, and C90-Dif images include some artifacts and poor results. However, AVSPI is generated by changing the polarization axis of the light and optical path simultaneously, and cancels these differences.

5.2.

Prototype Endoscope Using a Single-Chip Polarization Camera

We tested another implementation using a “PolarCam”^TM (4D Technology Corporation)²⁴ polarization camera, which has a monochromatic single-chip polarimetric image sensor with aluminum-wire grid pixelated mosaic patterns²⁵ and a monochromatic CCD image sensor. The pixelated polarizers, which use an array of micropatterned wire grids with four directions ( $0 \deg / 45 \deg / 90 \deg / 135 \deg$ ), are mounted and only the 0 deg and 90 deg pixels are used for polarimetric image processing, just as in the previously discussed BS-type prototype. These two polarization monochromatic images (C0 and C90) are simultaneously provided in real time.

Figure 9(a) shows this single-chip polarimetric endoscope prototype with an explanation diagram. The rigid borescope with sequential color lightings and ring-shaped LED light shown in Fig. 8 are attached with a C-mount adapter lens. When subtracted polarization (L0-Dif and L90-Dif) is used, the two images sampled at slightly different pixel positions are subtracted from each other; the resultant images look very noisy and these undesirable artifacts are difficult to remove using image calibration or interpolation techniques. Here, AVSPI can be adapted to solve the pixel registration problem, as shown in Fig. 9(b). Sampling points of averaged co- and cross-polarized images have virtually the same position “ $P$ ” at the midpoint of the diagonal where the 0 deg and 90 deg pixels meet because those pixels change the role of the co- and cross-polarization pixels according to the switching L0 and L90. Thus, the subtraction process can be executed without degradation.

Fig. 9

Prototype endoscope using a single-chip polarization camera. (a) Prototype schematic of a single-chip polarimetric endoscope; (b) Advantage of AVSPI which can compensate pixel mis-registration used for subtraction.

If PolarCam had checker-board-patterned pixelated polarizers with only $0 \deg / 90 \deg pixels$ , a higher-resolution AVSPI image would be obtained by the averaged process using four pixels.

5.3.

Extinction Ratio and Averaged Subtracted Polarization Image Quality

Although realizing a higher extinction ratio of the polarization capturing is quite important, practical polarimetric cameras have lower extinction ratios due to the influences of optical interference, optical cross-talk, and misregistration of the pixelated polarizer. In this section, the extinction ratio tolerance of the prototype polarimetric endoscopes used to enhance microstructures is discussed.

A new definition of the averaged extinction ratio $q$ is introduced for the use of AVSPI as follows:

Eq. (10)

q = \frac{MEAN {(I [L 0 C 0] + I [L 90 C 90]) / 2}}{MEAN {(I [L 0 C 90] + I [L 90 C 0]) / 2}} .

Here, MEAN{} denotes mean pixel value of all or the central target window of the image.

It is quite hard to measure extinction ratios of more than 100 with 8-bit (0 to 255) cameras, because image noise on the cross-polarized black image drastically influences the extinction ratio with meaningless results. To overcome this problem, the variable exposure time technique was developed.

Table 4 shows the calculated extinction ratios for the manually rotated polarizer setup and two prototype endoscopes according to the above measurement procedure.

Table 4

Comparison of the extinction ratios of the three experimental systems.

	Manually rotated polarizer setup (M)	BS-type polarimetric endoscope (B)		Single-chip polarimetric endoscope (S)
	Manually rotated polarizer setup (M)	Exposure time (same)	Exposure time (different)	Single-chip polarimetric endoscope (S)
R	6500	86	98	26
G	3100	83	106
B	400	82	108

In cases where a manually rotated polarizer setup is used, the exposure times are $10^{3}$ and $10^{6} μ s$ for co- and cross-polarized images, respectively. It is reasonable that $q$ is around 3000 in the G-channel; note that the longer wavelength channel has a higher extinction ratio.

The BS-type color polarimetric endoscope has an extinction ratio of around 85 when the same exposure time is used and around 100 when different exposure times are used ( $5 \times 10^{5} μ s$ and $10^{7} μ s$ for co- and cross-polarized images, respectively). These values are somewhat different, but errors are within a tolerable range.

The single-chip polarimetric endoscope has the lowest extinction ratio of around 26; therefore, the same exposure time can be used for both co- and cross-polarized images.

Figure 10 compares the three experimental systems listed in Table 4 by providing the three different AVSPI images of a porcine gastric mucosa that were obtained using them.

Fig. 10

Comparison of the AVSPI images obtained by the three experimental systems. (a) Manually rotated setup (M) and BS-type endoscope (B). (b) BS-type polarimetric endoscope (B) and single-chip polarimetric endoscope (S).

Frozen porcine stomach samples from a meat factory are thawed and washed and these porcine gastric mucosa samples are observed.

Figure 10(a) shows the comparison between a manually rotated setup (M) and a BS-type polarimetric endoscope (B). Figure 10(b) shows the comparison between a BS-type polarimetric endoscope (B) and a single-chip polarimetric endoscope (S). Note that this experiment was performed using the same mucosa sample in all three setups. Although certain differences exist in the captured area, the magnitude, lighting uniformity, and distinctive texture characteristics of the enhanced microstructure are very similar, which suggest that the contrast enhancement procedure using AVSPI is sufficiently robust so as to not be influenced by extinction ratio differences of more than about 30.

5.4.

Additional Image Processing for Enhancement

The GEI processing in Sec. 3 has the problem in which GEI using AVSPI are almost gray and without color, as shown in Fig. 5(l). Therefore, we introduce new image processing for enhancement named PISI referred in Sec. 1. It is synthesized using CII and the weighted summation of the GGI so that the groove region is enhanced with a blue hue, while the background maintains its mucosal color.

Eq. (11)

{\begin{cases} PISI (R) = CII (R) - W \times GGI \\ PISI (G) = CII (G) - W \times GGI \\ PISI (B) = CII (B) \end{cases} .

$W$ denotes a weighting parameter which is decided from the appearance and the typical value is 0.5. As GGI can be generated from either CII or AVSPI, PISI can be generated from these two image processing above.

The other additional image processing discussed is referred to as “colored AVSPI” (CAVSPI) and uses neither groove detection processing nor blue color enhancement processing to generate PISIs. Rather, through experiments with the porcine gastric mucosa, we found that AVSPI can use intensity images to enhance detailed microstructures that are almost invisible and that have no color information. Therefore, the luminance component from AVSPI and the chromaticity component from a CII are synthesized together to generate CAVSPI.

Eq. (12)

[\begin{array}{l} CII (Y) \\ CII (Cr) \\ CII (Cb) \end{array}] = RGB_to_YCrCb ([\begin{array}{l} CII (R) \\ CII (G) \\ CII (B) \end{array}]), [\begin{array}{l} AVSPI (Y) \\ AVSPI (Cr) \\ AVSPI (Cb) \end{array}] = RGB_to_YCrCb ([\begin{array}{l} AVSPI (R) \\ AVSPI (G) \\ AVSPI (B) \end{array}]), [\begin{array}{l} CAVSPI (R) \\ CAVSPI (G) \\ CAVSPI (B) \end{array}] = YCrCb_to_RGB ([\begin{array}{l} gain \times AVSPI (Y) \\ CII (Cr) \\ CII (Cb) \end{array}]) .

Here, RGB_to_YCrCb() and YCrCb_to_RGB() mean color space conversion and inverse conversion functions between RGB color space and YCrCb or the luminance and chromaticity color space. The typical value of gain is set to 4.0.

Figure 11 shows a comparison of enhanced microstructure images using the BS-type prototype endoscope. Frozen and thawed porcine gastric mucosa samples are observed. Here, two additional types of image processing are used on a $658 \times 492 pixel$ image without $1 / 2$ sampling, and LPF ( $5 \times 5 pixels$ ) and space differential mask ( $7 \times 7 pixels$ ) are used.

Fig. 11

Enhanced microstructure on the porcine gastric mucosa using the BS-type endoscope: (a) CII, (b) PISI using CII, (c) AVSPI, (d) PISI using AVSPI, and (e) CAVSPI.

Figure 11 columns (a) and (b) (on the left side of the vertical dotted line) show close-up CIIs and PISI using CII of four different portions from several samples. Columns (c), (d), and (e) (on the right side of the vertical dotted line) show AVSPI, PISI using AVSPI, and CAVSPI, respectively. The surface microstructure is nearly invisible in column (a) and (b), even though image enhancement is applied in the case of (b). Conversely, detailed structures can be seen in AVSPI in column (c), but it has no color as discussed above. PISI [shown in column (d)] shows more accurate groove structures than (b) with correct mucosal color, but groove information seems a little poor. CAVSPI [shown in column (e)] has both the detailed microstructure information of AVSPI and correct background mucosal color, which is completely different from intensity-based edge detection such as (b). These results indicate that the proposed enhancement procedure using AVSPI and online endoscope can be quite effective.

6. Conclusion

An image enhancement method for endoscopy of otherwise invisible surface microstructures on translucent mucosae using polarimetric imaging has been proposed. What sets this method apart from other methods can be summarized as the following three points:

First, as the paper has theoretically and experimentally discussed, microstructures cannot be viewed under white nonpolarized lighting; however, the subtracted polarization technique can significantly enhance a microstructure’s contrasts. Polarimetric imaging is not just contrast enhancement of CIIs. It does not detect shadow information but depolarized information of the surface translucent medium such as mucosa and provides a completely different image that has no disturbance from base structures such as deep-layer blood vessels under the mucosa. Compared with the narrow band imaging, which is widely used, polarimetric imaging has the advantage of obtaining full color information and brighter images.

Second, AVSPI, which uses two pairs of co- and cross-polarized images that are captured under two different linearly polarized illumination procedures and then averaged, is introduced. AVSPI can compensate the image quality difference caused by surface oblique lighting, which is commonly used in endoscopy. Since AVSPI averages result from different polarized incident lightings, only depolarized characteristics are observed, and real optical anisotropy, such as birefringence, cannot be detected; however, from the experiments using porcine mucosa discussed in this paper, surface microstructures seem to have no optical anisotropy.

The quality of GEIs obtained using the proposed technique was evaluated using GLCM texture analysis, which showed that AVSPI is more effective than CII. Several microstructure enhancement techniques were then developed to obtain better endoscopic medical diagnoses. PISI and CAVSPI are also introduced in order to improve AVSPI-obtained GGIs.

Third, new prototype endoscopic hardware with extinction ratios of around 100 and 26 has been constructed using various CCD polarimetric cameras, rigid scopes, and polarized ring-shaped two-channel LED lights. This hardware was used to reveal the relationship that exists between image quality and extinction ratio. Here, AVSPI has a large advantage in its ability to improve the image quality degradation of subtracted polarized images that occurs due to different optical paths being used with beam-splitter, nonuniform lighting, and polarimetric camera pixel misregistrations.

Although the extinction ratio of the prototype endoscope is a factor of 10 smaller than that of the manually rotated setup, the quality of AVSPI-obtained images was almost the same, which indicates that the polarimetric endoscopy technique using AVSPI is practical for implementation in future hardware designs. Discussion of the relationship between image quality and extinction ratio values may be a new topic in this field.

Continuing studies will most likely focus on downsizing the polarized LED lighting at the tip and developing a small polarimetric color image sensor for in vivo observations.

Acknowledgments

The author is particularly grateful for the assistance received from Prof. N. Yahagi. MD and Y. Ochiai. MD of Keio University on the experimental method using porcine stomach tissue. I also thank Y. Tatsumi. MD at the Panasonic Healthcare Center, Dr. T. Azuma, Mr. K. Nobori for their important technical discussion, Drs. K. Orita and K. Shimono from Panasonic’s Advanced Research Division, and Dr. Brahm Pal Singh from Panasonic Eco Solutions Company for their kind support for my work. And I received encouragement and motivation for this research work from Mr.Y.Kitaoka and Mr.Y.Takenaga at Panasonic AVC networks company.

References

1.

R. Ozawa, “,”Electronic endoscope system,” (2006). Google Scholar

2.

K. Kanamori, “Image enhancement of surface micro-structure on mucous membrane by polarized image,” in 19th Symp. on Sensing via Image Information (SSII2013), (2013). Google Scholar

3.

K. Kanamori, “Polarimetric endoscope for image enhancement of surface micro-structure on mucosa,” IPSJ Trans. Comput. Vision Appl., 6 34 –38 (2014). http://dx.doi.org/10.2197/ipsjtcva.6.34 Google Scholar

4.

K. Kanamori, “Image enhancement of surface micro-structure on mucosa for polarimetric endoscopy,” Proc. SPIE, 9318 93180O (2015). http://dx.doi.org/10.1117/12.2075787 PSISDG 0277-786X Google Scholar

5.

Nizar A. Mullani, “Dermoscopy epiluminesence device employing cross and parallel polarization,” U.S. Patent No. US7, 167,243 (2007).

6.

S. L. Jacques, J. C. Ramella-Roman and K. Lee, “Imaging skin pathology with polarized light,” J. Biomed. Opt., 7 (3), 329 –340 (2002). http://dx.doi.org/10.1117/1.1484498 JBOPFO 1083-3668 Google Scholar

7.

J. C. Ramella-Roman et al., “Polarized light imaging with a handheld camera,” Proc. SPIE, 5068 284 –293 (2003). http://dx.doi.org/10.1117/12.518788 PSISDG 0277-786X Google Scholar

8.

V. Cerny, Z. Turek and R. Parizkova, “Orthogonal polarization spectral imaging,” Physiol. Res., 56 141 –147 (2007). PHRSEJ 0862-8408 Google Scholar

9.

Q. Zhu et al., “Experimental and theoretical evaluation of rotating orthogonal polarization imaging,” J. Biomed. Opt., 14 (3), 034006 (2009). http://dx.doi.org/10.1117/1.3130268 JBOPFO 1083-3668 Google Scholar

10.

M. H. Smith et al., “Mueller matrix imaging polarimetry in dermatology,” Proc. SPIE, 3911 210 –216 (2000). http://dx.doi.org/10.1117/12.384904 Google Scholar

11.

A. Pierangelo et al., “Use of Mueller polarimetric imaging for the staging of human colon cancer,” Proc. SPIE, 7895 78950E (2011). http://dx.doi.org/10.1117/12.878248 Google Scholar

12.

T. Novikova et al., “Polarimetric imaging for cancer diagnosis and staging,” Opt. Photonics News, 23 (10), 26 –33 (2012). http://dx.doi.org/10.1364/OPN.23.10.000026 Google Scholar

13.

J. Qi et al., “Polarized multispectral imaging in a rigid endoscope based on elastic light scattering spectroscopy,” Biomed. Opt. Express, 3 (9), 2087 –2099 (2012). http://dx.doi.org/10.1364/BOE.3.002087 BOEICL 2156-7085 Google Scholar

14.

J. Qi et al., “Narrow band

3 \times 3

Mueller polarimetric endoscopy,” Biomed. Opt. Express, 4 (11), 2433 –2449 (2013). http://dx.doi.org/10.1364/BOE.4.002433 BOEICL 2156-7085 Google Scholar

15.

S. Rivet, A. Bradu and A. Podoleanu, “70 kHz full

4 \times 4

Mueller polarimeter and simultaneous fiber calibration for endoscopic applications,” Opt. Express, 23 (18), 23768 –23786 (2015). http://dx.doi.org/10.1364/OE.23.023768 OPEXFF 1094-4087 Google Scholar

16.

S. Manhas et al., “Demonstration of full

4 \times 4

Mueller polarimetry through an optical fiber for endoscopic applications,” Opt. Express, 23 (3), 3047 –3054 (2015). http://dx.doi.org/10.1364/OE.23.003047 OPEXFF 1094-4087 Google Scholar

17.

K. Kagawa et al., “Variable field-of-view visible and near-infrared polarization compound-eye endoscope,” in 34th Annual Int. Conf. of IEEE EMBS, 3720 –3723 (2012). Google Scholar

18.

N. Thekkek et al., “Modular video endoscopy for in vivo cross-polarized and vital-dye fluorescence imaging of Barrett’s associated neoplasia,” J. Biomed. Opt., 18 (2), 026007 (2013). http://dx.doi.org/10.1117/1.JBO.18.2.026007 JBOPFO 1083-3668 Google Scholar

19.

N. T. Clancy et al., “Polarised stereo endoscope and narrowband detection for minimal access surgery,” Biomed. Opt. Express, 5 (12), 4108 –4117 (2014). http://dx.doi.org/10.1364/BOE.5.004108 BOEICL 2156-7085 Google Scholar

20.

A. Pigula et al., “Video-rate dual polarization multispectral endoscopic imaging,” Proc. SPIE, 9333 93330N (2015). http://dx.doi.org/10.1117/12.2079726 PSISDG 0277-786X Google Scholar

21.

R. M. Haralick, K. Shanmugam and I. Dinstein, “Textual features for image classification,” IEEE Trans. Syst. Man. Cybern., SMC-3 (6), 610 –621 (1973). Google Scholar

22.

Gray-Level Co-Occurrence Matrix (GLCM): the Mathworks, Inc., (2015) http://jp.mathworks.com/help/images/gray-level-co-occurrence-matrix-glcm.html January ). 2015). Google Scholar

23.

Imaging Polarimeters: FluxData, Inc., (2015) http://www.fluxdata.com August ). 2015). Google Scholar

24.

PolarCam SnapShot Micropolarizer Cameras: 4D technology, (2015) http://www.4dtechnology.com/products/polarcam.php August ). 2015). Google Scholar

25.

T. York et al., “Bioinspired polarization imaging sensors: from circuits and optics to signal processing algorithms and biomedical applications,” Proc. IEEE, 102 (10), 1450 –1469 (2014). http://dx.doi.org/10.1109/JPROC.2014.2342537 IEEPAD 0018-9219 Google Scholar

Biography

Katsuhiro Kanamori received his BS and MS degrees in applied physics from Waseda University, Japan, in 1984 and 1986, respectively. He joined Panasonic in 1986 and received his PhD from Chiba University in 2001 for his study of all-purpose, high-speed color image processing algorithm. His current interests are R&D of the new image sensors and processing for biomedical imaging with physics-based approaches.

Citation Download Citation

Katsuhiro Kanamori "Averaged subtracted polarization imaging for endoscopic diagnostics of surface microstructures on translucent mucosae," Journal of Biomedical Optics 21(7), 071105 (24 December 2015). https://doi.org/10.1117/1.JBO.21.7.071105

Published: 24 December 2015

Access the abstract

JOURNAL ARTICLE
13 PAGES

DOWNLOAD PAPER SAVE TO MY LIBRARY

GET CITATION

CITATIONS

Cited by 5 scholarly publications and 1 patent.

Explore citations on Lens.org

RIGHTS & PERMISSIONS

Get copyright permission Get copyright permission on Copyright Marketplace

KEYWORDS

Polarization

Polarimetry

Image enhancement

Endoscopes

Cameras

Image processing

Endoscopy

1.

Introduction

2.

Theoretical Background