2.1.

Overview of the Proposed Method

Figure 1 shows a flow chart of the proposed method. After capturing an image using a face-capturing camera, the eye region of the image is detected [steps (1) and (2) in Fig. 1]. A detailed explanation is provided in Sec. 2.3. The pupil center in the eye region is detected [step (3) in Fig. 1] (see Sec. 2.4). In addition, the SRs are localized [step (4) in Fig. 1], a detailed description of which is given in Sec. 2.5. To calculate the gaze point accurately, it is important to detect the exact centers of the pupil and the SR.^12,13 In Sec. 2.6, we describe the compensation method used to obtain a more accurate pupil boundary that is not distorted by an eyelid. The gaze point of each eye is then calculated using a geometric transform [step (5) in Fig. 1]. Then, the gaze positions of the two eyes are combined using a fuzzy algorithm based on quality measurements [step (6) in Fig. 1], a detailed description of which is provided in Sec. 2.7 through Sec. 2.9.

Fig. 1

Flow chart of the proposed method.

2.2.

Proposed Gaze Detection System

Our gaze detection system includes two parts, an NIR illuminator and a single camera. As shown in Fig. 2, a conventional Web camera with a universal serial bus (USB) interface is used. The NIR cutting filter of the camera is removed and a NIR passing filter is included in its place.^9,14–16 The NIR light-emitting diode (LED) illuminator is positioned below the camera. In this way, we can capture the facial image of the user without any adverse effects from changes in the visible environmental light conditions. In previous research, the edge between the pupil and the iris was shown to be clearer with an NIR light of wavelength 850 nm or longer.¹⁷ That is, the edge is hardly seen with the visible light in the special case of Asian people whose iris is dark. However, the camera sensor becomes less sensitive with an increase in wavelength.¹⁸ Considering both the clearness of the edge and the camera sensitivity, we chose an NIR illuminator with a wavelength of 850 nm for our system. This wavelength avoids the glare from entering the user’s eyes. To capture a magnified image of the user’s face, a zoom lens is attached to the USB camera. The specifications of the camera and the NIR LED illuminator are as follows:

Fig. 2

Proposed gaze detection system.

2.3.

Eye Region Detection

In this section, we describe the method of detecting the eye region in a facial image, which is step (2) in Fig. 1. Figure 3 shows examples of facial images captured by our gaze detection system. To reduce motion blur and increase the camera’s depth of field (DOF), we used a zoom lens with a larger F# of 5.5 and reduced the camera exposure time. F# is usually defined by the ratio of focal length to the diameter of the camera lens (effective aperture).²⁰ Hence, the larger the focal length or the smaller the diameter of the camera lens is, the F# increases. The DOF is the Z-distance range in which a focused image can be obtained. In general, the DOF of a lens increases as the F# of the lens increases. In addition, with a reduced camera exposure time, motion blur decreases. However, the captured image becomes darker with a lens with a larger F# and reduced camera exposure time, but a bright SR from the NIR illuminator can still be seen, as shown in Fig. 3. We, therefore, propose a method for detecting the eye region in a dark image based on the SR, as shown in Fig. 4. Our gaze detection system uses NIR light, and the eye region is detected using the SR near the pupil. By observing the captured image, the SR that occurs near the pupil of the user generally has a maximum gray value due to the high reflectance rate of the cornea surface. Therefore, when a window mask ($3\times 3\text{pixels}$) is sequentially searched from the upper-left corner to the bottom-right corner, if all pixel values in the window mask are bigger than the fixed threshold value, it is determined that the window mask includes the SR. Three different results are possible: the user’s eyes are not detected, only one eye is detected, and both eyes are detected.

Fig. 3

Example of images captured by our gaze detection system.

Fig. 4

Example of detecting the eye region based on the SR: (a) both eyes are detected, and (b) an erroneous detected region is found along with the detection of both eyes.

Figure 4 shows an example of detecting an eye region based on the SR. Figure 4(a) shows an example in which the user’s eyes are accurately detected. In this case, because the SR pixel values are brighter than the pixel values of the other skin regions, only the user’s eyes are properly detected. However, in Fig. 4(b), there are three regions where all pixel values in the window mask are larger than the fixed threshold value. In this case, if one eye is closed, other skin regions can be incorrectly determined as an eye region. In addition, because the SR pixel value can be changed according to the Z-distance, using a fixed threshold value causes an eye detection error. We, therefore, propose the following method for detecting an eye region. In a captured facial image, when a user opens their eye, both the SR and the pupil can be seen. The SR is generally the brightest area, and the pupil is the darkest. Therefore, if the difference in value between the SR and the pupil is maximized within a certain region, the region is regarded as an eye candidate area. Figure 5 shows a flow chart of the proposed method used for eye region detection.

Fig. 5

Flow chart of the proposed method for eye region detection.

To detect an eye region, the input image is divided into $M\times N$ sub-blocks [step (2) of Figs. 5 and 6(b)]. The size of each sub-block is $64\times 60\text{pixels}$, and total 500 sub-blocks ($25\times 20$) are defined as shown in Fig. 6(b). In each sub-block, the maximum and minimum pixel values are measured. Then, the difference between the maximum and minimum values is also calculated [step (3) of Fig. 5]. Based on the difference value, the sub-blocks are sorted in descending order [step (4) of Fig. 5]. For example, assuming that there are five sub-blocks that have the difference values of 200, 220, 205, 202, and 230, respectively, the sub-blocks are sorted as follows: the fifth sub-block (230), the second block (220), the third block (205), the fourth block (202), and the first block (200). The four candidate sub-blocks with the first, second, third, and fourth highest difference values are selected as eye candidate regions [step (5) of Figs. 5 and 6(c)]. With this example, the fifth sub-block (230), the second block (220), the third block (205), and the fourthblock (202) are selected. The average value of all candidate regions is then calculated as the threshold (${\mathrm{th}}_1$) [step (6) of Fig. 5]. The two regions, whose average gray level is smaller than ${\mathrm{th}}_1$, are determined to be an eye region [step (7) of Figs. 5 and 6(d)].

Fig. 6

Image results of eye region detection based on the SR: (a) the original image, (b) the input image divided into $M\times N$ sub-blocks, (c) the resulting image after selecting four candidate regions, and (d) the final eye detection image.

Finally, once the two eye regions are detected, the position of the maximum gray value is determined to be the SR position. An example of images of eye region detection are shown in Fig. 7. In successive images, the accuracy of the eye region detection based on the SR is improved by tracking the SR within the specified range based on the average position of the $Y$ coordinate of the eye position in the previous image.

Fig. 7

Image results of eye region detection.

2.4.

Pupil Center Detection

To calculate a detected gaze, it is necessary to accurately detect the center position of the pupil and the SR. In this section, we describe the method for detecting the pupil center in an eye region, which corresponds to step (3) in Fig. 1. Figure 8 shows a flow chart of the proposed method used for pupil-center detection.

Fig. 8

Flow chart of the proposed method used to detect the center of the pupil.

Based on the detected eye region, as shown in Fig. 7, the region of interest (ROI) for pupil detection can be defined as shown in Fig 9(a). The brightness of the ROI region is changed through a stretching of the histogram [step (2) of Figs. 8 and 9(b)]. For the stretching of the histogram, the average brightness of the input ROI region is calculated. Based on the average brightness, the brightness of the input ROI image is compensated. After the histogram is stretched, the ROI image is transformed through two binarization stages. The first stage is to separate the pupil from the other regions [binarization image 1, as shown in step (3) of Figs. 8 and 9(c)]. The second stage is to separate the SR from the other regions [binarization and morphology image 2, as shown in step (8) of Figs. 8 and 9(h)].^14,15,21 From the processed image achieved through the stretching of the histogram, we can obtain the maximum and minimum pixel values. The thresholds for binarization images 1 and 2 are determined based on the maximum and minimum pixel values, respectively, by which the binarized image can be robust to variations in illumination. Regions other than the pupil region are then erased through morphological processing [step (4) of Figs. 8 and 9(d)]. With morphology processing, erosion and dilation are conducted four times each to erase any unnecessary parts in the eye image, such as the eyelashes. Figure 9(c) shows parts of the eyelashes. After the morphological processing shown in Fig. 9(d), the smaller parts of the eyelashes are merged together and erased. Next, the largest region is found through component labeling [step (5) of Fig. 8 and 9(e)]. The edge of the largest region is then detected using a canny edge detection algorithm [step (6) of Fig. 8 and 9(f)].²² Most of the outer edge line of the largest region is extracted using a convex hull algorithm [step (7) of Figs. 8 and 9(g)].^21,23 Using the outermost edge line image [Fig. 9(g)] and the binarization and morphology image 2 [Fig. 9(h)], the image in Fig. 9(i) is obtained. Figure 9(i) is obtained by removing the overlapped area of Figs. 9(g) and 9(h) from Fig. 9(g). Figure 9(g) is obtained to compensate for the distortion of the pupil by the SR. The pupil center is then detected using an ellipse-fitting algorithm [step (10) of Figs. 8 and 9(j)]. The pupil shape is not usually a circular one.²⁴ In addition, when a user gazes at the monitor corner, the shape of the pupil is more distorted. Hence, we used the ellipse-fitting algorithm to detect the pupil boundary instead of circle fitting one considering the detection accuracy and speed. The result of the pupil detection is shown in Fig. 9(k).

Fig. 9

Image results of pupil center detection: (a) the original image; the resulting images through (b) the stretching of the histogram, (c) the binarization of (b), (d) the morphological processing of (c), (e) the component labeling of (d), (f) the use of the canny edge detection algorithm on (e), (g) the convex hull of (f), (h) the SR binarization of (b), (i) the removal of the overlapped areas of (g) and (h) from (g), and (j) the application of the ellipse-fitting algorithm of (i); and (k) the final pupil detection image.

Figure 10 shows the results of the pupil center detection. To obtain a more accurate gaze position, the detected pupil center is not represented as an integer value, but rather as a float value.

Fig. 10

Resulting images of pupil center detection.

2.5.

Corneal Specular Reflection Center Detection

In this section, we describe the method for detecting the corneal SR center in an eye region; this corresponds to step (3) of Fig. 1. Based on the detected eye region, as shown in Fig. 7, the ROI for detecting the corneal SR is defined in Fig. 11(a). The ROI region is binarized, which discriminates the SR region from the other regions [Fig. 11(b)]. Because other regions, such as a reflection on the lachrymal gland, can have a similar gray level as that of the SR, component labeling is performed on the binarized image.^9,16 The region nearest to the detected pupil center is selected as the corneal SR region [Fig. 11(c)]. In addition, the geometric center of the selected SR region is determined as the SR center [Fig. 11(d)].^9,16

Fig. 11

Resulting images of corneal SR center detection: (a) the original image, the resulting images through (b) binarization and (c) component labeling, and (d) the final SR detection image.

To obtain a more accurate gaze position, the detected SR center is not represented as an integer value, but instead as a float value. As shown in Fig. 11(d), the geometric center of the selected SR region is determined as the SR center. That is, with the pixels of the selected SR region, the center positions of the $X$ and $Y$ axes are calculated. Since the $X$ center position is calculated by dividing the summation of all the $X$ positions of the pixels (of the SR region) by the number of pixels, the calculated $X$ center position can become a floating number (not an integer one). Like this, the calculated $Y$ center position can also be a floating number. If we convert these floating numbers into integers by truncating the decimal fraction, the subpixel information of the SR position is not considered for calculating gaze position. Hence, in order to obtain a more accurate gaze position, we use the floating number without the truncation of the decimal fraction in the intermediate calculation. Consequently, the final calculated gaze position is represented as the floating number, and we represent it as the integer just at the final stage of calculation.

2.6.

Compensation for Pupil Distortion

To calculate a detected gaze, it is necessary to accurately detect the pupil and the SR center position. The pupil is often covered by an eyelid, and pupil shape is frequently distorted, as shown in Fig. 12. This can degrade the accuracy of the pupil detection.

Fig. 12

Pupil distortion by an eyelid²⁵ and the results of detecting the pupil boundary and center: (a) the original image and (b) the resulting image after detecting the pupil boundary and center (the dotted lines show the correct pupil boundary and center, whereas the solid lines represent the detected pupil boundary and center).

To compensate the distortion of the pupil shape, we compared two methods. The concept of the first method is shown in Fig. 13. We first find the horizontal axis that has the largest distance between two pixels of the pupil boundary. Hereafter, for convenience, we call this the horizontal major axis. The upper and lower pixels of the pupil boundary are then separated based on the horizontal major axis. By calculating the average distance between the center of the horizontal major axis and the upper (or lower) pixels, we can determine the occluded region (among the upper or lower pixels) that shows the smaller distance. We then replace the pixels of the occluded region (the upper pixels in Fig. 13) with the pixels of a nonoccluded area (the lower pixels in Fig. 13). The detailed steps of the first method are shown in Fig. 14. First, the ROI region for the pupil detection is obtained as shown in Fig. 14(a). Through histogram stretching, binarization, morphological processing, and canny edge detection [Figs. 14(b) through 14(d)], the horizontal major axis in the image is detected using convex hull processing, as shown in Fig. 14(e).

Fig. 13

Concept of compensation method 1.

Fig. 14

Flow chart of compensation method 1 for the distortion of the pupil region by the eyelid.

When the horizontal major axis is found, the upper and lower pupil boundary pixels are divided as shown in Fig. 14(e). By calculating the average distance between the center of the horizontal major axis and the upper (or lower) pixels, we can determine the occluded region (among the upper or lower pixels) that shows the smaller distance. We then replace the pixels of the occluded region (the upper pixels in Fig. 14) with the pixels of the nonoccluded area (the lower pixels in Fig. 14), as shown in Fig. 14(f). In addition, we obtain the image of Fig. 14(g) for detecting the corneal SR region through binarization and morphological processing. First, we obtain the overlapped area of Figs. 14(f) and 14(g). Then, this overlapped region is removed from Fig. 14(f), and we can obtain Fig. 14(h). The reason why this overlapped region is removed is as follows. As shown in Fig. 14(d), the edge line detected by the canny operator includes the boundary of the corneal SR that is overlapped on the pupil boundary. Hence, the boundary of the corneal SR shows a concave shape attached on the pupil boundary, which causes the incorrect detection of a pupil boundary by the ellipse-fitting algorithm. To overcome this problem, we remove the overlapped region between the pupil boundary [Fig. 14(f)] and the SR region [Fig. 14(g)] and used the remaining boundary of Fig. 14(h) for the pupil edge detection by the ellipse-fitting algorithm.

The pupil center is then detected using the ellipse-fitting algorithm, as shown in Fig. 14(i). The resulting images from compensation method 1 are shown in Fig. 15.

Fig. 15

Resulting images from compensation method 1.

The second method for pupil shape compensation is similar to the first, except that, instead of replacing all pixels in the occluded area (the red-dotted pixels in Fig. 13), only the occluded pixels are replaced (the red-dotted pixels in Fig. 16). If a pixel (in the upper region) with a distance from the center of the horizontal major axis is smaller than the threshold, it is determined as a pixel to be compensated. Here, the threshold is determined as the average distance between the center of the horizontal major axis and the pixels of the lower region. The concept of the second method is shown in Fig. 16.

Fig. 16

Concept of compensation method 2.

The detailed steps of the second method are shown in Fig. 17. First, the ROI for the pupil detection is obtained, as shown in Fig. 17(a). Through histogram stretching, binarization, morphological processing, and canny edge detection [Figs. 17(b) to 17(d)], the image in Fig. 17(d) is obtained. In addition, we obtain the image in Fig. 17(e) for detecting the corneal SR region through binarization and morphological processing. By removing the overlapped area of Figs. 17(d) and 17(e) in Fig. 17 from 17(d), we can obtain Fig. 17(f), and the horizontal major axis is found. Based on the horizontal major axis, the upper and lower pupil boundary pixels are divided, as shown in Fig. 17(f). By calculating the average distance between the center of the horizontal major axis and the upper (or lower) pixels, we can determine the occluded region (among the upper or lower pixels) that shows the smaller distance. In addition, the average distance ($D_1$) between all the points of the lower region and the center of the horizontal major axis is calculated as the threshold. Only the lower region is used to obtain the threshold because the eyelid usually covers the pupil in the upper area. The distances between all the points in the upper region and the center of the horizontal major axis are calculated. If a distance is smaller than the threshold, the pixel point is compensated by $D_1$, as shown in Fig. 17(g). The outermost line is then obtained using the convex hull algorithm, as shown in Fig. 17(h). Additionally, the pupil center is detected using the ellipse-fitting algorithm, as shown in Fig. 17(i). The image results of compensation method 2 are shown in Fig. 18.

Fig. 17

Flow chart of compensation method 2 for the distortion of a pupil region by the eyelid.

Fig. 18

Image results from compensation method 2.

2.7.

Quality Score Measurements of Both Eyes

Our proposed gaze detection method uses a fuzzy algorithm to combine the gaze points of both eyes based on their quality scores. In this section, the method for obtaining the quality scores of both eyes is described, corresponding to step (4) of Fig. 1. We use four quality measurements: distortion (occlusion) by the eyelid, distortion (occlusion) by the SR, the level of circularity of the pupil, and the distance between the pupil boundary and the SR center.

Factor 1 (the amount of distortion by an eyelid) is calculated as the ratio of pixels of the pupil edge occluded by an eyelid [Fig. 19(a) to the number of pixels of an undistorted pupil edge in the upper region (based on the horizontal major axis) [Fig. 19(b)]. Factor 1 can be expressed using Eq. (1)

Fig. 19

Concept image of factor 1.

The second quality measure is the amount of distortion caused by the SR. Factor 2 is calculated as the ratio of the number of pixels of a pupil edge occluded by the SR [the number of pixels along the dotted line of square B in Fig. 20(b)] to the number of pixels in the distorted pupil edge in the lower part [the number of pixels along the solid line of square A in Fig. 20(a)]. Factor 2 can be expressed through Eq. (2)

Fig. 20

Concept image of factor 2 (a) pupil edge region in the case where the distortion is occluded by the SR (solid line in square A) (b) occluded pupil edge by the SR (dotted line in square B).

The third quality measure is the level of circularity of the pupil. Factor 3 is calculated as the ratio of the major to minor axes lengths of the pupil. The major and minor axes are detected using an ellipse-fitting algorithm, as described in Sec. 2.4 and Sec. 2.6. A pupil boundary is usually similar to a circular shape when a user gazes at the center of a monitor and becomes elliptical in shape when the user gazes at the corner of a monitor. In the latter case, factor 3 increases, and the consequent error when detecting an accurate pupil boundary and center can be increased.We, therefore, use factor 3 as the third quality measure. The closer to the circle shape the pupil boundary is, the smaller the factor 3 becomes (consequently close to 1). Factor 3 can be expressed through Eq. (3)

The fourth quality measure is the distance between the SR center and the pupil boundary pixel nearest to the SR center. The longer the distance between the pupil boundary pixel and the SR center, the less distortion that occurs because of the SR, as shown in Fig. 21. Therefore, the larger the fourth quality measure, the more accurate the pupil center detection.

Fig. 21

Concept image of factor 4 (a) when the SR occludes the pupil edge (dotted line) (the shortest distance (A) between the pupil boundary pixel and the SR center), and (b) when the SR does not occlude the pupil edge (the shortest distance (B) between the pupil boundary pixel and the SR center).

Using these four quality measurements, we can evaluate the accuracy of the detected pupil centers and the confidence levels of the calculated gaze points of both the left and right eyes. Based on the four quality measurements, the weight values for the gaze points of the left and right eyes are determined using a fuzzy algorithm (see Sec. 2.8).

2.8.

Obtaining the Weight Value for the Gaze Position based on a Fuzzy Algorithm

In this section, the method for obtaining the weight values of the gaze points of the left and right eyes based on a fuzzy algorithm is described. The final gaze position can be calculated more accurately by combining the gaze points of both eyes when considering the weight value. The basic concept of combining the gaze points of both eyes is shown in Fig. 22.

Fig. 22

Concept of combining the gaze points of both eyes.

Based on the four quality factors of each eye, as explained in Sec. 2.7, the gaze point score of each eye is obtained based on the fuzzy algorithm, as shown in Fig. 23. The weight values for the gaze positions of the left and right eyes are then determined using Eq. (4). The final gaze position is determined using the gaze positions of the left and right eyes when considering these weight values

Fig. 23

Procedure for calculating the scores using a fuzzy algorithm and the two weight values determined through Eq. (4).

To obtain the scores from Eq. (4), a fuzzy algorithm is used, as shown in Fig. 23. Using $w_1$ and $w_2$ and the gaze points [$G_{\text{eye}}(x,y)$] of each eye, the final gaze position [$G(x,y)$] is obtained using Eq. (5)

To use the fuzzy algorithm, it is important to determine the membership function. The membership function is designed according to the input value (factors 1 through 4 of Sec. 2.7), as shown in Fig. 24.

Fig. 24

Input fuzzy membership function.

The larger the values of factors 1 through 3 [Eqs. (1)–(3)], the larger the distortions of the pupil shape that occur. This usually decreases the confidence level of the gaze detection accuracy. However, for factor 4, a larger value indicates that the distortion of the pupil shape is smaller. This generally increases the confidence level of the gaze detection accuracy. All factors for the input value of the fuzzy algorithm are normalized within the range of 0 to 1 and for factor 4, by subtracting the value from 1 after normalization. All factors, 1 through 4, with smaller values represent a higher confidence level of the gaze detection accuracy.

The input values of the fuzzy algorithm can be categorized into two types: low (L) and high (H), as shown in Fig. 24. The fuzzy membership function for the input value is designed as shown in Fig. 24. The output membership function of the fuzzy algorithm is shown in Fig. 25. In addition, the output value (the score in Fig. 23) can be categorized as: low (L), middle (M), and high (H). The relationships between the input (factors 1 through 4) and output values (scores) are defined using the fuzzy rules shown in Table 2. As explained previously, factors 1 through 4 with smaller values represent a higher confidence level of gaze detection accuracy. Thus, if the value of all the factors 1 through 4 of one eye are low (L), for example, it means that the accuracy of the gaze point of the eye can be considered high, and we assign the output value (score of Fig. 23) as high (H). If the value of two of the factors 1 through 4 are low (L) and the other two factors are high (H), we assign the output value (score of Fig. 23) as middle (M). If the value of all the factors 1 through 4 of one eye is high (H), it means that the accuracy of the gaze point of that eye can be considered low, and we assign the output value (score of Fig. 23) as low (L).

Fig. 25

Output fuzzy membership function.

Table 2

Fuzzy rules of the relationship between the input (factors 1 through 4) and output values (scores).

The output (score) can be obtained using the fuzzy membership function and fuzzy rules. With one input value, we can obtain two outputs using the input membership function shown in Fig. 24. Because the number of inputs is four (factors 1 through 4), eight outputs are obtained using the input membership function.

For example, as shown in Fig. 26, two outputs [0.3 (L) and 0.7 (H)] are obtained by factor 1. Another two outputs [0.3 (L) and 0.7 (H)] are obtained by factor 2. In this way, factors 3 and 4 also produce two outputs [0.3(L) and 0.7(H)], respectively. Based on these four pairs of outputs, we can obtain the combined set as {[0.3 (L), 0.3 (L), 0.3 (L), 0.3 (L)], [0.3 (L), 0.3 (L), 0.3 (L), 0.7 (H)], [0.3 (L), 0.3 (L), 0.7 (H), 0.3 (L)], … [0.7 (H), 0.7 (H)¸ 0.7 (H)¸ 0.7 (H)]}. With one subset, we can determine one value (0.3 or 0.7) and one symbol (L, M, or H) based on the minimum or the maximum method and fuzzy rules of Table 2.^26,27

Fig. 26

Example of obtaining two output values from one input using the input membership function.

For example, with one subset [0.3 (L), 0.7 (H), 0.7 (H), 0.7 (H)], we can select 0.3 based on the minimum method. In addition, we can obtain L as the output based on Table 2 (when factors 1 through 4 are L, H, H, and H, respectively, we can state that the output is L, as shown in Table 2). We therefore obtain 0.3 (L) from [0.3 (L), 0.7 (H), 0.7 (H), 0.7 (H)] and call 0.3 (L) the inference value (IV) in this paper. If we apply the maximum method, we obtain 0.7 (L) as the IV. Because the number of subsets is 16 ($2\times 2\times 2\times 2$), the total number of IVs is also 16.

With one IV, we can obtain either one or two outputs (scores), as shown in Fig. 27. As shown in Fig. 27, if the IV is 0.7 (M), the corresponding outputs are ${\mathrm{S}}_1$ and ${\mathrm{S}}_3$. In this way, we can obtain multiple outputs (${\mathrm{S}}_1,{\mathrm{S}}_2,\dots {\mathrm{S}}_{\mathrm{N}}$) from 16 IVs. From the multiple outputs (${\mathrm{S}}_1,{\mathrm{S}}_2,\dots ,{\mathrm{S}}_{\mathrm{N}}$), we can determine one final output score based on a defuzzification method.^26,28 There are various defuzzification methods, of which five were chosen for this experiment: first of maxima (FOM), last of maxima (LOM), middle of maxima (MOM), mean of maxima (MeOM), and the center of gravity (COG)^26,28 FOM selects the first output IV calculated by the maximum. LOM selects the last output IV calculated by the maximum. MOM selects the middle of the first and last output IVs calculated by the maximum. MeOM selects the mean of the output IVs calculated by the maximum.²⁸

Fig. 27

Example of obtaining the output (score) from the IV and one final output (score) by the defuzzification method: (a) FOM, LOM, MOM, and MeOM, and (b) COG.

In Fig. 27, when all the IVs are assumed to be 0.7 (H), 0.6 (M), and 0.3 (L), the maximum IV is 0.7 (H). In addition, the two outputs (${\mathrm{S}}_1$ and ${\mathrm{S}}_3$) are calculated using 0.7 (H). The final outputs (scores) by FOM, LOM, MOM, and MeOM are therefore ${\mathrm{S}}_1$, ${\mathrm{S}}_3$, $({\mathrm{S}}_1+{\mathrm{S}}_3)/2$, and $({\mathrm{S}}_1+{\mathrm{S}}_3)/2$, respectively. For the COG, the output (score) is determined as S in Fig. 27(b) from the geometrical center [G in Fig. 27(b)] of the union area of three regions (${\mathrm{P}}_1$, ${\mathrm{P}}_2$, and ${\mathrm{P}}_3$).

2.9.

Calculating the Gaze Position by Combining Both Eyes with the Weight Values

With the output scores (Fig. 27) of the left and right eyes, we can obtain the weight values for the gaze positions of both eyes using Eq. (4). Using Eq. (5), we can obtain the final gaze position considering the weight values. To do so, we obtain the gaze positions of the left eye [$G_{\text{lefteye}}(x)$, $G_{\text{lefteye}}(y)$] of Eq. (5) and right eye [$G_{\text{righteye}}(x)$, $G_{\text{righteye}}(y)$] of Eq. (5) as follows. At the initial calibration stage, each user gazes at nine positions (${\mathrm{Q}}_1,{\mathrm{Q}}_2,\dots ,{\mathrm{Q}}_9$) on a monitor, as shown in Fig. 28. The nine positions of the pupil centers are obtained as shown in Fig. 29. Because the user’s head may move while gazing at the nine positions, the nine positions of the pupil centers are readjusted based on the SR positions in each image. We then obtain the nine positions of the pupil centers (${\mathrm{P}}_1,{\mathrm{P}}_2\dots ,{\mathrm{P}}_9$), and can define the relationship between each pupil subregion and monitor subregion. For example, pupil subregion 1 corresponds to monitor subregion 1. To obtain this relationship, we applied a geometric transform^9,16 and obtained the four matrices of the geometric transforms, as shown in Fig. 30.⁹ If the measured pupil center belongs to pupil subregion 1, geometric transform matrix 1 is used to calculate the gaze position on the monitor. If it belongs to pupil subregion 4, geometric transform matrix 4 is used to calculate the gaze position on the monitor.⁹

Fig. 28

Subregions of the monitor plane for a geometric transform.

Fig. 29

Example positions of the pupil and the SR centers when the user gazes at nine calibration positions: (a) the upper-left calibration point (${\mathrm{Q}}_1$), (b) upper-center calibration point (${\mathrm{Q}}_2$), (c) upper-right calibration point (${\mathrm{Q}}_3$), (d) middle-left calibration point (${\mathrm{Q}}_4$), (e) middle-center calibration point (${\mathrm{Q}}_5$), (f) middle-right calibration point (${\mathrm{Q}}_6$), (g) lower-left calibration point (${\mathrm{Q}}_7$), (h) lower-center calibration point (${\mathrm{Q}}_8$), and (i) lower-right calibration point (${\mathrm{Q}}_9$).

Fig. 30

Relationship between each pupil subregion and monitor subregion.

Using this, we can obtain the gaze positions of the left eye [$G_{\text{lefteye}}(x)$, ${\mathrm{G}}_{\text{lefteye}}(y)$] of Eq. (5) and the right eye [$G_{\text{righteye}}(x)$, $G_{\text{righteye}}(y)$] of Eq. (5). In addition, using Eq. (5), we can obtain the final gaze position on the monitor by considering the weight values.