2.1.

Construction of a Wide-Field Video-Capillaroscopy System

We developed a portable wide-field video-capillaroscopy system capable of imaging skin capillaries over a wider field of view than general capillaroscopy (Fig. 1). By combining a USB 3.0 camera (Toshiba Teli BU1203MC, Tokyo, Japan) having a large number of pixels with a high-resolution telecentric lens (VS Technology x1 VS-TCT1-65/S, Tokyo, Japan, numerical aperture = 0.111), it was possible to image an area of $7.4\mathrm{mm}\times 5.5\mathrm{mm}$ at a resolution of $4000\times 3000\text{pixels}$, and the field of view per pixel was $1.85\mu \mathrm{m}$. We evaluated the spatial resolution of the system using a USAF1951 test target (Edmund Optics #58-198, New Jersey). The working distance of the lens was 65 mm, and a cylindrical acrylic lens barrel with a φ12 mm aperture developed using a 3D printer was attached to the tip of the lens so that a focused image could be obtained by simply pressing it against the skin. Four shell-shaped white LEDs (OptoSupply LP-WA4K3131A, Hong Kong, China) were used as light sources inside the lens barrel to be illuminated at an angle of 45 deg to the skin surface. This microscope was connected to a PC with a cable (OMRON SENTECH NU3MBASU3S-2m, Ebina, Japan). The images were displayed and recordings were obtained using a viewer software (TechnoScope MKS-01, Saitama, Japan).

Fig. 1

Appearance and configuration of the developed system. (a) Schematic diagram and (b) internal structure of the proposed wide-field portable high-resolution video-capillaroscopy system.

2.2.

Capturing Video Images of Capillaries

2.3.

Image Processing

We performed image processing to extract capillaries from the video images in two stages: pretreatment of images (image stabilization, generation of a still image with capillary structure from the video image, removal of brightness unevenness, and contrast enhancement) and detection of capillary regions using semantic segmentation and quantifying their area and number. The image-processing procedure is shown in Fig. 2, and a detailed explanation is given below. We used programs written in Python for image processing in the following execution environment: Windows 10 Pro 64-bit operating system with a Core i7-6950X 3.00/3.50 GHz processor and 128 GB of memory.

Fig. 2

Schematic diagram of image processing. Pre-processing, U-Net training, and capillary area detection using the trained model are shown. Image size (horizontal pixel number, vertical pixel number, frames) is shown in the lower right area of figure.

2.4.

Image Stabilization

In the video images captured, we observed a motion blur resulting from the body movement of either the operator or the participant. To correct this blur and stabilize the image, we used template matching. The central $3900\times 2900\text{pixels}$ of the first frame of the video image were extracted as the template. In 150 frames of the video image of $4000\times 3000\text{pixels}$, the areas of size $3900\times 2900\text{pixels}$ with high similarity to the template were extracted. By creating a video image using the extracted images, we obtained the stabilized video image without motion blur. We used the matchTemplate function of OpenCV (version 4.1.0)³⁸ for processing and the TM_CCOEFF_NORMED method to calculate the similarity.

2.5.

Generation of Still Images from Video Images

In the video images of the skin capillaries, the capillaries themselves cannot be distinguished from their surroundings and can only be recognized when RBCs pass through them. Therefore, if a single frame is extracted, the capillary structure may have gaps at locations with no RBCs. Therefore, we processed 150 frames of images to depict capillary regions without gaps as places through which RBCs have passed. A method of averaging the 150 frames could be considered. However, if RBCs passed only rarely, the contrast would be low and the vascular regions might not be sufficiently depicted. Considering this, we focused on the minimum instead of average pixel values and processed the image as follows. In the color image, RBCs appeared redder than the surroundings. This makes the pixel value of the green channel at the RBCs smaller than that at the periphery when the image is divided into its constituent red (R), green (G), and blue (B) channels. By focusing on a single pixel and recording the color of the frame wherein the pixel value of the G channel is the lowest across the 150 frames, the information from the frame wherein RBCs exist can be obtained. Thus, a still image in the area where RBCs flowed in 150 frames was generated.

2.6.

Removal of Brightness Unevenness

In the images captured during this study, the peripheral parts were darker than the central parts due to the unevenness of the skin surface shape and the irradiated light intensity. In our system, this unevenness of brightness was conspicuous due to the wide field of view compared with the conventional system. This unevenness of brightness might compromise capillary visibility and make manual annotation for deep learning inaccurate. Therefore, we removed the brightness unevenness as follows:

2.7.

Contrast Enhancement

Drops and mistakes are likely to occur during annotation if the vascular area is unclear. Therefore, contrast enhancement was performed on still images after correcting the brightness to facilitate annotation. This process was performed through histogram equalization (using OpenCV’s equalizeHist function) and gamma correction ($\gamma =0.3$) for each of the R, G, and B channels.

2.8.

Semantic Segmentation of Capillary Regions

In this study, the U-Net, which is an existing segmentation algorithm, was used to extract the capillary regions. The U-Net has a U-shaped structure and can even accurately classify fine structures by maintaining the position information of each pixel during convolution and deconvolution. In this method, the input and ground-truth images containing the object’s annotated position are used as pairs to generate the trained model for image processing. Thereafter, the trained model is used on images different from those used for training, and the target objects are automatically identified and detected. This method was executed on Python (version 3.6.7). NumPy (version 1.16.2), Keras (version 2.1.6), and TensorFlow (version 1.10.0) were used as libraries to run U-Net.

2.9.

Evaluation of Differences Between Participants and Sites and Changes due to Skin Barrier Destruction

To analyze the differences in capillary variables among individuals and sites, we evaluated the variables at seven locations on the inner forearm. Further, to evaluate the external stimulation before and after skin barrier destruction through tape-stripping of each participant, we used data of four of the five participants who underwent barrier destruction through tape-stripping until TEWL was tripled, excluding one who exhibited significant blurring in the video image. The differences were evaluated using a paired $t$-test.

3.1.

Wide-Field Imaging of Skin Capillaries

By evaluating the spatial resolution of the wide-field video capillaroscopy using the USAF test target, a gap of $3.5\mu \mathrm{m}$ could be observed. Although this resolution is not higher than that of some commercial capillaroscopy equipment, such as Dino-Lite (IDCP Medtech, Almere, Thenetherland), it is comparable to the capillary width, with inner and outer diameters that are 3 and $10\mu \mathrm{m}$,²⁰ respectively. These values are sufficient for capillary observation. Observations of the skin before and after tape-stripping via water and a wrap film indicated that the skin texture and the specular highlights were suppressed throughout the entire field of view (Fig. 3). In addition, after contrast enhancement, many capillaries were observed in addition to stains and sweat glands. This result shows that, using the proposed system, the capillary structures could be observed in a wide field of view of $7.4\mathrm{mm}\times 5.5\mathrm{mm}$. Owing to its wide field of view, it was possible to virtually observe the same field of view repeatedly before and after skin barrier destruction without using complicated operations, such as switching to a low-magnification configuration. The sample images captured with and without water and wrap film are shown in Fig. S2 in the Supplemental Material.

Fig. 3

Large area skin capillary imaging result. Image of whole field of view of $7.2\mathrm{mm}\times 5.3\mathrm{mm}$ (a) before and (b) after removal of brightness unevenness and contrast enhancement.

In the single-frame image, RBCs were partially present in the capillaries in some areas, and the capillaries were observed indistinctly. However, even at such a site, we continuously monitored the capillary structure in the still image generated from 150 frames (Fig. 4). If the density of the observed capillaries can be quantified through image analysis, the variables corresponding to the FCD can be extracted from a wider field of view than that used in existing technology.

Fig. 4

Visibility of capillaries in a still image generated using 150 frames of a video image. Examples of contrast-enhanced images of (a) a single-frame and (b) a still image prepared using 150 frames of a video image taken after barrier destruction with water and wrap film. Enlarged images of size $800\times 600\text{pixels}$ ($3.7\mathrm{mm}\times 2.8\mathrm{mm}$) at the same site are shown. Visibility of the capillaries shown with arrows in (b) is better than that in (a).

Although it was not the focus of this study, because RBCs were visible in some frames and not in others for the same capillaries, dynamic indicators such as blood flow velocity and supply rate to the capillaries could be determined.

3.2.

Estimation Results for the Capillary Region

An example of the detected capillary regions after semantic segmentation is shown with the input and ground-truth images in Fig. 5. Some capillary regions detected by U-Net appeared to correspond to those in the ground-truth at first glance while other areas did not. Therefore, for a quantitative analysis of the correspondence between detection results and ground-truth, we evaluated the degree of agreement at the pixel level and the capillary region level, the correlation between the number and area of capillaries obtained in both, and the Bland–Altman plot.

Fig. 5

Semantic segmentation of capillaries. (a) Input images generated using 150 frames and enhanced contrast of different participants and sites. (b) Ground-truth images prepared manually from each input image. (c) Detection results after segmentation. Three examples of 11 results are shown. Number of capillary regions, total area (pixel), and average area (pixel), respectively, are mentioned under each binarized image.

3.3.

Verification of Estimation Performance at the Pixel Level

The estimation performance at the pixel level was calculated using indices that are generally used for segmentation performance evaluation. Here, we used images of $1300\times 1000\text{pixels}$ (equivalent to $2.4\mathrm{mm}\times 1.9\mathrm{mm}$) that were extracted from the full field of view. The accuracy, precision, recall, specificity, Dice, and IoU values were calculated for each image, along with the average values between images. Although accuracy and specificity were high (96.9% and 98.0%, respectively), precision, recall, Dice, and IoU were low (30.2, 47.1, 35.7, and 24.5, respectively). In many capillaroscopic images, the contours of skin capillaries were unclear. Therefore, both the U-Net and manual annotation generated errors in the width of the extracted blood vessels. Unlike the case wherein a car is detected through an in-vehicle camera, the capillary was approximately a few pixels thick in the image used in this study. Therefore, although the error was only a few pixels, it could contribute significantly to TP, resulting in low precision, recall, Dice, and IoU values. However, even after obtaining such errors, if the same capillaries as those in the manual analysis are extracted, it should be possible to estimate the same number and area of capillaries as those estimated through the manual analysis. Thus, the proposed method may be useful for characterizing capillaries, such as FCDs. This attribute is verified in the next section.

3.4.

Verification of Estimation Performance at the Capillary Region Level

As noted previously, when the overlap of regions between ground-truth and U-Net estimation was not perfect, the traditional measures (e.g., Dice and IoU) at a pixel level were significantly reduced due to the small widths of the capillary structures in our images. In such cases, the same capillaries could have been detected.

To verify whether we could extract capillary regions overlapping the ground-truth even partially, we calculated TP, FP, and FN for each capillary and calculated precision, recall, and Dice (Figs. S4 and S5 in the Supplemental Material). As a result, precision, recall, and Dice were 44.2%, 94.6%, and 58.0%, respectively, and all values improved compared with the pixel-level evaluation (30.2, 47.1, and 35.7). In particular, the high value of recall indicated that most manually annotated capillary regions could be detected. On the other hand, precision and Dice showed only slight improvements from the pixel-level values, indicating that there were many areas of FP. A further improvement in performance is expected by expanding the number of input images and optimizing parameters to improve the trained model and reduce FP.

3.5.

Verification of Capillary Number and Area Estimation Performance

Using the same images as those in the previous section, the number and area (total and average) of capillary regions estimated by U-Net were compared with the manual analysis results. The number and area of extracted capillary regions correlated well with those of manually annotated regions [Fig. 6(a)]. We obtained the regression equations for the number, total area, and average area as $y=2.014x$ ($R=0.96$), $y=1.311x$ ($R=0.99$), and $y=0.604x$ ($R=0.99$), respectively ($y$ and $x$ are values of regions detected using U-Net and those annotated manually, respectively). In Fig. 6(a), the discrepancy between the regression line and $y=x$ was larger for the total area than that for the number. We assumed that this may be attributed to the difficulty in determining whether it was a single or different capillary when the linear capillaries were partially blurred in the image shown in Fig. 4(b). Even in such cases, the area estimation accuracy was considered to be higher because the same capillaries could be extracted using both methods. We assumed that the regression equation of the total area estimation did not agree with $y=x$ owing to estimation errors of width because of the blurred capillary outlines.

Fig. 6

Comparison of capillary area detection results obtained through U-Net and ground-truth obtained through manual annotation: (a) number, total area, and average area of regions detected using U-Net plotted to ground-truth. Red solid, red dashed, and gray solid lines represent the regression line, 95% confidence interval, and $y=x$ line, respectively. (b) Bland–Altman plot of each parameter based on ground-truth and the values detected by U-Net. Red and blue dashed lines represent the mean difference between the ground-truth and detection result, and $\pm 1.96$ standard deviations of differences that indicate the LoA.

The Bland–Altman plot³⁹ calculates the agreement interval, which was within 95% of the differences of the AI, compared to the ground-truth, fall [Fig. 6(b)]. In the plot for the number of capillaries in this study, the mean difference between the ground-truth and detection results was 120, and all points were within the 95% limit of agreement (LoA) ($-168$ to $-72$). The total area was also distributed with the mean difference of $-\mathrm{10,790}$ and with all points within the 95% LoA ($-\mathrm{25,519}$ to 3939). For the mean area, the mean difference was 101, and all but one of the data points were within the LoA ($-39$ to 241). These results indicate that the estimations of the number and area were similar to manual annotations performed by a human.

Subsequently, we verified whether the capillary regions could be similarly extracted with a wide field of view of $3900\times 2900\text{pixels}$ (equivalent to $7.2\mathrm{mm}\times 5.3\mathrm{mm}$). Consequently, capillary regions were extracted from the entire image (Fig. 7). Because the regression line in Fig. 6 did not agree with $y=x$, instead of taking the detection results in U-Net as the number and density of capillaries as is, we substituted the detection results into $y$ in the above-mentioned regression equations to obtain values corresponding to $x$.

Fig. 7

Semantic segmentation result of the large-area image. Segmentation result for the image shown in Fig. 3(a) with a whole field of view ($7.2\mathrm{mm}\times 5.3\mathrm{mm}$, $3900\times 2900\text{pixels}$).

As a result, in the case of Fig. 7, we estimated the number, total area, and average area as 1234, 428, 279, and 379 pixels, respectively, in the $7.2\mathrm{mm}\times 5.3\mathrm{mm}$ ($4000\times 3000\text{pixels}$) region. This result shows that more than 1000 capillaries can be quantified from a wide field of view using the proposed method. During microscopic observation of capillaries under visible light, they appear blurry even if the depth increases slightly due to light scattering on the skin. Due to the limited depth of field of the imaging system as well as the aforementioned factor, capillaries distributed at a uniform depth near the very surface layer were observed.

3.6.

Evaluation of Differences Between Locations and Participants

The variables of the capillaries at seven locations in the inner forearm of each participant under normal conditions were compared (Fig. 8). For each participant, $\sim 1000$ capillaries were estimated. There was variation in each location, even for the same participant. It is known that the laser Doppler flowmetry values that measure the blood flow, including the vessels (arterioles, venules, etc.) deeper than the capillary, vary considerably for each measurement point.⁴⁰ It is also known that laser Doppler imaging in a 2.5-cm square ($6.25{\mathrm{cm}}^2$) region exhibits uneven blood flow.⁴¹ The capillaries targeted in this study may also exhibit heterogeneity on a similar scale, and such a difference may reflect the local condition of the tissue at each location (Fig. S1 in the Supplemental Material).

Fig. 8

Number and area of capillaries predicted before barrier destruction. (a) Estimated number, (b) total area, and (c) averaged area for four participants who damaged the stratum corneum by tape-stripping until TEWL was three times the initial value. Average values of seven places are shown, and the error bar is the standard deviation. The $p$ values less than 0.1 are shown.

Although the proposed method can image a wide field of view of $7.4\mathrm{mm}\times 5.5\mathrm{mm}$ wide, considering the difference between locations, it is desirable to take multiple locations to characterize the individual participants. In participants 2 and 3, the total capillary and average areas were small compared with participant 1. This difference may be used to characterize the skin condition and microcirculatory function of each participant.

3.7.

Changes in Observed Capillaries due to Barrier Destruction

For each participant, the number and area (total and average) of observed capillaries before and immediately after barrier destruction by means of tape-stripping at the exact location were calculated (Fig. 9). As a result, the number of observed capillaries decreased in participants 1, 3, and 4, whereas in participant 2, it increased. The capillaries’ visibility depends on their RBC content. Thus, this result suggests that the effect of barrier destruction on capillary blood flow varies among individuals. Such differences may help characterize the responsiveness of an individual’s skin microcirculation.

Fig. 9

Change in predicted capillaries after skin barrier destruction. (a) Estimated number, (b) total area, and (c) average area before and after skin barrier destruction in four participants, similar to Fig. 8. Average values of seven places are shown, and error bar indicates the standard deviation. $p$ values less than 0.1 are shown.

3.8.

Advantages and Future Challenges of the Proposed Method

We proposed a method for observing and quantifying many capillaries simultaneously with a wide field of view. This method may provide more reliable capillary indices than conventional capillaroscopy. Other modalities used for skin capillary observation include photoacoustic imaging, OCT-A, reflectance confocal microscopy, and two-photon microscopy. These systems require specialized and expensive components, such as complicated and precisely controlled optics, lasers, and ultrasonic transducers. The proposed system comprises simpler and cheaper elements and has a greater potential for clinical applications. In addition, because the shape and distribution of individual capillaries can be directly visualized, it can provide information that cannot be obtained through laser Doppler flowmetry, which computes average blood flow values over a large number of vessels.

This study proposed a technique that can lead to the development of a future diagnostic technique with such advantages, and there is considerable room for future verification. One such consideration is the validation of the capillaries observed and quantified using the proposed method based on a comparison with existing methods. For example, Kelly et al. used a capillaroscopy with a field of view width of $\sim 1.5\mathrm{mm}$ and reported that the capillary density was 42 to 50 capillaries per ${\mathrm{mm}}^2$ in the normal skin of the inner forearm.¹⁶ This corresponds to $\sim 1600$ to 1900 capillaries in the field of view of our estimation. Gronenschild et al. also reported 30 to 100 capillaries per ${\mathrm{mm}}^2$ within observations of fingers with a field of view of less than 3 mm, which corresponds to $\sim 1100$ to 3800 capillaries in our system.¹² Although our results (Fig. 6) showed variability among participants, the estimated numbers were probably smaller than the ones reported owing to differences in the body part or participant characteristics, along with the dependency of the detected capillaries on the device specification. Therefore, the same sites as those observed by existing devices must be observed, and the capillaries that can be quantified must be identified. Although the proposed device has a resolution ($<3.5\mu \mathrm{m}$) that allows for discriminating the outer diameter of capillaries ($>10\mu \mathrm{m}$), it is difficult to discriminate it if two or more are adjacent to each other or if they overlap. As this discrimination can potentially cause errors in capillary count, observing with another device may be useful for verifying and improving the results. In addition to a higher magnification capillaroscopy, photoacoustic imaging, OCT-A, reflectance confocal microscopy, and two-photon microscopy are useful for obtaining the ground-truth of the 3D capillary structure that cannot be observed using the proposed system.

In addition, in this study, skin was observed after tape-stripping to confirm the response to physical stimuli, but contrary to expectations, a decrease in the number and area of capillaries was observed. Based on reports in the literature³⁷ wherein it is stated that tape stripping is followed by a mild inflammatory response and an immediate increase in blood flow in the microvasculature, an increase in the number and area of capillaries after stripping was expected. However, in the results of this study, the opposite was observed in some cases. Because there is limited information on the response of blood flow after tape stripping, it is desirable to understand the phenomenon by comparing it with new findings in the future and using the proposed method in combination with other blood flow measurement methods.

Furthermore, the number and characteristics of the participants included in this study were limited. Their skin types⁴² were not assessed precisely but were roughly in the range of type 2 to 4. Expanding the data to include different skin types, the presence of skin diseases, and various body parts and measurement sites (Fig. S1) is expected to improve accuracy and robustness.