2.1.

Optical Path Design

A self-developed light splitter was used for simultaneously acquiring color and NIR images. This module comprised a dichroic sheet (DMLP650, Thorlabs Ltd., Newton, New Jersey) and a set of relay lens. The dichroic sheet was used to divide light into two beams. One light beam with the wavelength between 810 and 870 nm was transmitted to the NIR camera. The other light beam with the wavelength between 400 and 650 nm was reflected to the color camera. To ensure the same FoV of color and NIR cameras, an adjustable structure was designed for mounting the dichroic sheet and reducing the mechanical error. Furthermore, the relay lens consisted of three sets of lens and filters, which were assembled for focusing the collected light on the camera sensor. Because the light transmission rates of the selected dichroic sheet and aspheric lens reached more than 95% in each favorable wavelength band (Fig. 2), the aperture size design ensured sufficient depth-of-view, reduced stray light, and increased brightness contrast of color imaging, and the whole light splitter was assembled in a very small cylindrical chamber (internal diameter: 2 cm and internal length: 4 cm), the loss of the optical signal caused by light-splitting was minimized.

Fig. 2

The light transmission rate of utilized lens in the light-splitting module. The green and blue lines separately showed the light transmission of white-light and NIR-light enhancement aspheric lens. Both wavelengths (wavelength: 300 to 850 nm and 810 to 870 nm) reached above 99%. The black and red lines, respectively, represent the transmission efficiency and reflection efficiency of the dichroic sheet. Both reached above 95%.

Besides that, the light-splitting module was coupled with a 0-deg direction of view, 70-deg visual field, 10-mm diameter, and 31-cm length thoracoscope. We compared several commercially available rigid thoracoscopes provided by different manufacturers (Karl Storz, Olympus, and Stryker), and they were all clinically approved. Finally, the one with the best performance in the NIR spectrum was selected for experiments.

2.2.

Imaging Acquisition

For the thoracoscopic imaging system design, it is difficult to choose a high-sensitivity, light-weight, and small-size camera. To balance the weight/size and sensitivity, we cooperated with Microview Inc. and developed a light-weight SCMOS NIR camera (weight: 250 g, HK-A5100-GM17, Microview, Beijing, China), which meets the requirement of sensitivity. Meanwhile, because NIR and color cameras were designed to have exactly the same cell size and pixel number (Table 1) and combined with the dichroic beam splitter, the dual-channel imaging inherently had the same FoV. This hardware feature ensured fast and accurate NIR and color imaging registration. Furthermore, the SCMOS NIR camera with high NIR-light imaging sensitivity facilitated weak fluorescent signal collection within 50-ms exposure time at each frame. Besides all these features, both cameras were very compact and light in weight (Table 1). To reduce the signal noise during acquisition, two filters (wavelength: 400–650 nm and 810–870 nm) were fixed in front of two cameras, respectively [Fig. 1(a)]. The acquired raw data were transmitted to the computer in the control station through the gigabit-per-second wires, so that the requirement of high-speed data transfer was guaranteed. As a result, the system could achieve real-time high-SBR dual-channel imaging with the frame rate above 20 fps.

Table 1

Key parameters of the newly developed FFWT system.

2.3.

Illumination Design

To simultaneously acquire color and NIR images, the NIR excitation light and white light were coupled into the thoracoscope. The NIR excitation light source (MW-SGX-785, Changchun Lei Shi Optoelectronic Technology Co. Ltd, Changchun, Jilin, China) used a 785-nm laser with continuously adjustable power from 0 to 5 W. The white-light illumination was provided by a 300-W LED light source (Kenon Nova 300, Karl Storz, Tuttlingen, Germany) in the visible light window. A customized 3-m long multimode bifurcated fiber bundle was used to couple the NIR and white-light sources. It was 4.8 mm in diameter, and a NIR excitation-light fiber ($400\mu \mathrm{m}$ in diameter) was in the center of the bundle surrounded by multiple white-light fibers (Fig. 1). This multimode bifurcated fiber bundle was connected to the thoracoscope using standard endoscopic light source coupling.

2.4.

Software Development

In order to realize the real-time intraoperative image registration, storage, and display, a customized software was developed. The image processing pipeline was shown in Fig. 3. The parameters controlled by our software includes exposure time, light intensity gain, and image binning for acquiring high-contrast dynamic images. All data collected from color and NIR cameras were transmitted into the computer through standard gigabit-per-second wires. Benefited from the system structure design, color and NIR images have the same FoV and pixel number. As a result, the dual-channel imaging registration was easy, quick, and accurate without using a complicated coregistration algorithm. Furthermore, the entire interface of the software can be upgraded and extended, so other algorithms can be also added into the software.

Fig. 3

The software pipeline of the FFWT system. The software provided the function of controlling expose time, gain, and mirroring of cameras. The color and fluorescence images were stored and fused, and then the color, fluorescence and fusing images were displayed for intraoperative identifying important anatomical or pathological features in real time.

2.5.

System Characteristics

2.6.

In Vivo Preclinical Experiments in Porcine Models with FFWT

The feasibility of FFWT was validated through in vivo animal experiments. The identification of the thoracic duct ($n=3$) and intersegmental plane ($n=3$) was performed in six 3- to 5-month-old miniature porcine models (30 to 35 kg). All experiments were approved by the Johnson & Johnson Beijing Medical Science Centers. Before surgery, the pigs were anesthetized with 2% isofluorane and monitored by a physiological monitor. Pulmonary segmentectomy experiments were performed using 12-mg ICG with a concentration of $5\mathrm{mg}/\mathrm{ml}$ through marginal ear vein injection. FFWT was used to image the pulmonary segment during the surgery. Real-time color, NIR, and fused images of the intersegmental plane were displayed on the screen and recorded as video data. After that, thoracic duct experiments were performed by injecting 1 mg ICG subcutaneously at a concentration of $5\mathrm{mg}/\mathrm{ml}$ into the right leg, and the thoracic duct imaging was performed around 13 min later.

2.7.

In Vivo Clinical Experiments in Lung Cancer Patients with FFWT

To evaluate the feasibility of the FFWT system in NIR image-guided segmentectomy, we recruited three patients previously scheduled for white-light thoracoscopic surgery. All patients signed the informed consent. This study was approved by the institutional ethics committee of Peking University People’s Hospital (2015PHB157-01) and registered at ClinicalTrials.gov (NCT02611245). The study’s inclusion criteria were as follows: (i) existing pulmonary nodules and prepared for thoracic surgery, (ii) no history of allergy to ICG, and (iii) aged between 18 and 75 years. The exclusion criteria were as follows: (i) preoperative liver dysfunction, (ii) known allergy to ICG, and (iii) with other uncontrollable complications. First, patients were intravenously injected with $0.4\mathrm{mg}/\mathrm{kg}$ ICG after ligating the segmental pulmonary arteries. Then, we performed NIR image-guided segmentectomy with FFWT (Fig. 6). The surgical procedure was recorded in video rate. For each patient, we calculated the SBR of selected frames in each video.

Fig. 6

Intraoperative imaging guided clinical segmentectomy using the FFWT system. The system provided surgeons a well-defined intersegmental plane with the SBR of $19.6\pm 0.8\mathrm{dB}$ (Video 1, MOV, 913 KB [URL: http://dx.doi.org/10.1117/1.JBO.22.6.066002.1]).

3.1.

System Parameters

The parameters of FFWT were shown in Table 1. NIR camera and color camera have high resolution for acquiring high-quality images and the same cell size for imaging identical FoV. Moreover, two cameras with light weight and small size ensured user friendly operation.

3.2.

Optical Resolution

We measured the optical resolution of FFWT at different distances (25, 50, and 75 mm), which were shown in Figs. 4(a)–4(d). The red outline part of Fig. 4(a) was magnified and placed in Fig. 4(b), which shows that the group number three, element six stripe did not lose the contrast with adjacent lines in vertical and horizontal directions. Therefore, the system’s vertical and horizontal resolution was determined to be $35\mu \mathrm{m}$ at a working distance of 25 mm. Furthermore, the spatial resolution of our system was $63\mu \mathrm{m}$ at the working distance of 50 mm [Fig. 4(c)] and $79\mu \mathrm{m}$ at the working distance of 75 mm [Fig. 4(d)].

Fig. 4

System characteristics evaluation. (a) A USAF 1951 test target color image collected by the FFWT system at a working distance of 25 mm. (b) The local enlarged image of (a). The three to six stripes did not lose the contrast with adjacent lines in vertical and horizontal directions (red outline). The corresponding spatial resolution of three to six stripes was $35\mu \mathrm{m}$. (c) and (d) The optical resolution of FFWT was separately 63 and $79\mu \mathrm{m}$ at the working distance of 50 and 75 mm. (e) The fused images of different ICG solutions (1, 0.1, and $0.01\mu \mathrm{M}$) and one PBS solution at the exposure time of 100 ms. (f) The SBR bar chart of different ICG solutions.

3.3.

System Sensitivity

Figures 4(e)–4(f) show the sensitivity measurements of FFWT. The fused images of four EP tubes at the exposure time of 100 ms were shown in Fig. 4(e). Figure 4(f) plots the quantitative calculation of SBR ($13.8\pm 0.3$, $8.3\pm 0.9$, and $5.3\pm 0.7\mathrm{dB}$) at the concentration of 1, 0.1, and $0.01\mu \mathrm{M}$. The minimal detectable ICG concentration of this system was $0.01\mu \mathrm{M}$ with the SBR of $5.3\pm 0.7\mathrm{dB}$.

3.4.

In Vivo Feasibility Verification in Porcine Models

Figure 5 shows the preclinical imaging of FFWT in six porcine models. To demonstrate the applications of the system for intraoperative lung cancer surgery guidance, the scenarios of thoracic duct visualization and intersegmental plane delineation were conducted. The representative color, NIR, and fused images of lung thoracic duct and pulmonary segments were shown in Fig. 5.

Fig. 5

Preclinical experiment results of the FFWT system. (a)–(f) Representative color, fluorescence and fused images of lung thoracic duct and intersegmental plane obtained by the system. (g) The average SBRs of the lung thoracic duct and intersegmental plane were $18.2\pm 0.5$ and $21.5\pm 0.3$, respectively. Scale bar: 2 mm.

Figures 5(a) and 5(b) show the color and NIR images of the thoracic duct. The damage of thoracic duct leads to a serious complication (chylothorax) and reoperation of chylothorax patients. However, it is hard to distinguish the thoracic duct from surrounding tissues due to the thoracic duct with similar shape and color in comparison to surrounding tissue.^29–31 With the NIR fluorescence imaging method, Fig. 5(c) shows the fused image that clearly indicated the location of the thoracic duct, which was difficult to distinguish from surrounding tissues in Fig. 5(a) only. This revealed that FFWT was feasible to visualize the thoracic duct and effectively avoid its unnecessary damage intraoperatively.

Meanwhile, pulmonary segmentectomy is a safe and minimally invasive technique in the lung cancer surgery. It has the potential to preserve lung function and reduce hospital costs, however, the identification of intersegmental plane requires the surgeons with a wealth of clinical experience and lacks a simple and effective technique to distinguish the targeted and nontargeted segments.²⁶ Figures 5(d)–5(f) are the color, NIR, and fused images of pulmonary segments collected by our dual-channel imaging system. Due to the lack of contrast between the ligation part and nonligation part, it is hard to identify pulmonary segments in the color image [Fig. 5(d)], whereas the boundary is easily seen in the fused image [Fig. 5(f)]. Furthermore, we use the SBR, which is defined as the ratio of the fluorescence signal in thoracic duct/ pulmonary segments to its surrounding tissues to evaluate NIR imaging effect and system performance. The quantitative measurements of the average SBR for lung thoracic duct and intersegmental plane in NIR images are $18.2\pm 0.5$, and $21.5\pm 0.3\mathrm{dB}$, respectively [Fig. 5(g)].

3.5.

In Vivo Near-Infrared Image-Guided Segmentectomy in Lung Cancer Patients

In lung cancer surgery, segmentectomy is an important surgical option, but surgeons lack an efficient way to define intersegmental plane with the conventional white-light thoracoscopy.⁸ To address this clinical challenge, we performed NIR imaging guided segmentectomy using the FFWT system. Three patients with early-stage lung cancers were recruited in this preliminary study. As shown in Fig. 6, the high SBR ($19.6\pm 0.8\mathrm{dB}$) of the fluorescent signal enabled easy and objective identification of the intersegmental plane, which revealed the potential of the clinical translation for using the system in MILCS.