2.1.

Overview of the Entire Imaging System

Figure 1 depicts the entire imaging system, which is composed of four distinct modules: (1) an image capture module, (2) an image processing module, (3) an image display module, and (4) a light source for simultaneous visible spectrum and NIR fluorescence imaging.

Fig. 1

Dual-spectrum imaging system with a goggle display. (a) The imaging system is composed of two complementary metal oxide semiconductor (CMOS) imaging sensors, long-pass and short-pass optical filters, an FPGA data acquisition and image processing board, and an HDMI goggle system; (b) a custom-built NIR and visible white light-emitting diode (LED) illumination module; and (c) an NIR-visible spectrum LED tracking pod.

The image-capture module is composed of two CMOS imaging sensors (Aptina MT9V034) housed in a custom image capture PCB. The custom image capture PCB board ensures that the imaging sensors are placed parallel right next to each other, and the distance between the two sensors is minimal. Placing the sensors close to each other ensures that the disparity between the two imaging sensors is small, and hence most of the imagers’ field of view can be used to capture visual information. One of the CMOS imaging sensors has a color Bayer pattern, while the second CMOS imaging sensor is monochromatic. The Bayer pattern imager is used to produce a color image, and the monochromatic sensor is used to capture NIR information. A high quantum efficiency of 35% at 800 nm for the CMOS imaging sensor allows for efficient imaging of fluorescence signals in the NIR spectrum. Spectral filters are placed in front of the imaging sensors: a short-pass filter with a cutoff wavelength of 690 nm (Semrock FF01-694) is integrated with the Bayer pattern imager, while a long-pass filter with a cutoff wavelength of 785 nm (Semrock BLP01-785R) is integrated with the monochromatic imager. Two polymer-based objective lenses (Sunex DSL208 15.8 mm F2.0) are used to minimize the weight and size of the complete imaging system. The weight of the entire system is $\sim 50\mathrm{g}$, and the size of the custom image capture PCB is 1.5 in. by 1 in. (see Fig. 1).

The two imaging sensors operate in a master-slave configuration. This mode of operation allows for data to be sent from the monochromatic/NIR imaging sensor to the color imaging sensor via a two-line low-voltage differential signaling (LVDS) serial bus. The color imaging sensor then combines the data from both sensors and produces a single 16-bit word that contains information from both sensors (the lower 8 bits of data correspond to the color imager and the upper 8 bits of data correspond to the monochromatic/NIR imager). The 16 bits of data are serialized and sent via a two-line LVDS bus to the image processing module, hence minimizing the number of wires utilized to transfer data from both sensors.

The image processing module is composed of a custom image processing PCB, an FPGA, and an optional PC. The custom image processing PCB is designed (1) to connect with the two imaging sensors and provide easy scalability for additional imaging sensors, (2) to mate with an FPGA board, and (3) to connect to the goggle display via an HDMI chip housed on the same custom image processing PCB board. The connection between the imager board and the image processing board is accomplished via a custom eSATAp connector. This connector allows for eight independent signals to be transferred between two entities and three shielding wires. We utilize the eight signals as follows: two lines for LVDS image data from both cameras, two lines for the external exposure control signals of the two imagers, two lines for sending and receiving control signals to and from the imagers (such as exposure time, region of interest, gain of the signal, and others) via a serial protocol, and two power lines (ground and 3.3 V). The shielding wires are grounded and are used to shield the LVDS data signals from external noise sources. The serial data from the two imaging sensors are deserialized in the Field-programmable gate array (FPGA) (Opal Kelly 3050, Xilinx Spartan 3). The FPGA board can either transfer the deserialized data to a PC via a USB 2.0 data link for image processing or execute the necessary image processing to provide a single combined NIR-visible spectrum image on the FPGA.

The image coregistration is accomplished via a software unit and a custom light-emitting diode (LED) tracking pod. The LED tracking pod contains two bright and thin LEDs (Fig. 2). One of the LEDs is centered at 850 nm (Everlight HIR19-21C), and the second LED covers the entire visible spectrum (Rohm Semi., SMLP12WBC7W). The LEDs are placed next to each other. Due to their minimal form factors, the LEDs are virtually collocated when viewed at a 50-cm distance. This allows for both NIR and color cameras to observe a single point in both the NIR and visible spectrums, respectively. Since extra bright LEDs are utilized in the tracking pod, the location of the LEDs in both images corresponds to pixels with saturated values. Note that the assumption here is that the rest of the image in both the visible and NIR spectrums is not saturated. The image processing algorithm, which can run either on the FPGA or the PC (described in the next section), computes the disparity between the NIR and visible spectrum images based on the tracking LEDs pods’ locations. An average disparity for the entire NIR image is computed when multiple tracking pods are utilized and applied on the entire image. A single combined image is computed by highlighting the pixel locations in the color image, where NIR pixels values are above a user-defined threshold.

Fig. 2

Block diagram of the NIR-visible imaging system. NIR-visible tracking pods are used to compute an average disparity between the two images. This disparity information is applied to the entire image to create a combined image in which the location of fluorescent data is highlighted on the color image data.

Since the FPGA is a compact device and can be easily carried in a pocket, the surgeon is free to move around the operating room, and the entire device does not impede the surgical flow. If the algorithm is executed on the PC, more optimization and advanced algorithms can be developed with significant time saving due to the easy implementation in the C++ programming language. Converting the same algorithm from C++ language to a hardware description language, such as Verilog or VHDL, is a very time-consuming task. Furthermore, images and videos captured from the imaging sensor can be recorded on the PC, which is not possible on the FPGA implementation at this point. The shortcoming of using a PC for image processing is that the surgeon would be tethered to a computer, which would restrict his or her movement in the operating room.

The image display module is composed of an HDMI decoder chip housed on the custom image processing PCB board and a goggle display device. The HDMI decoder chip receives display data from the FPGA and provides high-definition video data at 1080p to the display goggle unit. The state timing machine is optimized to display information on any device capable of handling HDMI inputs, such as LCD monitors, TVs, and projectors. We use a lightweight and ergonomic goggle display unit to display the final image to the health professional.

The illumination unit²¹ is composed for both NIR excitation and as a visible spectrum illumination source. The NIR excitation light source is constructed from 16 high-power 760-nm NIR LEDs (Roithner LaserTechnik H2A1-H760) covered with a spectral bandpass filter centered at $769\pm 20\mathrm{nm}$ (Edmund Optics 84–121).The spot size of the NIR light source is $\sim 20\mathrm{cm}$ in diameter with an average optical power of $5\mathrm{mW}/{\mathrm{cm}}^2$. The NIR light source is placed 70 cm from the illuminated subject. The white light source is composed of a visible spectrum LED panel with an NIR blocking filter. The NIR blocking filter eliminates background signals when capturing the fluorescence image. Two separate NIR-visible light sources are used to provide uniform illumination to the animal in the experiment. The NIR sources are placed on either side of the imaging system as depicted in Fig. 2.

2.2.

Disparity Algorithm with Light-Emitting Diode Tracking Pods

Accurate computation of the disparity information between the NIR and color images is of paramount importance when generating a combined NIR-color image. Misregistration between these two images can create an incorrect assessment of the tumor location and can lead to both positive margins and damage to healthy tissue. In order to achieve high-precision overlay images, the locations of the LED tracking pods in both images are used to compute the disparity information for the entire image. The complete disparity algorithm is depicted in Fig. 3.

Fig. 3

Flowchart of the disparity image processing algorithm using NIR-visible tracking pods. The algorithm is implemented on both FPGA and PC for real-time display of combined NIR-color images to the surgeon.

The first step in this algorithm is calibrating both the NIR and color cameras. The main purpose of this step is to refer the NIR and color pixel’s coordinates from the image planes (M) to a camera reference coordinate system (m) while taking into account the optical, geometrical, and digital characteristics of the camera. This is achieved by multiplying the pixel coordinates $(x,y,1)$ with a $3\times 3$ matrix $A$, as shown in Eq. (1).

Matrix $A$ represents the intrinsic parameters of the camera, where ($u0$, $v0$) are the pixel coordinates of the principal point, $\alpha$ and $\beta$ are the scale factors for the coordinate system, and $\gamma$ is the skew of the two orthogonal image axes. In order to compute the intrinsic camera parameters, images are obtained in both the NIR and visible spectrums of a black-and-white checkerboard pattern obtained from different viewing perspectives.

The next step in the image processing algorithm is to identify the location of the tracking pods in both the NIR and visible spectrum images. Since extra bright LEDs are used for the tracking pods, the pixels corresponding to the tracking pod LED will be saturated or will be close to saturation. Therefore, both NIR and color images use a threshold value corresponding to 95% of the dynamic range of the pixel. For example, since the maximum digital value of the pixel is 255, the threshold value is set to 242. It is important that the rest of the image is not saturated to prevent erroneous flagging of the tracking pod locations.

After the NIR and visible spectrum images are at the threshold, the locations of all pixels with values above the threshold are stored in separate lists for both spectrums. The locations of the pixels in the visible spectrum list are compared to the locations of pixels in the NIR list by performing a local search. The local search is limited to a pixel neighborhood of $\pm 15\text{pixels}$. The pixel location is marked as a valid point if a saturated pixel is found in both images. Otherwise, the pixel location is removed from the list. Finally, after all the points are searched, the disparity information between the NIR and color images is computed based on the weighted disparity average shown in Eq. (2). Therefore,

Since the NIR and visible spectrum images are placed next to each other on the custom image capture PCB, translation predominately accounts for the disparity between both the images generated from the sensors. Also, computing an average translation disparity between the two images is easily implemented on both FPGA and PC for real-time (27 fps) imaging. The disparity computation can be extended to include both an estimation of translation and rotation for a better and more accurate overlay between both images at the cost of higher computational complexity.

The disparity between the NIR and visible spectrum images is a function of depth. Since the two cameras view the same scene at different spectrums, stereo vision algorithms that estimate depth, and, therefore, disparity cannot be used for this application. The LED tracking pods allow the same point in space to be viewed in both color and NIR images, and hence estimate the disparity between the two images. The disparity information computed from the tracking pods has the highest accuracy at the depth where the tracking pods are located. Since the tracking pods are placed next to the subject that is imaged, part of the subject will be closer and part of it will be further away from the tracking pods, depending on the subject’s three-dimensional structure. Hence, the disparity will be different across the imaging plane and will introduce error in the overlay image when a single (global) disparity metric is employed.

The disparity error estimation is illustrated in Fig. 4. In this figure, the square depicts the location of the tracking pods that are accurately determined via the image processing algorithm described in the previous section. A global disparity estimate is used for all pixels in the image based on the location of these LED pods. The circle depicts part of the scene that is further from the tracking pods and the triangle depicts part of the scene that is closer to the imaging camera. These three points in space at depths $D-\mathrm{\Delta }D$, $D$, and $D+\mathrm{\Delta }D$ will be projected to three different points on the imaging plane with different disparities [$2(\mathrm{l}+x_1)$, 2l, and $2(\mathrm{l}-x_2)$]. By applying the triangular similarity theorem, we derive a closed form expression for the error estimate and present it in Eqs. (3) to (4). In Eq. (3), $D$ is the distance between the sensor and the LED tracking pods, $F$ is the focal length of the lens, $d$ is the distance between the NIR and visible sensors, and l is the distance between the targeted pixel and sensor center. Using Eq. (3), the relationship between the disparity error estimate and the target depth is shown in Eq. (4), where $\mathrm{\Delta }D$ is the depth difference from the initial position $D$, and $\mathrm{\Delta }{\mathrm{l}}_{\mathrm{NIR}}$, $\mathrm{\Delta }{\mathrm{l}}_{\text{visible}}$ are the corresponding distance changes on the NIR and visible sensor pixel arrays, respectively. Under a normal working distance, $\mathrm{\Delta }D$, $d$, and $F$ are much smaller than $D$. Hence, error estimation $e$ can be simplified to form a first-order linear relation with depth difference $\mathrm{\Delta }D$.

Fig. 4

Disparity error estimate illustration.

3.1.

Disparity Error Estimation

The disparity error between the NIR and color images is evaluated using a single tracking pod. The tracking pod is initially placed at either 45 or 65 cm to emulate two different working distances between the surgeon and the subject. The disparity at these working distances is computed and applied to all pixels in the image.

To emulate the fact that portions of the scene are closer or farther from the working distance, the tracking pod is moved $\pm 6\mathrm{cm}$ from the working distance. Since the LEDs on the tracking pod are minimal in size, a single point in space emits both white light and an NIR spectrum. The corresponding points of the LED tracking pods are determined from both images at different depths, and the disparity is computed. A disparity error measurement is computed by subtracting the global disparity at the working distance (45 or 65 cm) from the disparity of the tracking pod at various positions near the working distance ($\pm 6\mathrm{cm}$). The results are presented in Fig. 5.

Fig. 5

Disparity error measurement.

The measurements in Fig. 5 indicate that for a working distance of 65 cm, a maximum disparity error of 2 mm between the two images is registered at a distance of 6 cm away from the tracking pods. In other words, if the surgeon initially views the tissue at 65 cm, both NIR and visible images are perfectly co-registered. If the surgeon then moves and leans toward the subject by 6 cm (i.e., distance between surgeon and tissue is 59 cm), the NIR and visible spectrum images will have a 2-mm co-registration error. The co-registration error increases to 3 mm for working distance of 45 cm at a distance of 6 cm away from the tracking pods.

3.2.

Sensitivity

The sensitivity of the proposed imaging system was tested by recording the NIR fluorescent signal responses for different ICG concentrations and different LS301^18,22,23 concentrations dissolved in 100% dimethyl sulfoxide (DMSO). ICG has been widely used for NIR fluorescence since its FDA approval in the late 1960s. LS301 is a tumor-targeted contrast agent developed at Washington University in St. Louis and is currently under preclinical development. The vials with different ICG and LS301 concentrations are placed 50 cm from the imaging sensors. A control vial with DMSO is also imaged and denoted as a background signal. Both visible and NIR imaging sensors are operating at 27 fps (i.e., a maximum exposure time of 30 ms). The exposure time for the color imaging sensor is typically around 1 ms due to the bright surgical LED light source. The NIR image sensor has a maximum exposure time of 30 ms to acquire the best signal-to-noise ratio fluorescence signal. The diluted samples are illuminated with a 780-nm excitation light source, and two different optical powers (5 and $10\mathrm{mW}/{\mathrm{cm}}^2$) are used for the test evaluation. The mean and standard deviation of the signal-to-background ratio (SBR) of the imaging system is calculated from a $30\times 30\text{pixel}$ region. The experiments are repeated with three different samples with the same ICG-DMSO and LS301-DMSO concentrations.

The sensitivity results from the imaging system are presented in Fig. 6. Figure 6(a) shows the signal intensity in log scale for different ICG and LS301 concentrations ranging from 500 pM to $50\mu \mathrm{M}$, respectively. Figure 6(b) presents the signal intensity in linear scale for the different ICG and LS301 concentrations, respectively. A higher illumination power increases the signal response from the same fluorescence sample. When the ICG and LS301 concentrations are higher than a certain threshold (500 nM for ICG and $1\mu \mathrm{M}$ for LS301 under $10\mathrm{mW}/{\mathrm{cm}}^2$ illumination; $1\mu \mathrm{M}$ for ICG and $10\mu \mathrm{M}$ for LS301 under $5\mathrm{mW}/{\mathrm{cm}}^2$ illumination), the fluorescent signal intensity exceeds the dynamic range of the imaging sensor. Hence, the imager output is saturated. Figure 7(a) presents the SBR as a function of the concentration of ICG and LS301 in log scale, and Fig. 7(b) presents the same information in terms of linear scale, where the vial with 100% DMSO is used as the background control negative sample. Defining the detectability of a system to be $\mathrm{SBR}=2$,¹³ the imaging system can detect 25 nM of ICG and 30 nM of LS301 under $10\mathrm{mW}/{\mathrm{cm}}^2$. Using an excitation illumination of $5\mathrm{mW}/{\mathrm{cm}}^2$, the minimum ICG detectability is 40 nM and 40 nM for LS301. On the other hand, as seen in Figs. 7(a) and 7(b), the higher excitation power leads to a lower SBR at higher concentrations, which occurs because the fluorescent signal is saturated while the background signal is higher. Therefore, selecting a proper optical power for the illumination module is always a tradeoff between keeping a high SBR and a high signal response.

Fig. 6

Sensitivity test using ICG-DMSO and LS301-DMSO: (a) logarithmic scale and (b) linear scale.

Fig. 7

Signal-to-background ratio (SBR); green line shows the $\mathrm{SBR}=2$ threshold: (a) logarithmic scale and (b) linear scale.

Figures 6 and 7 also indicate that ICG achieves a higher signal response and SBR compared to LS301. The reason for this is that LS301 is a cypate-based contrast agent, which has a lower quantum efficiency than ICG. On the other hand, as a tumor-targeted contrast agent, LS301 will clear out from most animal organs 24 h after injection and only remains in the cancerous tissue. This effect leads to a much higher fluorescent contrast for imaging the cancerous tissue from the healthy tissue. Hence, LS301 is used in our in vivo study in mice to further validate the performance of the imaging system.

3.3.

In Vivo Study in Mice

The NIR/visible spectrum imaging system was validated through in vivo studies using a subcutaneous breast cancer mouse model. Four-to-six-week-old BalbC mice were given subcutaneous flank injections with 100,000 4T1Luc murine breast cancer cells. When the tumors were at least 10 mm in size, the mice were injected with $10\mu \mathrm{l}$ of $60\mu \mathrm{M}$ tumor-targeted NIR fluorescence contrast agent LS301, via the tail vein. Images were taken 24 h postinjection of LS301. During in vivo study, the image-capture module was set up at a 50-cm working distance, and the illumination module was placed at a 50-cm distance. All images were captured at 27 fps. During the imaging experiments, the mouse was kept anesthetized using a cocktail of ketamine/xylazine or through intubation of isofluorane. Following the imaging experiment, the mouse was sacrificed through cervical dislocation, and its organs were harvested, imaged, and preserved for histologic evaluation.

In Fig. 8, the mouse object and the LED tracking pod are placed in the same scene. Figure 8(a) presents the NIR image mapped using a false color map, demonstrating that the system can detect in vivo fluorescent signals from the tumor areas. A threshold value of 38 has been assigned to eliminate the background signal. Figure 8(b) presents the color image captured by the visible sensor. Without the disparity cancellation algorithm, the NIR and color images have a large disparity error, as shown in Fig. 8(c). Figure 8(d) depicts the final NIR-color image after the disparity algorithm is applied using the tracking LED pods. The tumor areas are clearly highlighted on the color image and accurately demarked after close visual evaluation of the final image.

Fig. 8

Mouse study test result: (a) near infrared (NIR) channel image, (b) visible channel image, (c) predisparity correction image, and (d) corrected image using a threshold detection algorithm.

Figure 9 shows the results of the same mouse object when the skin is deflected to reveal the tumors. The tracking LED pod allows for an accurate estimate of the disparity between the NIR and color images. The tumors are accurately highlighted in the final color image. Figures 8 and 9 contain the signals from the tracking LED pods, and they are shown as a red dot for the NIR image and a white dot for the visible image. The tracking signals were simultaneously recorded with the image of the mouse in order to determine the correct co-registration.

Fig. 9

Mouse study test result (open skin): (a) NIR channel image, (b) visible channel image, (c) predisparity correction image, and (d) corrected image using a threshold detection algorithm.