1.

Introduction

Surgical resections are often the primary treatment option for both benign and malignant tumors. Curative surgery for many cancers requires a wide local excision to achieve negative margins.^1,2 Although a greater extent of tumor resection is critical for reducing tumor recurrence and improving prognosis, wide negative margins for brain surgery are not possible.^3–5 At the margins, the delicacy and unique functionality of the surrounding healthy tissues necessitate extreme precision during excision for neurosurgery, as the risk of damaging normal brain tissue is the impairment of patient’s cognitive function for their lifetime. In intraoperative neurosurgery, the ability to assess tumor margins with high accuracy would benefit patient’s survival, while preserving quality of life after surgery.

Visual appearance and palpation have been the standard methods used by surgeons to differentiate tumor from healthy tissues. These techniques are ineffective at the margins for many tumors, where high contrast and resolution are required intraoperatively. Bright tumor-specific indicators, such as targeted fluorophore, can visually enhance tumor margins, which is ideal for residual tumor cleanup and removal of disturbed tumor fragments. Compared to other noninvasive imaging technologies, such as magnetic resonance imaging, computed tomography, positron emission tomography, and ultrasound, advanced optical imaging is the most pragmatic means to provide high-resolution, video-rate, and noncontact intraoperative imaging for surgeons.

In the past 30 years, various optical imaging techniques have been investigated to guide surgery and to delineate tumor margins,^6–12 which can be categorized into three mechanisms: (1) structural contrast enhancement, (2) metabolic contrast enhancement, and (3) molecular contrast enhancement.

To verify tumor margins detected by fluorescence images, the in vivo imaging must be compared with pathology, which provides the ground truth of cancer regions. The different tissue appearances from the two imaging modalities makes this multimodal registration task challenging. Moreover, variations in tissue optical properties and tumor depth make correlating fluorescence images to histopathological slides more difficult. Elliott et al.²⁶ utilized IRDye800CW-labeled affibody specific to epidermal growth factor receptor to guide glioma resection. The tumor was correlated to fluorescence contrast using a multistep image registration. However, in their approach, fluorescence images were directly registered to histological slides. Structural landmarks in fluorescence images were selected as control points for registration, which are fewer and less obvious, compared to those in reflectance images. Lu et al.²⁷ applied principal component analysis to extract macro images from HSI and registered these extracted macro images to histological images, thus evaluating HSI as an intraoperative visual aid for surgical resection. However, both registration methods compared to a single-layer histological slide, which was selected subjectively and might not contain complete tumor information.

In this paper, a multistep registration framework is established to correlate fluorescence images to stacked histological images, which can be applied to evaluate fluorescent probes in delineating tumor cores and margins. Since variable experimental parameters and artifacts may affect accurate evaluation of fluorescence imaging, a series of preprocessing steps are applied to calibrate fluorescence imaging for improved accuracy and standardization, as stated in Ref. 28. An optical attenuation model is applied to integrate multiple histological slides and to project deeper tumor regions to the surface. Preliminary aspects of this model are presented in Image-Guided Procedures, Robotic Interventions, and Modeling panel within 2018 SPIE Medical Imaging.¹¹ The whole framework is verified using a mouse model of brain tumor labeled with NIR molecular probe, BLZ-100 (Blaze Bioscience Inc., Seattle, Washington).²⁴ Multimodal scanning fiber endoscope (mmSFE), a wide-field multimodal imaging device that captures fluorescence and reflectance simultaneously with exact spatial registration, is used to image BLZ-100 tumor indicator ex vivo.

2.

Materials and Method

2.3.

Multimodal Scanning Fiber Endoscope Imaging

The mmSFE imaging system is specifically designed for wide FOV, high-quality, video-rate (30 Hz), concurrent multiple fluorescence imaging.^31,32 Three diode lasers at 488, 638, 785 nm are combined and sent to a single-mode optical fiber, which is driven by a piezo tube actuator and scanned in a spiral pattern [Fig. 1(a)]. Fluorescence and reflected light are then collected by a concentric ring of high numerical aperture multimodal optical fibers [Fig. 1(b)]. The mmSFE has an overall flexible shaft diameter of 1.2 to 2.4 mm and a rigid tip length of 9 mm [Fig. 1(c)], depending on the size and number of these collection fibers. Four imaging channels can be produced concurrently, one for reflectance (638 nm) and three for fluorescence including one NIR channel, using separate filters and Hamamatsu MPPC detectors. In this study, a 2.4-mm diameter endoscope was used for BLZ-100 imaging. The NIR fluorescence is projected to the green channel with the reflectance as grayscale.

Fig. 1

Schematic illustration of mmSFE (a) a spiral scanning pattern generated by a single-mode optical fiber, (b) a concentric ring of multimodal collection fibers with varying diameter from 2.4 to 1.2 mm, and (c) a 9-mm rigid tip.

The whole brain surface was first imaged using the mmSFE at 1 cm away from the scope distal tip, which was then sectioned into 2-mm slices coronally. Tissue sectioning was performed using the sharpest instruments possible to allow for clean cutting and to prevent damage associated with crushing or tearing of tissue. Double-edged blades with a thickness of 0.10 mm (Electron Microscopy Sciences Cat. # 72003-01) were used to cut the brain that was 3 to 10 h post euthanasia. Cutting blocks were used for high-quality sectioning (Braintree Scientific 1 mm Coronal). Immediately after sectioning, the sliced brains were placed in ice-cold PBS buffer and imaged using mmSFE at 1 cm away from the scope tip. Reflectance and fluorescence images of the slices were captured with dimmed laboratory lights on. In summary, five slices were imaged using mmSFE for each mouse.

2.4.

Histology

After mmSFE imaging, the 2-mm brain slices were fixed in 10% neutral buffered formalin solution for 48 h. Then, routine hematoxylin and eosin (H&E) staining was done for histological evaluation. The formalin-fixed tissues were embedded in paraffin, which were later cut into $4\text{-}\mu \mathrm{m}$-thickness slides coronally at $100\text{-}\mu \mathrm{m}$ intervals. For each tissue block, 5 to 6 slides were cut. The cut slides were stained with H&E and imaged using a Leica color pathology slide scanner (Aperio Scanscope AT, Leica Biosystems Imaging, Inc.). The whole protocol for mmSFE imaging and histology processing is shown in Fig. 2.

Fig. 2

Protocol for mmSFE imaging and histology processing: the whole brain is imaged using mmSFE and then sectioned into 2-mm slices coronally. The sectioned slices are imaged using mmSFE. Routine H&E staining is performed on each 2-mm slice.

2.5.

Calibration of Near-Infrared Fluorescence Imaging for Tumor Margin Delineation

2.5.1.

Preprocessing for multimodal scanning fiber endoscope fluorescence imaging

The fluorescence data acquired using mmSFE were preprocessed for accurate evaluation and standardization, which consisted of the two steps shown below.

Specular reflection detection and removal

Specular reflections are usually observed from moist tissue surface, which can be a problem in minimally invasive surgeries, where movement of the scope is limited, and the reflections cannot be moved off regions of interest. To provide quantitative evaluation of fluorescence imaging, these undesired reflections need to be removed. Figure 3 shows the pipeline to remove specular reflection. Specular pixels in reflectance channel were first detected using MATLAB function multithresh, which applies Otsu’s algorithm to segment specular reflection. A dilated specular mask was then generated using MATLAB function imdilate with a $3\times 3$ square structuring element. Since mmSFE captured coregistered reflectance and fluorescence images, the generated specular mask from reflectance would apply to both reflectance and fluorescence channel. Each pixel in dilated specular mask was then replaced by the average of neighbor pixels without specular reflection for both reflectance and fluorescence images. A neighbor size $7\times 7$ was applied. If the neighbor pixels were all affected by specular reflection, we increased the neighbor size until there were pixels without specular reflection.

Fig. 3

Pipeline for specular reflection removal: specular reflection regions are first detected, and a dilated specular mask is generated. Each pixel in the dilated specular mask is then replaced by the average of neighbor pixels.

Fluorescence normalization

The fluorescence background was previously measured by placing a nonfluorescent white reference 1 cm away from the scope tip with the NIR excitation source, which is later subtracted from raw fluorescence to reject ambient light. To compensate for inconsistent distance between the scope distal tip and various points on the imaged tissue, a ratiometric method is applied. From any point on the tissue surface, the fluorescence and reflectance generated will travel along identical paths to the scope end-face. Therefore, the difference in distance can be compensated using the ratio between these two images. Since mmSFE can capture reflectance and fluorescence images concurrently, there is no registration error between these two channels, which makes this ratiometric method more robust. The method is first tested using a uniform gelatin phantom (10%), mixed with ICG. Gelatin powder (G2500, Sigma-Aldrich) is mixed with distiller warm water (40°C). The gelatin/water mixture is stirred until all the gelatin is fully dissolved, resembling a uniform liquid. ICG (1340009, Sigma-Aldrich) solution is then dispersed in the mixture. The final concentrations of ICG are $1\mu \mathrm{M}$. An example for this normalization process is shown in Fig. 4. To avoid divide-by-zero errors, each pixel in the reflectance images is added by 1.

Fig. 4

Ambient light rejection and distance compensation (F-fluorescence and R-reflectance): ambient light is first subtracted from the raw fluorescence. A ratiometric method is applied to compensate for inconsistent distance between the scope distal tip and the various points on the imaged tissue.

2.5.2.

Multistep fluorescence registration and two-dimensional tumor map synthesis

A multistep image registration procedure was developed and applied to the preprocessed fluorescence image to enable pixel-wise comparisons with stacked histological slides. A surface macrohistological image was first synthesized from 3-D histological slide deck. Then, a multistep registration was then conducted between mmSFE reflectance image and the surface macrohistological image. The transformations calculated from reflectance image were applied to the corresponding fluorescence image. A 2-D tumor map was extracted from histological slides using optical attenuation model and average intensity projection (AIP), which was used as ground truth for comparison. The detailed procedure follows the steps described below.

Surface macrohistological image synthesis

NIR light penetrates deeper in tissue, which excites the fluorescence beneath tissue surface. To better correlate fluorescence with histological images from both surface and deeper tissues, a 3-D histological slide deck is first generated. Each slide in the 3-D deck is registered to its adjacent slide using MATLAB functions “imregtform” and “imregdemons,” which perform affine transformation and nonrigid registration, respectively. The top-view surface pixels are first extracted from the 3-D histological slide deck and projected to a 2-D plane to synthesize a surface macrohistological image, which is then used to register with mmSFE reflectance image. This surface macrohistological image provides compensation for some incomplete histological slides being cut from uneven tissue surface in paraffin blocks (Fig. 5).

Fig. 5

Synthesized macrohistological image: surface projection is applied to synthesize a macrohistological image from 3-D histological slide deck, which compensates for the incomplete slides.

Multistep registration

Since mmSFE can capture reflectance and fluorescence images concurrently, there is no spatial registration error between these two images. The surface macrohistological image is registered to the mmSFE reflectance image captured from the 2-mm brain slice, which contains more structural information for registration. In step 1, the boundary of the tissue is identified by leveling the images using MATLAB function “graythresh.” Binary masks of tissue for both reflectance image and macrohistological image are generated. MATLAB function imregtform is used to register these two binary masks for affine transformation. The obtained affine transformation matrix T1 is applied to the fluorescence image captured from the same 2-mm slice using imwarp; in step 2, structural landmarks are selected in both macrohistological image and mmSFE transformed reflectance image from step 1 to serve as control points for a piecewise linear registration using MATLAB function cpselect and a deformed transformation grid T2 is generated using MATLAB function fitgeotrans. The processes of cutting, fixing, and staining of the tissue to create histological slides introduce local deformations, which cannot be corrected using affine transformation. The piecewise linear registration divides the images into triangular meshes, which are individually registered through linear transformations.³³ Small spacing of control points allows modeling of local deformation and can address local geometric distortions, whereas large spacing is used to model global deformations. Finally, the deformed grid T2 is applied to the transformed fluorescence image from step 1. The entire pipeline is shown in Fig. 6.

Fig. 6

Pipeline for multistep registration between surface macrohistological image and mmSFE reflectance image: affine transformation and piecewise linear transformation are applied to correct global transformations (rotation, translation, and scaling) and local distortions, respectively. The obtained affine transformation matrix T1 and deformed grid T2 is applied to both reflectance and fluorescence image.

Two-dimensional tumor map synthesis

A method based on optical attenuation model and AIP was applied to generate a projected 2-D tumor map, which integrated tumor information from different tissue depths. For each digitized H&E-stained image in a 3-D histological slide deck, the tumor profile was outlined by an animal pathologist technician with extensive experience in mouse brain tumor. The outlined tumor profiles were extracted to form a 3-D tumor map deck. Then optical attenuation model was used to simulate attenuated fluorescence emitted from deeper tumor before projecting the tumor maps to the outer surface. The details of these steps are described below.

The tumor regions are outlined from the 3-D slide deck, and an optical attenuation model is applied to this 3-D slide deck to compensate the scattering and attenuation in tissue. The effective attenuation coefficient can be calculated as follows:

Eq. (1)

$$\begin{gathered} {\mu }_{\mathrm{eff}}=\sqrt{3{\mu }_a({\mu }_a+{\mu }_s^{\prime })} \\ {\mu }_s^{\prime }={\mu }_s(1-g), \end{gathered}$$

where ${\mu }_{\mathrm{a}}$ (${\mathrm{cm}}^{-1}$) is the absorption coefficient equal to 0.1 in biological tissue;³⁴ $g$ is the anisotropy factor equal to 0.9,³⁵ and ${\mu }_{\mathrm{s}}$ (${\mathrm{cm}}^{-1}$) is the scattering coefficient equal to $125{\mathrm{cm}}^{-1}$ estimated for cortical layers from 300 to $1500\mu \mathrm{m}$ in the NIR range of 830 nm.³⁴

The attenuated intensity for each pixel can be calculated as

Eq. (2)

$$I=I_0*e^{-{\mu }_{\mathrm{eff}}d},$$

where $I_0$ is the initial intensity of the emitted fluorescence, and d is the distance traveled through the tissue. In this paper, we used ${\mu }_{\mathrm{eff}}$ (${\mathrm{cm}}^{-1}$) equal to $1.94{\mathrm{cm}}^{-1}$ for shallow cortical layers and $3.34{\mathrm{cm}}^{-1}$ for depths $>1500\mu \mathrm{m}$, measured by Abdo et al.³⁴ in the NIR range of 830 nm. After the attenuation model was applied to 3-D tumor map deck, AIP was used to project attenuated tumor maps to the outer surface, as shown in Fig. 7. The synthesized 2-D tumor map was used as ground truth and compared with the registered fluorescence image captured from the 2-mm slice.

Fig. 7

A 2-D tumor map synthesis using optical attenuation model and AIP: an optical attenuation model is applied to the tumor regions extracted from 3-D H&E slide deck. AIP is used to integrate attenuated tumor maps from different tissue depths as a 2-D tumor map, which is later used as ground truth and compared with fluorescence.

2.6.

Performance Metric and Statistical Analysis

To evaluate the proposed pipeline in correlating fluorescence images with histological images and to evaluate BLZ-100 in tumor margin identification, brain slices with tumor and fluorescence labeling were included in the performance analysis. Slices with nonspecific binding fluorescence were excluded. Figure 8 shows an example of nonspecific binding, where BLZ-100 was accumulated in the choroid plexus, which is identifiable from other brain tissues during surgery. For each included slice, the synthesized 2-D tumor map was used as ground truth. An additional region of interest representing the tumor margin was created from the $300\text{-}\mu \mathrm{m}$ peripheral aspect of the 2-D tumor map using MATLAB function imerode. About 5000 pixels, accounting for $4.5{\mathrm{mm}}^2$ in area, were randomly selected from the tumor core, tumor margin, and background regions, enabling objective comparisons of the fluorescence. Target-to-background (T/B) ratio was then calculated in the tumor core and margin for each slice. Fluorescence intensity was used for fluorescence image thresholding to delineate tumor boundary. A pixel-wise comparison between 2-D tumor map and thresholded fluorescence image was conducted. The optimal threshold was determined using Youden’s index of the receiver operating characteristic (ROC) curve,³⁶ which optimizes the sum of sensitivity and specificity. Sensitivity, specificity, and DSC, which is a popular metric for evaluating image segmentation and registration,³⁷ were calculated accordingly. The AUC of ROC curve was performed to determine the accuracy to differentiate between normal tissue and tumor, which evaluates the proposed methods independently of the threshold. To evaluate the performance of using synthesized 2-D tumor map as ground truth, the average sensitivity, specificity, DSC, and AUC were calculated and compared to those of using selected best slide (first complete slide in the 3-D slide deck) and randomly selected slide from the 3-D slide deck as ground truth. Group averages and variances were analyzed using paired samples $t$-tests and $F$-test to assess differences in the performance using stacked slides, selected best slide, and randomly selected slide as ground truth, in terms of the accuracy in differentiating between tumor and normal brain.

Fig. 8

Example of BLZ-100 nonspecific binding in the choroid plexus: (a) BLZ-100 binds to choroid plexus shown in fluorescence image (b and c) corresponding histological images of the choroid plexus.

3.

Results

3.1.

Distance Compensation Performance

The proposed method, to compensate for inconsistent distance between the scope and the tissue, was tested using a uniform gelatin phantom. The mmSFE scope tip was placed perpendicular to the phantom at 1 cm away. Fluorescence and reflectance images were captured and combined as a ratio image, as shown in Figs. 9(a)–9(c). A 20 pixel-wide line profile was generated for each image at the same coordinates. The average intensity of the 20 pixels was plotted along the profile, which showed large variations in fluorescence (blue) and reflectance (red). The fluorescence and reflectance intensity at the image center was much greater than that at the periphery. The combination of reflectance and fluorescence as a ratio image (yellow) reduced these variations along the profile [Figs. 9(c)–9(d)].

Fig. 9

Proposed distance compensation method tested using a uniform gelatin phantom mixed with ICG ($1\mu \mathrm{M}$). The (a) reflectance and (b) fluorescence images collected using mmSFE from the gelatin phantom at 10 mm away. (c) The combined ratio image using reflectance and fluorescence image. (d) The average intensity of 20 pixels plotted through line profiles in (a)–(c) shows large variations in fluorescence (blue) and reflectance (red) along the profile. The combined ratio image (yellow) reduces these variations.

3.2.

Multistep Image Registration

The surface macrohistological image was created from 3-D H&E slide deck and registered to calibrated mmSFE reflectance image from 2-mm slice. Affine transformation was applied to correct global deformations, such as scale difference, orientation difference, while piecewise linear registration was applied to correct local deformations. Figure 10 shows examples of registration results: original mmSFE reflectance image before registration (a, f, k); mmSFE green fluorescence image overlaid on reflectance image after registration (b, g, l); thresholded mmSFE fluorescence image overlaid on reflectance image after registration (c, h, m); synthesized 2-D green tumor map overlaid on grayscale macrohistological image (d, i, n); and manually delineated tumor map overlaid on selected best slide (e, j, o). Alpha blending was applied to overlay fluorescence image over reflectance, which maintains the structural information present in the reflectance while adding the molecular information of the fluorescence distribution. As can be seen in Fig. 10(n), the synthesized 2-D tumor map projected deeper tumor directly onto the surface. Tumor map from selected slide may underestimate the tumor information, as seen in Figs. 10(e), 10(j), 10(o). Fluorescence hotspot colocalized well with the tumor regions on the synthesized 2-D tumor map. However, the spatial distribution of BLZ-100 was not uniform. BLZ-100 showed strong accumulation in tumor core, while the margin appeared more diffuse, as seen in Figs. 10(b), 10(g), 10(l).

Fig. 10

Examples of multistep registration: (a, f, k) original mmSFE reflectance image; (b, g, l) mmSFE fluorescence image overlaid on reflectance image after registration; (c, h, m) thresholded mmSFE fluorescence image overlaid on reflectance image after registration; (d, i, n) synthesized 2-D tumor map overlaid on macrohistological image; (e, j, o) manually delineated tumor map overlaid on selected best slide.

3.3.

Overall Framework Performance and Statistical Analysis

A total of 50 brain slices were used for mmSFE fluorescence imaging. Four slices became incomplete during the histological process. The confusion matrix of the remaining 46 slices is shown in Table 1. The number of slices with tumor that were successfully labeled using BLZ-100 was 29. These true-positive cases were used to evaluate this fluorescence imaging agent in tumor margin identification. For true negatives, six slides had no tumor present and displayed no fluorescence label. Eight slides with nonspecific BLZ-100 binding in the choroid plexus were excluded in performance analysis. Three false-negative cases were found, where little or no fluorescence was observed in slices with tumor.

Table 1

Confusion matrix of BLZ-100 performance in labeling medulloblastoma tumor model (Med-114FH).

	Condition positive (slices w/tumor)	Condition negative (slices w/o tumor)
Test outcome positive (w/ fluorescence)	29	8
Test outcome negative (w/o fluorescence)	3	6

For the 29 included true-positive cases, T/B ratios for the tumor core and margin were calculated using 5000 different points ($4.5{\mathrm{mm}}^2$) randomly selected from tumor core, margin, and normal regions. The mean T/B ratios ($\mathrm{mean}\pm \text{standard deviation}$) for tumor core and margin were $8.64\pm 5.76$ and $4.82\pm 2.79$, respectively, with $p$-value $1.86\times {10}^{-06}$ determined by paired $t$-test. Fluorescence intensity was used to threshold tumor boundary, and ROC curve was generated accordingly. ROC curve analysis showed marked difference in the performance among using integrated stacked slides as ground truth, using selected best single slide as ground truth, and randomly selected single slide as ground truth, in terms of differentiating between tumor and healthy tissue. Mean sensitivity, specificity, DSC, and AUC of ROC curve were investigated to compare the methods to generate tumor map. The synthesized 2-D tumor map performed better than the selected best slide and randomly selected slide, as shown in Table 2 and Fig. 11. The average DSC for synthesized 2-D tumor map was significantly greater than that for selected best slide (0.582 versus 0.398 with $p$-value $2.36\times {10}^{-09}$) and that for randomly selected slide (0.582 versus 0.396 with $p$-value $2.28\times {10}^{-07}$). The average AUC for synthesized tumor map was greater than that for selected best slide (88% versus 85.5%), but there was no statistically significant improvement. The average AUC for synthesized tumor map was significantly greater than that for randomly selected slide (88% versus 84.1% with $p$-value $2.18\times {10}^{-03}$). The variance of the AUC using synthesized tumor map as ground truth was significantly less than that using selected slide as ground truth (0.057 versus 0.087, $p$-value 0.0145) and that using randomly selected slide as ground truth (0.057 versus 0.085, $p$-value 0.0212).

Table 2

Mean sensitivity, specificity, DSC and AUC in differentiating between normal tissue and tumor, compared among the synthesized 2-D tumor map, selected best slide, and randomly selected slide.

	Sensitivity	Specificity	DSC	AUC
“All” slides (synthesized 2-D tumor map)	0.819 (0.797 − 0.842)	0.821 (0.795 − 0.846)	0.582 (0.510 − 0.654)	0.880 (0.859 − 0.901)
Selected best slide	0.817 (0.781 − 0.852)	0.800 (0.773 − 0.831)	0.398 (0.322 − 0.474)	0.855 (0.823 − 0.887)
Randomly selected slide	0.817 (0.768 − 0.866)	0.784 (0.734 − 0.834)	0.396 (0.322 − 0.470)	0.841 (0.810 − 0.872)

Note: The 95% CI of the mean is indicated in the parentheses.

Fig. 11

Mean AUC (top row) and mean DSC (bottom row) in differentiating between normal tissue and tumor, is compared among the synthesized 2-D tumor map, selected best slide, and randomly selected slide. Significant differences between the groups are determined by paired $t$-test. The 95% confidence interval (CI) of the mean is indicated by the error bar.

4.

Discussion and Conclusion

The overall goal of this paper is to develop a pipeline to evaluate the performance of fluorescence imaging agents in terms of tumor identification. This work is divided into two parts, first the fluorescence image is calibrated for accurate fluorescence quantification. As stated in Ref. 28, the performance of fluorescence imaging is affected by the variable experimental settings, including lesion depth and camera–tissue distance. To connect fluorescence imaging to clinical outcome and to ensure a reproducible performance, calibration of fluorescence imaging is required. To address this problem, a specular reflectance rejection algorithm is applied, and a background subtraction method is used to reject ambient light. Inconsistent distance and angle is then compensated by combining reflectance image and fluorescence as a ratio image. When testing in the gelatin phantom, the ratiometric method compensated nonuniform fluorescence. As can be seen on the intensity plot (Fig. 9), fluorescence is much higher in the center of the image than at the periphery. The ratio image corrected for these differences over the width of the image and the uniform fluorescence (80% to 100% of the maximum fluorescence) is extended from 50% to 80%. Second, a multistep registration framework is established to correlate fluorescence images to stacked histological images. To preclinically evaluate an imaging agent, a proper registration between the fluorescence and histological images is needed. However, current registration methods compare to single selected H&E slide,^26,27 which is subjective and may underestimate fluorescence from deeper lesions, especially for NIR fluorescence agents. In this paper, an optical attenuation model and AIP are applied to integrate multiple histological slides and project deeper tumor regions to the surface. One advantage of this proposed method is the use of stacked H&E slides from surface to deeper tissues, to generate a synthesized 2-D tumor map, which provides an approximation of light in “seeing” tumor from deeper tissue. Then global and local deformations are corrected using affine transformation and piecewise linear registration. Structural landmarks from the reflectance image are selected as control points for piecewise linear registration. NIR agent, BLZ-100, is used to evaluate the proposed pipeline. The 2-D synthesized tumor map is compared to the selected slide as ground truth. In this study, we select the complete slide with the best quality and a random slide in the 3-D slide deck as comparison. Sensitivity, specificity, DSC, and AUC are calculated as evaluation metrics. The optimal fluorescence intensity threshold for tumor delineation is calculated based on Youden’s index, which optimizes the sum of sensitivity and specificity. There is no significant difference in sensitivity and specificity when comparing the three methods. However, we obtain a significantly higher DSC when using the synthesized tumor map as ground truth. The reasons are illustrated as follows: (1) the selected slide only accounts for a single plane from the 3-D tumor, which underestimates the tumor information. The underestimated tumor map results in a reduced number of true-positive pixels and an increased number of negative pixels, as seen in Figs. 10(e), 10(j), 10(o). In contrast, there are more true-positive pixels and fewer negative pixels when using synthesized 2-D tumor map as ground truth. (2) The imbalanced pixel number between normal tissue and tumor aggravates the discrepancies in the ratio between the number of negative pixels and the number of true-positive pixels (synthesized tumor map versus selected best slide: 7.42 versus 18.57, with $p$-value $2.55\times {10}^{-05}$ determined by paired $t$-test). The significant discrepancies in the ratio between the number of negative pixels and the number of true-positive pixels result in a significant difference in DSC, as shown in the equation

The experiment is conducted on ex-vivo tissue as preclinical evaluation. The performance of this pipeline needs to be investigated for in-vivo intraoperative imaging, which can be challenged by the complex tissue properties and the variable experimental arrangement. For example, we fixed the mmSFE position with one perspective for tissue imaging in this study. During freehand surgical applications, mmSFE position will be adjusted for multiperspective views in real-time video. The change of the scope–tissue angle may diminish the importance of the proposed distance compensation method. One assumption of the proposed method is that the fluorescence and reflectance light travels along identical paths to the scope, which is valid for neoplastic lesions topically labeled from the epithelial layer.²² However, systemic labeling of lesions using NIR fluorescence can emit from deeper tumors, which makes the light paths not identical to the scope. A tilted orientation of scope-to-tissue surface may magnify this error. On the other hand, the multiperspective views can be used for elimination of specular reflections without loss of information. Current specular removal method uses surrounding pixels to replace high-intensity pixels from specular reflection, which only corrects high intensity but neglects the original structural features beneath the specular reflection. Future work is needed to design new specular reflection rejection algorithm, which corrects specular reflection using the same region from a different perspective view.

One limitation of this study is that only the 29 slices with tumor and fluorescence labeling are included in the performance analysis, which limits a comprehensive evaluation of BLZ-100 in tumor identification. However, we evaluate the proposed pipeline in correlating fluorescence images to stacked histological images instead of testing the clinical effectiveness of BLZ-100. In addition, five H&E slices are integrated to generate a synthesized 2-D tumor map, which accounts for only 0.5 mm in depth. Depending on the type of tissue, the penetration depth of light is $<1\mathrm{mm}$ at 400 nm and increases to about 5 mm at 700 to 900 nm using skin as an example.³⁸ The NIR fluorescence detection depth of the mmSFE is measured using ICG solution from $100\mu \mathrm{M}$ to 100 nM diluted in dimethyl sulfoxide. ICG solutions from $100\mu \mathrm{M}$ to 100 nM in microhematocrit capillary tubes (Fisher Scientific, 1.1 mm ID) are buried in NSG mouse brain (0.5, 1, 2, and 3 mm from brain surface). The mmSFE is used to capture the ICG fluorescence at 1 cm away from the brain surface. T/B ratio is calculated accordingly. The mmSFE detection depth is shown in Table 3. From the table, we can see that mmSFE can detect 100 nM ICG at a depth of 0.5 mm and $1\mu \mathrm{M}$ ICG at a depth of 1 mm in mouse brain tissue. Since the estimated BLZ-100 dye concentration accumulated in the tumor is up to the equivalent of $1\mu \mathrm{M}$, applying five layers of H&E slices (total depth of 0.5 mm) to synthesize the 2-D tumor map is underestimated. To better represent the information from deeper lesions detected by the NIR probes, more histological slices, 1 mm in total depth, should be integrated as a synthesized 2-D tumor map and compared with the mmSFE NIR fluorescence image. Moreover, the optical attenuation model is limited by the assumption of straight-line fluorescence attenuation (orthogonal to tissue surface) which is incorrect for highly scattering brain tissue. More complex models, such as Monte Carlo light scattering simulation, can be used to better simulate fluorescence scattering in tissue.³⁹ In contrast, maximal intensity projection (MIP) can be regarded as a simplified model, which has similar results compared to the optical attenuation model. However, the 2-D tumor map synthesized using MIP does not include tumor depth information. Instead, the optical attenuation model and Monte Carlo light scattering simulate the fluorescence scattering in the tissue, which can provide more realistic tumor maps, including tumor depth information. Last but not least, we used single NIR imaging agent for one type of tumor detection in one mouse model. The efficiency of the proposed pipeline in calibrating different imaging agents to histology for different tumor types needs to be further investigated. In our future work, we aim to automate the whole pipeline, thus providing reliable tool to evaluate fluorescent probes in delineating margins. The calibration techniques can be realized in real time and implemented using mmSFE for standardized in vivo fluorescence imaging, along with preprocedure and possibly postprocedure calibrations. The multistep registrations can be automated using laser-induced fiducial landmarks⁴⁰ and algorithm-selected landmarks.⁴¹

Table 3

The mmSFE detection depth measured in T/B ratio.

When BLZ-100 NIR fluorescence is compared to three other fluorescence agents in Elliott et al.,²⁶ in general, their performance in detecting glioma tumor area is comparable. In terms of mean accuracy determined by ROC curve analysis for wild-type tumor, BLZ-100 performed slightly better than the single fluorescence agents (BLZ-100: 88% versus IRDye680: 77% versus PpIX: 87% versus ABY029: 69%), but slightly worse than the combination of these agents (combination: 90%).²⁶ In terms of enhanced fluorescence contrast, BLZ-100 accumulates in both tumor cores and tumor margins with mean T/B ratios of 8.64 and 4.82, respectively. The tumor core showed 1.8-fold higher T/B ratio values compared with tumor margin ($p$-value $1.861\times {10}^{-06}$ determined by paired $t$-test). The margin fluorescence appears more diffuse, which could be scattering fluorescence from deeper tumor lesions into nontumor regions closer to the surface. While PpIX shows weak contrast in tumor margins and appears to contain more high spatial frequency features, sharper edges, and patterns, IRDye680RD enhances the necrotic regions provided by the EPR effect and ABY-029 shows enhanced fluorescence in both tumor core and margin but limits to epidermal growth factor receptor (EGFR) overexpressed tumor type.²⁶ As can be seen, these imaging agents can provide complementary properties (metabolism, perfusion, and EGFR expression) and may be combined with BLZ-100 in the future to provide more accurate tumor contrast for guiding surgery.⁴² For example, in addition to the systemically applied BLZ-100, the QRH-882260 + Cy5 peptide⁴³ can be topically applied intraoperatively to the residual tumor during surgery to provide better visualization of tumor margins. The adoption of multiagent strategy for more accurate tumor delineation can be achieved by using mmSFE, which can detect multiple spectral bands of fluorescence concurrently. There is essentially no registration error across fluorescence channels, which makes multiagent imaging using mmSFE more powerful. Moreover, mmSFE can be modified for an additional OCT channel.⁴⁴ The combination of en face fluorescence imaging and 3-D morphological imaging allows molecular and structural contrasts simultaneously, which has been recently demonstrated using a xenograft mouse model of human colorectal cancer ex vivo.⁴⁵