2.1.

Microscope Filter-Wheel Setup

We build upon the digital microscope filter-wheel setup presented by Wisotzky et al.^23,24 and change the illumination unit from an LED source with its challenging inhomogeneous spectrum to a 175-W xenon (Xe) light source of LEJ GmbH, Germany. Compared to the previously used LED source, the Xe light exhibits a wider and brighter spectrum in the interval of interest from 400 to 850 nm, plotted in Fig. 2. Additionally, the internal DC supply of the Xe source reduces time-based fluctuations of the luminous flux, which allows quantitative long-term microscopic measurements. The luminous flux typically lies at $\sim 500$ lumen and the radiometric power of $\sim 1\mathrm{mW}/\mathrm{nm}$ is nearly homogeneous in the spectral range of interest. As this is a low level of intensity, during the data acquisition all other light sources, e.g., ceiling and operation light, in the room must be turned off to avoid scattered and indirect light from these sources.

Fig. 2

The spectrum of the 175-W xenon light source used in our setup. The relevant interval for illumination is 400 to 850 nm. In this interval, specific wavelengths are selected (solid vertical lines for the microscope filter-wheel setup) for illumination such that the illumination intensity stays almost constant and intensity peaks (e.g., between 810 and 845 nm) are skipped.

Further, we expand the multispectral range from 16 to 19 channels. To the 16 channels in the visual spectrum [400 to 700 nm, with a step size of 20 nm and full-width at half-maximum (FWHM) of 10 nm], we add three channels in the near-infrared (NIR) spectrum (750, 800, and 850 nm). These channels have been chosen because of the wider homogenous spectrum and to avoid the high peak between 800 and 850 nm, see Fig. 2. Each of these three filters has FWHM of 25 nm, see Table 1.

Table 1

The spectral peak and FWHM of each band for both setups and all used cameras and filters. The bands are sorted, starting with lowest λ, and the numeration does not correspond to the index positions on the sensor.

The sensor and optics specifications remain the same as presented by Wisotzky et al.²³ The sensor output resolution is $1920\times 1080\text{pixels}$ and the used focal length at maximum zoom is 65 mm. The working distance (WD) is about 210 mm.

2.2.

Hyperspectral Camera Setup

The alternative HSI setup consists of up to two hyperspectral snapshot cameras of XIMEA, Germany. A snapshot camera has the advantage of acquiring the complete hyperspectral dataset in a single shot, whereas on the other hand, the spatial resolution is reduced due to the larger filter array. The first camera holds a $4\times 4$-filter array, resulting in 16 HSI-bands between 460 to 630 nm with different FWHMs, from $\mathrm{FWHM}=4.69\mathrm{nm}$ at $\lambda =460\mathrm{nm}$ in band 7 up to $\mathrm{FWHM}=20.18\mathrm{nm}$ at $\lambda =478\mathrm{nm}$ in band 0, cf., Table 1. The second camera holds a $5\times 5$-filter array, sensitive from 600 to 975 nm. Since the filters of the camera show two main peak responses in some pixels of the $5\times 5$-filter array, a 675-nm long-pass filter is placed in front of the camera, resulting in 25 HSI-bands between 675 to 975 nm with an FWHM range from $\mathrm{FWHM}=3.99\mathrm{nm}$ at $\lambda =693\mathrm{nm}$ in band 9 up to $\mathrm{FWHM}=25.00\mathrm{nm}$ at $\lambda =955\mathrm{nm}$ in band 25, cf., Table 1. Each band has a specific response pattern containing primary and eventually secondary peaks in the sensitive interval of the sensor, see Fig. 3. Therefore, the band response is not a correct spectral signature as the signal is a combination of the sensor level response and the system level components. To measure a correct spectral signature, it is required to calibrate the signal through spectral correction.²⁵ Further, the exposure times for both hyperspectral cameras will differ due to different sensor response characteristics. For the $4\times 4$ HSI camera, the peak response intensity for each band is about 20%, except band 3 with $\sim 10\%$, see Fig. 3(a). For the $5\times 5$ HSI camera, the peak response intensity for each band is lower with about 10%, see Fig. 3(b). Hence, the exposure time has to be longer for the $5\times 5$ HSI camera to achieve a similar sensor response. These response gaps of the sensors, i.e., response intensity difference between band 3 and all others of the $4\times 4$ sensor as well as bands 0 to 4 and bands 15 to 24 on one side and bands 5 to 14 on the other side of the $5\times 5$ sensor will be addressed in the calibration process.

Fig. 3

The specific sensor responses for (a) all 16 bands of the $4\times 4$ spectral pattern of the used visual hyperspectral snapshot camera and (b) all 25 bands of the $5\times 5$ spectral pattern of the used NIR hyperspectral snapshot camera presented by the manufacturer XIMEA, Germany.

The sensors of both cameras have a resolution of $2048\times 1088\text{pixels}$, in which the filter area is $2048\times 1024\text{pixels}$ for the $4\times 4$ HSI camera and $2045\times 1080\text{pixels}$ for the $5\times 5$ HSI camera. Both cameras hold a 75-mm $f/2.8$ lens and are aligned on a rack, focusing on the same point in a distance of $\sim 250\mathrm{mm}$, which is the smallest possible WD for the setup and thus, the WD of the hyperspectral camera setup. We use two cameras to increase the number of spectral channels over the complete visual and NIR range (465 to 975 nm) while keeping a reasonable resolution. We tested two configurations for scene illumination, the Xe source of the microscope filter-wheel setup (in a distance of $\sim 210\mathrm{mm}$) and the surgical LED light, see Fig. 4, in a distance of $\sim 300\mathrm{mm}$. The large difference of the luminous flux of the LED light (which is much higher) and the Xe source make a robust calibration of the complete system indispensable. Additionally, the intensity spectrum of the LED source is inhomogeneous with a high peak at $\sim 460\mathrm{nm}$ and high intensity in the range of 525 to 680 nm. In the near UV range (up to 430 nm) and the NIR range (beginning at 750 nm), the illumination intensity is very low, which results in a low signal-to-noise ratio (SNR).

Fig. 4

The spectrum of the surgical LED light source used in our setup. Compared to the used Xe source, cf., Fig. 2, this LED spectrum is very inhomogeneous, especially the intensity is decreasing rapidly starting at 680 nm, which results in low illumination intensity for the NIR spectrum.

2.3.

Calibration Chain

The introduced difficulties and impacts on data quality require a robust calibration with reference data. Since the measurement of the reference data includes uncertainties, errors will propagate into the corrected data, which cannot be quantified in the postprocessing. In order to make the acquired data comparable between the two setups and other spectral acquisition techniques, a joint calibration chain is developed in MATLAB, sketched in Fig. 5. The calibration process has to take the following different aspects into account: (a) sensor and lens effect correction, (b) denoising, (c) transforming the measured data to physical units, (d) spectral correction, (e) spectral channel registration, and (f) systematic shift correction. All corrections are performed on the measured sensor data as an entire unit, to also correct effects such as crosstalk.

Fig. 5

The general calibration workflow for both setups. The calibration steps (a)–(c) as well as (e) and (f) are needed for both setups, whereas step (d) is only performed for the hyperspectral camera setup. Calibration step (a) corrects the internal setup effects of the used sensors and lenses. The sensor noise is reduced in step (b). In step (c), the measured sensor data are converted to physical reflectance values. Step (d) ensures that the specific spectral information is achieved and crosstalks are eliminated. The steps (e) and (f) are necessary for visualization and validation of the spectral behavior of both setups.

2.3.5.

Intensity correction

Due to uncertainties of the provided calibration information as well as the described internal and external effects, e.g., differences in WD, systematic shifts occur in the reconstructed data for each part in the setups, i.e., filter-wheel setup, $4\times 4$ HSI camera, and $5\times 5$ HSI camera. These identified shifts are corrected in the last calibration step using

Eq. (5)

$$I_{\text{final}}=\alpha I_{\mathrm{res}},$$

where the scalar $\alpha$ is the scaling factor and $I_{\mathrm{res}}$ is the reflectance intensity from Eq. (2). This scaling factor $\alpha$ is a proportionality constant, which establishes the correlation between spectral illumination intensity and sensor signal response. For constant WD and sensor, it is sufficient to specify $\alpha$ only once.

A chart (ColorChecker Classic, X-Rite, Inc., USA) with 18 colored and 6 gray-scale tiles of well-defined color spectra²² (see Fig. 6) is scanned with both setups and both illumination options and analyzed to identify $\alpha$ and correct the spectral information assuring an accurate comparison between both approaches. Each color spectrum is known in terms of reflectance, with SD for each data point, in the range of 390 and 1000 nm with a step size of 10 nm. This information is used as reference for the scaling calibration and generated using the broadband spectroradiometer specbos 1211UV-2, JETI Technische Instrumente GmbH, Germany. The respective SD over all reference measurements varies between all wavelengths and all color tiles, which is consistent with the literature.²⁹ It lies between 0.2%, i.e., 0.002, (e.g., for tile “foliage” in the range of 430 to 460 nm, or for tile “blue” in the range of 590 to 630 nm) up to 11.0%, i.e., 0.110, (e.g., for tile “light skin” 710 to 730 nm, or tile “purple” 720 to 730 nm). Some color tiles, e.g. “white 9.5,” “light skin,” “bluish green,” “blue,” “orange yellow,” or “yellow,” show larger SD over the whole spectral range, whereas other colors (e.g., “dark skin,” “cyan,” or “magenta”) have only small SD.

Fig. 6

This color chart, ColorChecker Classic, X-Rite, Inc., USA, is used for spectral and scale correction. The 18 colored and 6 gray-scale tiles are used as reference spectra as each color spectrum is well-known. The names of the different colors are written in the corresponding fields to allow a comparison of the measured spectra.

The presented calibration chain does not take into account thermal effects or scattered light, thus these need to be avoided as described in Sec. 2.1. To allow comparison of measured clinical data, acquired with both setups, also the magnification has to be comparable. This analysis is done using a checkerboard allowing the comparison of the segmentation results.

2.4.

First Clinical in vivo Measurements

To prove both presented approaches in the clinical environment, we have acquired first clinical in vivo data. The measurements have taken place during two otorhinolaryngology surgeries. One measurement has been performed during a neck dissection and another during a parotidectomy. In both cases, the surgeon annotated the individual tissue types for later analysis. The HSI camera setup uses two snapshot cameras that both are focusing on the same situs area with a slightly different angle. In both of the resulting images, the surgeon manually annotates the tissue types of interest. Thereby, the surgeon tags areas that cover exactly the same anatomical regions in both images and the two reconstructed spectral information can be combined to one full spectrum. The regions of different tissue types have been analyzed individually to reconstruct the specific spectral behavior of each type.

For the visualization of interesting spectral behaviors, the spectral data are enhanced selecting relevant spectral intervals by a weighting matrix $\mathbf{W}$ with size $41\times 41$ due to $41\lambda$-bands and the principal component analysis (PCA)³⁰ using

3.1.

Calibration Results

3.1.2.

Calibration chain

Figure 7 substantiate the need of the described calibration chain. It shows that acquired pixel values do not represent the actual reflectance information and highly differ between different setups. The comparison of Figs. 7(a) and 7(b) shows that only the combination of spectral calibration and conversion to reflection will result in a physically correct spectrum.

Fig. 7

These curves substantiate the need of the described calibration chain. (a), (b) The two reference spectra (□, magenta tile and ⋄, cyan tile) of the hyperspectral camera setup. (a) The raw data, the filled markers represent the raw sensor counts, and the unfilled markers represent the sensor counts corrected with the specific correction matrix $\mathbf{C}$ from Eq. (4). (b) The corrected and white balanced data (the dotted lines show the reference data of the color chart). The unfilled markers represent the calculated reflectance data $I_{\mathrm{res}}$ using Eq. (2) without applying Eq. (4) and the filled markers represent the fully corrected data $I_{\text{final}}$, which is the fully corrected physical reflectance. For the microscope filter-wheel setup, (c) shows the same behavior of the “magenta” tile. (d) The pixel counts (dotted line) are completely different to the HSI camera setup but the reconstructed spectrum (black line) behaves as the reconstructed line of the HSI camera setup and the reference data, which shows the reconstructed magenta spectrum of both setups with the specific reference data of the color chart as dotted line.

For the two different illumination possibilities of the HSI camera setup, the exposure time is longer with Xe illumination, as the luminous flux of the surgical LED light is much higher, cf., Sec. 2.2. The exposure time must be selected such that the full capacity of the sensor sensitivity is optimally used for all bands while a saturation of single bands is avoided. Hence, some channels do not use the full sensor sensitivity and the calculated $I_{\mathrm{res}}$ could be underestimated. To use the full capacity of the sensor sensitivity, the exposure time for the $4\times 4$ HSI camera is lower compared to the $5\times 5$ HSI camera, resulting in a lower frame rate for the $5\times 5$ HSI camera. This is caused by the differences in the sensor response, cf., Fig. 3, as well as the differences in the illumination intensity through the different wavelengths. A low frame rate has no effect on the calibration results, but has to be considered for clinical measurements. Thus for the $5\times 5$ HSI camera, an exposure time is chosen, which does not use the full sensor sensitivity but keeps an acceptable frame rate. This results in a statistical bias. Therefore, the introduced calibration step (f) intensity correction is essential for accurate and comparable signal processing. It will correct such underestimations as well as differences in WD between the individual cameras using the reference curves. The optimal exposure times identified for our setups and illumination settings are presented in Table 2.

Table 2

The identified optimal exposure times per frame for our two setups. For the filter-wheel setup, the stated exposure time is per wavelength band.

Setup	Filter-wheel	4×4 camera	4×4 camera	5×5 camera	5×5 camera
Illumination	Xe	Xe	LED	Xe	LED
Exposure time (ms)	42	65	0.5	110	7.5

The characteristics of the reference color spectra have been correctly reconstructed for all three settings (microscope filter-wheel setup and hyperspectral camera setup with two illumination possibilities) using the proposed calibration steps (a)–(e). A systematic scale shift is observed in the reconstructed spectral points for both setups. This shift depends on the camera type, the illumination setting and distance as well as the WD. Calibration step (f) corrects most of these effects. A weighting factor $\alpha$ is applied, according to the different systematic behaviors of the different setups, see Table 3.

Table 3

The identified weighting factors α to correct systematic scale shifts of the setup calibration.

Setup	Filter-wheel	4×4 camera	4×4 camera	5×5 camera	5×5 camera
Illumination	Xe	Xe	LED	Xe	LED
$\alpha$	2.01	1.00	1.07	1.22	1.03

The filter-wheel setup shows a clearly visible systematic shift, which has to be compensated with $\alpha =2.01$. In the NIR, this shift is growing due to illumination inhomogeneities and a small SNR. The growing part of this shift will not corrected by $\alpha$, cf., Fig. 12. For the $4\times 4$ HSI camera, the internal and external effects, e.g., WD and illumination setting, compensate each other which results in no ($\alpha =1.00$) or no significant ($\alpha =1.07$) difference for the Xe and LED illumination, respectively. The same occurs for the $5\times 5$ HSI camera with LED illumination ($\alpha =1.03$). Due to the above-mentioned challenges with very low SNR as well as the high deviation of the reference data and measurements in NIR, the values above $\lambda >850\mathrm{nm}$ are neglected for $\alpha$-optimization. For the $5\times 5$ HSI camera with Xe illumination, the shift is larger with $\alpha =1.22$.

Figure 8 plots all 18 reconstructed color spectra, acquired with the filter-wheel setup and corrected with the identified weighting factor $\alpha$. The data fits to the reference spectra (of the color chart—dotted lines in Figs. 8–10). All reference data are within the calculated SD of the reconstructed data points as well as all reconstructed data points are within the respective SD of the reference color board. For all color tiles of the chart, the deviation between reference and calculated intensity increases starting at $\sim 680\mathrm{nm}$, due to the following reasons: First, the SD of the reference spectra is increasing at about $\lambda >650\mathrm{nm}$. In the visual range (420 to 680 nm), the SD is below 1%, while for the red and NIR spectrum it increases from 2% up to 11%. Second, for the NIR range, the images contain low signal intensities and the SNR is decreasing. The SNR in the visual range is $\mathrm{SNR}>3$ and $\mathrm{SNR}\approx 2$ at 700 nm, whereas at 400, 750, and 800 nm, it is $\mathrm{SNR}\approx 1$ and for 850 nm it is decreasing to $\mathrm{SNR}\approx 0.1$.

Fig. 8

Calibration results of all analyzed color spectra with the filter-wheel setup. The reference data, plotted as dotted lines, could be fully reconstructed for all curves. The reconstructed data points have a distance of 20 nm in the visual range and a distance of 50 nm in the NIR. The results are comparable to the results of the hyperspectral camera setup (Figs. 9 and 10).

Fig. 9

Calibration results of all 24 analyzed color spectra with the hyperspectral snapshot camera setup and Xe illumination. The reference data, plotted as dotted lines, could be fully reconstructed for all curves. In the region of NIR ($\lambda >840\mathrm{nm}$), the deviation $d$ between reconstruction and reference is increasing, but the results are comparable to the results of the microscope filter-wheel setup (Fig. 8) as well as the HSI camera setup with LED illumination, see Fig. 10.

Fig. 10

A selection of eight out of the 24 measured reference spectra acquired with the hyperspectral camera setup used with LED surgical light. The reference information is plotted as dotted lines. The results show the same behavior for all scanned spectra as with the Xe illumination setup, cf., Fig. 9.

Figure 9 shows the 18 reconstructed and corrected color spectra and six different gray scales with its respective reference spectrum for the hyperspectral camera setup and Xe illumination. The plots demonstrate that the reconstructed data of the $4\times 4$ camera ($460\mathrm{nm}<\lambda <650\mathrm{nm}$) fit the reference values very well. All differences $d$ between reconstructed and reference intensities are within the permissible error tolerances (measured SD as well as stated reference SD). The reconstructed data of the $5\times 5$ camera show higher differences $d$ between reconstructed and reference intensities, especially for the reconstructed data at about $\lambda >840\mathrm{nm}$. With increasing $\lambda >840\mathrm{nm}$, the difference $d$ increases. This effect has several reasons. As described above, the SD of the measured and reference data is increasing in the NIR range, whereas the SNR is very low in that spectral range. Thus it becomes complicated to reconstruct the signal. Further, the large spreading of the highest intensity value between the individual bands on the $5\times 5$ HSI sensor [cf., Fig. 3(b)] results in the behavior that some bands do not use the full sensor sensitivity while other bands could reach saturation. This explains the high variability for the last six data points ($\lambda >930\mathrm{nm}$) as well as the increased deviation of the first data point ($\lambda =693\mathrm{nm}$), since the sensor response is very low for these bands. The same effect occurs at $\lambda =400\mathrm{nm}$ for the filter-wheel setup as the light sources are clipping at $\lambda =400\mathrm{nm}$ to avoid UV illumination. The same behavior, described for the HSI camera and Xe illumination, is observed for the HSI camera with LED illumination. Figure 10 shows a selection of eight reconstructed color spectra.

Compared to the filter-wheel setup, the SNR is higher at each measured wavelength in the visual range and slightly smaller in the NIR, except for $\lambda =850\mathrm{nm}$ where the SNR is higher. For Xe illumination, the average SNR for the visual range ($4\times 4$ camera) is ${\mathrm{SNR}}_{\mathrm{avg}}=116.55$ (${\mathrm{SNR}}_{\min }=6.19$ and ${\mathrm{SNR}}_{\max }=394.45$) and for the NIR range ($5\times 5$ camera) ${\mathrm{SNR}}_{\mathrm{avg}}=0.84$. For the LED illumination, the average SNR for the visual range is ${\mathrm{SNR}}_{\mathrm{avg}}=108.68$ (${\mathrm{SNR}}_{\min }=3.57$ and ${\mathrm{SNR}}_{\max }=413.36$) and for the NIR range ${\mathrm{SNR}}_{\mathrm{avg}}=0.93$ (${\mathrm{SNR}}_{\min }=0.02$ and ${\mathrm{SNR}}_{\max }=9.19$). As can be seen in Fig. 11, the SNR between both illumination options is similar. However, it depends on the chosen exposure time and the used light source. Since the luminous flux of the surgical LED light is much higher compared to the Xe source, the SNR is increased in the NIR using LED illumination. Overall, Fig. 11 shows that the SNR is low for the NIR due to several different effects. The illumination intensity decreases for the LED source in the NIR, which results in a lower light reception on the sensor. Additionally, the filter properties of the $5\times 5$ snapshot camera play an important role as stated above, cf., Fig. 3(b). In the spectral range of 680 to 850 nm, the SNR is relatively homogenous between $0.6\le \mathrm{SNR}\le 0.8$ and shows larger fluctuation for $\lambda >885\mathrm{nm}$. This behavior corresponds to the different peak intensities of the individual bands. The bands with very low intensity of $I_{\text{peak}}\approx 0.04$ are related to the wavelengths with very small SNR.

Fig. 11

The average wavelength-dependent SNR values of each setup. In the visual range, the HSI camera setup shows an increased SNR for both illumination settings compared to the microscopic filter-wheel setup. In the NIR range, the general SNR is much lower. The microscope setup and the HSI camera setup with LED illumination show similar SNR, while for the HSI camera setup with Xe illumination, the SNR is even smaller in the range of 680 to 880 nm.

To compare all three introduced settings, the difference $d$ between the reconstructed spectral point to its reference value has been calculated for each setting. Figure 12 shows that the variances from the reference values are reduced when the scale correction using the presented $\alpha$ of Table 3 is applied. The unfilled markers in Fig. 12 show the gap between the reference data and the reconstructed data using calibration steps (a)–(e), whereas the filled markers represent the results applying the complete calibration pipeline (a)–(f). In this case, all data points are around $d=0$, except for the data of $\lambda >850\mathrm{nm}$ due to the described uncertainties. Hence, the measured data of all three settings become comparable using the calibration pipeline introduced here and the setups can be used for optical tissue analysis, as shown in the next section.

Fig. 12

The quality of the complete calibration chain. The unfilled markers show the calibrated setups without the correction factor $\alpha$. In this case, the filter-wheel setup shows a systematic shift. Due to the smaller SNR of the NIR data, the gap $d$ between reference and reconstructed data (“difference”—plotted on the $y$ axis) is increasing for all setups. The filled markers show that a correction factor $\alpha$ can adjust all settings to the reference and allows comparison among the setup data.

3.2.

First Clinical in vivo Measurements

We have scanned the situs with our setups during two surgical standard procedures. Figure 13(a) shows the situs of both surgeries (patient 1 on the left and patient 2 on the right). In Fig. 13(b), the view to the situs of the $4\times 4$ snapshot camera is shown. The view of the $5\times 5$ snapshot cameras appears in a similar way with less intensity. The microscope filter-wheel setup has been located right next to the snapshot cameras. Figure 14 shows two measured views of two different patients with the filter-wheel setup, from which the optical behaviors of different tissue types are extracted. Compared to the views of the snapshot cameras, the scene is shifted and in the first case additionally rotated. The left image of patient 1, the neck dissection, shows the scene at $\lambda =680\mathrm{nm}$ illumination, whereas the right image of patient 2, the parotidectomy, shows the scene at $\lambda =460\mathrm{nm}$ illumination. Although both views are from different patients, these images show that different tissue types show different optical behaviors under different wavelength illuminations, see also Fig. 15. Clearly visible is this fact in Fig. 13, as the mosaic pattern with all 16 visual wavelength bands is visible and these pattern behave differently for different tissue types.

Fig. 13

The area of the situs that has been scanned with our setups. (a) The complete situs of the two surgeries and (b) the two views measured with the $4\times 4$ HSI camera, with its sketched location in the situs. The camera positions of the RGB overview image and HSI image are not identical, which produces projection differences.

Fig. 14

Two analyzed views of the two patients measured with the filter-wheel setup. (a) The image acquired with illumination at $\lambda =680\mathrm{nm}$ and (b) the image acquired with $\lambda =460\mathrm{nm}$. The interesting tissues are labeled as A, artery; B, vein; C, nerv; D, parotid gland; E, connective tissue; and F, muscle.

Fig. 15

The spectra of several analyzed in vivo tissue behaviors in the visual range. (a) The reflectance of vein and artery of patient 1 and (b) the reflectance behavior of muscle, parotid gland, nerve, and fibrous connective tissue. All tissue types show different optical behaviors through the analyzed spectrum.

Our first two clinical in vivo measurements of human soft tissue show different visible behaviors between the various wavelengths for different tissue types. After the surgeon annotated the tissue types in the captured images, binary masks are created to evaluate each type independently and reconstruct the physical reflectance $R$, shown in Fig. 15. As the visible range (460 to 640 nm) is of highest interest for intraoperative visualization, this interval of the reconstructed spectra is presented in Fig. 15. In the data of patient 1 [Fig. 15(a)], the neck dissection, the focus of the analysis is on the blood vessels, as artery (in this case: arteria carotis) and vein (vena jugularis) transport blood with oxygenated and deoxygenated hemoglobin (Hb), respectively. The analyzed spectral behavior of captured arteria carotis and vena jugularis correspond to the optical properties of human blood presented in the literature.^34,35 The reflectance for oxygenated Hb (arterial blood) is higher in the range of 540 to 590 nm and in the other parts of the visual spectrum lower than the reflectance for deoxygenated Hb (venous blood). Further, oxygenated Hb shows the two characteristic minima in the range of 530 to 585 nm. For the data of patient 2, the parotidectomy, the focus of the analysis is on several different soft tissue types present in the situs: muscle, nerve, parotid gland, and fibrous connective tissue. The visible range of the reconstructed spectra is shown in Fig. 15(b). These spectral data correspond to in vivo measurements using a spectrophotometer.^36,37 Nervous tissue reflects with the highest intensity in the spectral range $\lambda <530\mathrm{nm}$. In the range $\lambda >530\mathrm{nm}$, it is similar to parotid gland tissue. The spectral behaviors of muscle and fibrous connective tissue are lower in terms of reflectance compared to nerve and parotid gland over the whole analyzed spectrum. Until $\lambda =530\mathrm{nm}$, both tissues show similar reflectance behavior, while starting at $\lambda >530\mathrm{nm}$ high differences exist between both tissues. These tissue differences in its spectral behaviors indicate that this knowledge can be used to allow intraoperative tissue differentiation.

Both introduced setups, microscopic filter-wheel as well as snapshot cameras, reveal the same tissue characteristics in both cases with respect to internal and external uncertainties. Since a 3D tissue structure is captured, the surface is never perfectly aligned to the camera and different tissue types have different orientations to the sensor. Therefore, the reconstructed optical behavior can differ in a single tissue structure if it appears at different areas in the image, which results in slightly different spectra for a single tissue type. This is shown in Fig. 15(b) for parotid gland curves (black curves). Both curves represent the same tissue type present in different areas in the image (with slightly different WD and other variances), resulting in a small deviation of both curves but showing the same overall characteristics.

For intraoperative usage, a reasonable visualization of the information extracted from the hyperspectral data is needed. A classical RGB image can be calculated from the hyperspectral data using the CIE color matching functions. Figure 16 shows the reconstructed RGB image from the raw pixel counts for every measured $\lambda$ as well as from the calibrated reflectance data, illustrating the need of the presented calibration pipeline. Without calibration, the wrong spectrum [cf., Fig. 7(c)] results in an overweighting of the green image channel, Fig. 16(a), while the presented calibration pipeline results in a realistic RGB appearance of the scene as it would appear in a conventional surgical microscope, Fig. 16(b). Such an RGB calculation from multispectral data is done at the expense of information loss, since the specific multispectral information is combined in three channels (RGB). Thus the different reflectance properties over the complete analyzed spectrum of the individual tissue types have to be used in an additional step to enhance interesting structures in the calculated RGB image. For example, during otorhinolaryngology surgeries the facial nerve as tissue of risk has to be completely preserved. Therefore in parotidectomy, nerve preparation requires most of the surgical time and is an extremely complex task, as the facial nerve splits into five delicate branches performing important functions as, e.g., facial expression and sensation. Thus the facial nerve should not be injured in any way and a better visualization of the nerve would be beneficial for the preparation as well as the complete surgical process keeping a constant watch on the nerve. To demonstrate a possible application of the presented approach in an intraoperative assistance system, the measured reflectance properties of patient 2 are used to highlight the nervous tissue in the finally calculated RGB visualization, see Fig. 17. The measured data in the spectral range of 460 to 480 nm are enhanced using knowledge about the general optical tissue behaviors³⁷ and the described PCA method of Eq. (6). Spectral differences between image regions, i.e., different tissue types, in the range of 460 to 480 nm are enlarged using an enhancement factor. This enhanced differences are then added to the blue channel of the calculated RGB image [cf., Fig. 16(b)] resulting in a blue appearance of nervous tissue structures (cf., Fig. 17). Using this image, the surgeon is able to identify the nerve easily and monitor it during the entire surgical procedure.

Fig. 16

The reconstructed RGB calculation using the same measured hyperspectral snapshot data and the CIE color matching functions. The measured raw pixel counts are used for RGB calculation in (a) which results in a false greenish RGB appearance, whereas in view (b) the corrected reconstructed spectral data are used for RGB calculation, which results in a realistic RGB appearance.

Fig. 17

The same reconstructed RGB-view as presented in Fig. 16 but with enhanced nerve visualization. In this case, after RGB calculation of the calibrated multispectral data using the CIE color matching functions, the resulted nerve enhancement is added to the B-channel, which results in a nerve enhanced RGB representation.