1.

Introduction

A rapid advancement of hyperspectral imaging techniques has broadened their biomedical applications including novel label-free identification of chemical substances and disease features in cells and tissues,^1,2 optical spectroscopic biopsies of diseased tissue,^3–6 and real-time in vivo monitoring of oxygenation states during a surgical procedure.⁷ For molecular and cellular imaging applications, recent label-free absorption-based high-resolution hyperspectral microscopy has demonstrated the intracellular mapping of endogenous chromophores including oxyhemoglobin (oxy-Hb) and hemozoin in single erythrocytes infected with Plasmodium falciparum (Pf) malaria parasites. In this study, statistical hyperspectral image analysis has been used to achieve spatial distribution of the intracellular molecular substances at a single cell level.⁸ A spectrally resolved spatial map in a multidimensional image data cube obtained by a hyperspectral imaging system contains numerous spectra attributed to multiple spectral origins from “pure” substances with no a priori information on the spectra. This makes the hyperspectral imaging technique uniquely different from conventional spectroscopic imaging methods. One of the end goals in the analysis of hyperspectral image data cubes is to delineate key individual spectra from their superposed spectrum, map their spatial distributions, and explain their origins. To this end, statistics-based image analysis algorithms have been extensively developed for the analysis and interpretation of hyperspectral image data cubes.

For improved measurement accuracy in hyperspectral imaging and analysis, a standardized calibration method to evaluate the performance of image sensors is as important as the validation of analysis algorithms. For an accurate test of image sensors, nonperishable reference materials with well-defined and reproducible spectral and spatial information, so-called “physical phantoms,” have been developed extensively. But many challenges still remain in fabricating tissue-mimicking physical phantoms with an extended shelf life and reproducible optical properties. When the desired optical properties involve complex spatial and spectral characteristics, the fabrication of physical phantoms is costly and time-consuming. To address these limitations, a digital tissue phantom (DTP) platform has recently been demonstrated using a hyperspectral image projector (HIP) based on digital micromirror devices (DMDs).⁹ DTPs are generated by acquiring hyperspectral data cubes from an actual biological specimen and reproducing them by an HIP. In effect, DTPs are hyperspectral data cubes with spectral and functional characteristics of the imaged tissues. In particular for imaging applications involving clinical experiments, DTPs of well-calibrated image scenes may serve as reference images from which the sensors under test collect images for instrument calibration, validation, and verification.

This study demonstrates another type of HIP system based on liquid crystal on silicon (LCoS) spatial light modulators (SLMs) to generate DTPs which were collected by a DMD-based hyperspectral microscope. While the spectral control in a DMD-based HIP system mainly relies on the light intensity control at a narrow band wavelength, a broad range of control mechanisms are available in an LCoS SLM-based system. For this reason, LCoS SLMs have been instrumental in characterizing and manipulating light when quantitative assessment and precise control of intensity, polarization state, wavelength, and wavefront patterns are needed.^10–12 A pixel-by-pixel control of the phase retardance in arrayed SLM devices, combined with polarimetric optics, allows generation of image patterns by effectively converting phase retardance into an intensity gray scale.¹³ Although the polarization sensitivity of the SLM poses many measurement challenges, this capability enables ubiquitous integration of LCoS SLMs into conventional imaging systems for novel imaging applications such as structured light illumination microcopy, multifocal microscopy, and holographic imaging.^14–16 Additionally, wavelength-controlled retardance enables pixel-by-pixel control of the reflectance of the light at specific wavelengths to readily manipulate the spectrum of the reflected light as well. Integrating spatial and spectral control in an arrayed LCoS SLM device enables the generation of a series of image patterns with well-defined spectra.

For hyperspectral imaging in clinical applications, spectrally and spatially controlled image data not only reproduce images indistinguishable from the real scene for further image analysis and medical training but also generate data cubes with clinically important spectral information embedded in the spectral scene or data cube as clinical standards for consistent diagnosis. These reproducible data cubes eliminate costly reproduction of clinical scenes, allowing valuable clinical data to be readily available to nonclinical laboratories for various applications. The data cubes may also be used as standards to test imaging devices for their measurement quality, performance consistency, and interlaboratory comparison. In particular, these unique scenes may be used as standards to calibrate the wavelength axes to achieve high quality hyperspectral imaging systems with improved accuracy in measuring spectral features.

2.

Hyperspectral Image Data Cubes

Our absorption-based hyperspectral microscope shown in Fig. 1 is described in detail elsewhere.⁸ In brief, a high numerical aperture achromatic objective lens collected the transmitted intensity at each wavelength, $I(x,y,{\lambda }_{\text{transmission}})$ of erythrocytes with spectral resolutions of about 6 nm and of a spatial diffraction limit, at 1.4 numerical aperture, in a three-dimensional (3-D) data cube format. Then the transmitted intensities normalized with a background signal through the cell-free region were converted to absorption intensities. The absorbance ($A_{\lambda }$) at each wavelength was calculated by the equation

Fig. 1

A schematic of the hyperspectral microscope using a spectral light engine. During data acquisition by a charge-coupled device (CCD) camera with scanned wavelength, narrow bandwidth, and controlled intensity for each bandwidth are achieved by selecting the number of “on” mirrors on each column corresponding to a narrow spectral bandwidth of the spectrally broadened white light and projected on a digital micrometer device (DMD). CL stands for condenser lens.

The main chromophore substances identified from the analysis of erythrocytes both normal and infected with a Pf malaria parasite at a parasitemia of 5% are intracellular oxyhemoglobin (oxy-Hb) and hemozoin molecules. For sample preparation, human O+ erythrocytes were purchased from Interstate Blood Bank (Memphis, Tennessee) and filtered with a Sepacell R-500 filter (Baxter, Deerfield, Illinois) with $2\times$ volume of incomplete Roswell Park Memorial Institute (RPMI) 1640 cell culture media (Invitrogen, ThermoFisher Sci., Waltham, Massachusetts). Erythrocytes were washed and suspended in RPMI 1640 at 50% hematocrit and stored at 4°C. Attachment and fixation of erythrocytes were done with an established procedure reported elsewhere.¹⁷ Oxy-Hb and deoxyhemoglobin (deoxy-Hb) are known to show different absorption spectra and their spectral ratio has been used as an indicator to assess the tissue oxygen level. Another form of Hb found in this study is hemozoin in infected cells. In infected erythorocytes, parasites consume Hbs and then release toxic free heme ($\alpha$-hematin, ferriprotoporphyrin IX). The hemes are sequestered via this linkage into an insoluble product, providing a unique way for the malaria parasite to detoxify the free heme which can be polymerized into $\mathrm{\beta }$-hematin in the form of an insoluble microcrystalline material called malaria pigment or hemozoin.^18–20 The spectral signatures of the hemozoin have been studied by conventional spectroscopic techniques such as Fourier transform infrared and Raman spectroscopy.^20–22 These conventional spectroscopic studies are based on cuvette measurements of pure substances extracted from the cells. On the other hand, neither sample extraction nor a priori knowledge of the content of substances is necessary for identifying the substances by hyperspectral imaging and analysis.

In hyperspectral data analysis, the spectrum of a pixel $(x,y)$ in the hyperspectral data cube is treated as a linear superposition of multiple spectra of “pure” individual substances called “endmembers.” In absorption-based microscopy, the endmembers are absorption spectra of intracellular molecular substances. The endmembers in this study are calculated using the sequential maximum angle convex cone (SMACC) algorithm of the Environment for Visualizing Images (ENVI, Exelis, Boulder, Colorado) image analysis software.^23,24 In short, the SMACC algorithm treats the endmembers as spectral basis vectors in the $N$-dimensional hyperspectral space. Ideally, each basis vector or eigenspectrum would correspond to each pure substance so that the spectrum, $H$, for each pixel is decomposed into a linear combination of multiple endmembers as

From the search of basis vectors in the data cube of single erythrocytes, the first six dominant absorption spectral endmembers are displayed in Fig. 2(a). Note that dominant extracted endmembers have the same spectral signatures as those of the well-known absorption spectra of oxy-Hb (red), hemozoin (green), and scattering (black) described in the literature elsewhere,^8,25,26 confirming that the SMACC algorithm successfully extracted the key absorption spectra of major components in normal and infected erythrocytes. Under the illumination with a high numerical aperture objective lens, the wavelength-dependent back scattering substantially contributes to the transmittance signal. In label-free imaging, differentiation of scattering signature is challenging, but the SMACC analysis on the data cubes effectively extracts an endmember corresponding to the scattering (black) from cell membranes and subcellular structures.

Fig. 2

Endmembers and their abundance maps. (a) The six most dominant endmembers corresponding to substances including oxy-Hb, hemozoin, and scattering. (b–g) Abundance maps of all substances show the spatial distribution of each signature spectrum. (b), (c), (d), (e), (f), and (g) correspond to back scattering, oxy-Hb, unknown substance, metHb, hemozoin, and unknown substance, respectively. In (f), a Plasmodium falciparum-infected cell, near the center of the field, contains hemozoin crystals formed by the malaria parasite inside the cytoplasm. The abundance of each endmember was calculated and is displayed in each panel in a gray scale where the label in each panel corresponds to the endmember shown in Fig. 2(a). The gray scale represents the spectral angle difference in radians where a larger value (white) corresponds to a smaller angle between the eigenstate vectors being evaluated and a closer match between the reference and the unknown pixel.

For further validation of the endmember extraction algorithm, the abundance of each endmember in each pixel is calculated. For this report, the two-dimensional (2-D) abundance images of these endmembers are displayed in Figs. 2(b)–2(g). The label for each panel corresponds to each endmember spectrum shown in Fig. 2(a). The gray scale of the images represents the spectral angle difference between the measured spectrum and the endmember, where the white gray scale corresponds to a smaller angle or a closer match between them. From the cell pathology interpretation, we confirmed that each abundance map matches to each sequestered region with the expected dominant substance (e.g., scattering, oxy-Hb, and hemozoin). For instance, Fig. 2(c) shows the regions of abundant oxy-Hb spectra that coincide with the cytoplasmic region of oxy-Hb-rich erythrocytes with a well-preserved biconcave shape. On the other hand, Fig. 2(b) of the scattering map confirms that the cytoplasm of uninfected cells exhibit no noticeable scattering (i.e., dark in the cytoplasm), while an infected cell (located near the center of the image) with a nonbiconcave shape shows a strong scattering signature (bright in the cytoplasm) due to complex subcellular structures induced by parasite infection.²⁷ The same cell shows a dark spot in the middle, shown in Fig. 2(f), and this proved to be localized hemozoin molecules from the abundance map of the hemozoin endmember.

For oxy-Hb or scattering, the gray scale image in Figs. 2(b) or 2(c) in addition to the corresponding endmembers in Fig. 2(a) in the third dimension constructs a 3-D $(x,y,\lambda )$ abundance data cube. In the next section, these abundance data cubes are reproduced by the HIP system to produce digital cell phantoms representing spectral and functional characteristics of specific substances (oxy-Hb and scattering) of the single cells. For the purpose of testing a transmission mode imaging system, projection of raw transmission data would be necessary. However, the processed absorption image data cube was used for projection for the development of standards suited for an imaging system capable of real-time image conversion from transmittance contrast to absorption.

3.

Hyperspectral Image Projector

Reproduction of a data cube of the abundance map with the associated endmember spectrum requires pixel-by-pixel spectral and spatial control to build a spectrally and spatially resolved 3-D data set. To this end, our HIP system illustrated in Fig. 3 uses two LCoS SLMs (Boulder Nonlinear Systems, Boulder, Colorado) integrated with a series of polarization optics.²⁸ In brief, a spectral SLM with $(512\times 512)\text{pixels}$ controls the spectrum and intensity of the light in concert while a second spatial SLM of the same kind replicates an abundance image by pixel-by-pixel intensity manipulation of the light spectrally controlled by the spectral SLM. A camera shown in the figure collects data cubes of the HIP-produced data with known spatially resolved radiometric and spectral information. Ultimately, an imager sensor under test would be placed in the same position.

Fig. 3

A hyperspectral image projector based on two-liquid crystal on silicon spatial light modulators (SLMs). PWGP stands for patterned wide grid polarizer. SLM stands for spatial light modulator. See text for further description.

In the spectral SLM, polarized white light is dispersed by a grating and projected onto the front surface of the SLM, where each wavelength band is matched to each column of the 2-D array pixels in the SLM. The intensity at each wavelength bandwidth is controlled by phase retardance at the polarization state of the reflected light, which results in the maximum transmission through a patterned wire grid polarizer (PWGP) (Moxtek, Orem, Utah) 1. This transmitted light is guided through a beam homogenizer and is projected onto the surface of the spatial SLM. Then the retardance or polarization state of the reflected light off each pixel in the spatial SLM is adjusted so that the transmittance through a PWGP 2 forms a final image where the resulting reflected intensity from each pixel of the 2-D array is proportional to the gray level of the abundance image for each endmember. Here, the gray scale represents the spectral angle difference in radians where a larger value (white) corresponds to a smaller angle between the eigenstate vectors being evaluated and a closer match between the reference and the unknown pixel.

Precise spectral and intensity control by the spectral SLM require accurate calibration of the column position versus wavelength as well as of bias voltage versus reflectance intensity. The total reflectance intensities of the white light collected from the spectral SLM under the change of bias voltage across the LC layer are shown in Fig. 4(a). The phase retardance in an LCoS SLM is controlled by the orientation of the LC molecules within the layer, which is determined by the bias voltage applied across the layer. The reflectance through the LC layer is determined by $R(\lambda )={\sin }^2[\delta /2\pi ]$. Here, the phase retardance, $\delta$, is defined as $\delta =2\pi \mathrm{\Delta }n(\lambda )d/\lambda$, where $\mathrm{\Delta }n(\lambda )$ is the wavelength-dependent birefringence and $d$ is the thickness of the LC layer.¹³

Fig. 4

Calibration of the SLM. (a) Spectral change with the variation of bias voltage across the LC layer of the device. The scale of the bit numbers $[0\text{-}{2}^{15}]$ in the legend is proportional to the bias voltage range of [0 to 5 V]. (b) and (c) are bit numbers (bias voltages) versus reflectance intensities measured from spatial and spectral SLM devices, respectively. Poorer extinction in the spatial device is due to stray light of the broad-band projection. (d) Spectra of the total reflected light collected from the spectral SLM with stepwise wavelength change. Each spectrum corresponds to the reflectance light spectra from a band of five contiguous columns in an “on” state as the band in the “on” state shifting from the second column to the 501st. (e) The wavelength peak of each spectrum in (d) versus the center positions of each corresponding five contiguous column bands in an “on” state.

A pixel-by-pixel bias control is possible in the LCoS SLM; therefore, control of the reflectance at a specific wavelength is achieved by controlling the bias voltages to pixels onto which a specific wavelength of the light is projected. For instance, to achieve monochromatic reflected light from the spectral SLM, different bias voltages are addressed to different pixels so that the reflectance intensities of the pixels in only one column are maximized (“on” state) and with minimized intensities from all the other columns (“off” state). Results in Figs. 4(b) and 4(c) are the peak intensities of the narrow band light spectra versus bias voltages applied to the spatial and spectral SLM, respectively. All the narrow (6 to 9 nm full width of the half maximum) band spectra reflected off the spectral SLM at different wavelengths are displayed in Fig. 4(d). To obtain this plot, pixels of five contiguous columns are maintained in an “on” state and all the others in an “off” state, while the position of the five column band is changed stepwise across the entire device. Polynomial functions of the fifth order to fit the data in Fig. 4(b) determined bias voltages for the desired reflectance intensities. By fitting the peak wavelength for each spectrum with the Gaussian function, Fig. 4(e) presents the wavelength peak of each spectrum in Fig. 4(d) versus the center positions of each of the corresponding five contiguous column bands in an “on” state. The linear relation implies even dispersion and projection of the white light across the spectral SLM.

In phase-controlled SLM devices, a 2-D image is propagated through a series of polarization optics; therefore, spherical or paraxial aberration through the optical train may result in spatial and spectral distortion due to refractive index mismatch.²⁹ To quality check the image projected by the spatial SLM, optical aberration was evaluated with a checkerboard pattern created using the lowest and the highest gray scales among the 15 bit levels. To replicate the spatial contrast of the original checkerboard pattern, the bias voltage versus the reflectance intensity relation from the previous section was used to address the bias voltages of the pixels of the SLM. In the spectral SLM, the white light illumination is reflected with all the pixels in an on state. In the final projected pattern presented in Fig. 5(a), there is no noticeable distortion, confirming there are no significant aberrations. Another important parameter to consider is the magnification of the projection. When an original image with the size ($Lo\times Lo$) and the pixel size of ($No\times No$) is projected by a spatial SLM of ($Ns\times Ns$) pixels, the Nyquist limit limits the final projected image size to $0.5(Ns/No)\times Lo=0.5\times (512/512)\times Lo=0.5\times Lo$ to maintain the spatial frequency information in the original image. Figure 5(b) is a projected image of approximately the quarter ($425\mu \mathrm{m}\times 425\mu \mathrm{m}$) area of a 1951 US Air Force resolution target which was obtained by a $20\times$ (0.4 NA) achromatic objective lens, showing a spatial resolution of approximately $1\mu \mathrm{m}$. The intensity profile in (c) across the lines shown in panel (b) resolves the line features with the full width of the half maximum of $\le 1\mu \mathrm{m}$ estimated from the boundary of the smallest line feature.

Fig. 5

(a) A projected checkerboard pattern directly imaged from the spatial SLM. Uneven distribution in the image intensity is due to incomplete collimation of the illumination, and this artifact may easily be removed by a better collimator, spatial compensation of phase retardance across the SLM, or a postprocessing intensity flattening algorithm. (b) A $425\mu \mathrm{m}\times 425\mu \mathrm{m}$ image of the partial area of a 1951 United States Air Force resolution test chart to evaluate the spatial frequency in the final projected image. (c) The intensity profile along the line shown in image panel (b) with resolved line features.

In the LCoS SLM HIP system, a series of images projected at different wavelengths builds a data cube. A desired light intensity at each band is achieved by applying bias voltages to the pixels of the spectral SLM. Here, the voltages were calculated from the endmember spectrum so that the intensity from each column of the SLM array is proportional to the intensity of the endmember spectrum at each corresponding wavelength. This method allows for accurate control of the intensity at each wavelength in the final projected data cube as the spectral SLM selectively reflects the light of a band of wavelengths one at a time. In the spatial SLM, a 2-D gray scale abundance map at each wavelength is used so that the retardance or polarization state of the reflected monochromatic light off of each pixel in the SLM is adjusted and its intensity after PWGP 2 is proportional to the gray level of the abundance image. A collection of 2-D images projected by the same manner at different wavelengths forms a data cube with a reproduced endmember spectrum in the third dimension. From the band-by-band projections in the projected data cube with stepwise wavelength change from 400 to 700 nm, the average intensity through the projected images was calculated for each pixel, and the resulting images of the back scattering, oxy-Hb, and hemozoin endmember are displayed in Figs. 6(a), 6(b), and 6(c), respectively. These projected images are good replications of the original gray scale images of the corresponding endmember abundance maps shown in Figs. 2(b), 2(c), and 2(f), respectively. To check the quality of the projected oxy-Hb data cube, an extracted spectrum from a cell cytoplasm region pointed with an arrow in the projected oxy-Hb image is presented in Fig. 6(d). The displayed spectrum is similar within a $3\sigma$ error bound to the one plotted in Fig. 2(a) (red spectrum) from an abundance image of oxy-Hb as their overall shapes are the same. Furthermore, the two local maxima, which are characteristic of the absorption spectrum of oxy-Hb molecules, are also evident in Fig. 6(d).

Fig. 6

Projected images of the abundance maps of (a) scattering, (b) oxy-Hb, and (c) hemozoin endmembers. From a band-by-band projected data cube with stepwise wavelength changes from 400 to 700 nm, the intensities through the projected images are averaged for each pixel for these images. These projected images are good replica within $3\sigma$ error bound of the original gray scale images shown in Figs. 2(b), 2(c), and 2(f), respectively. Note that the frame of the spatial SLM device is shown around the edge of the images. Extracted spectra of oxy-Hb from a cytoplasm region in the region of interest pointed with an arrow in the projected oxy-Hb image (b), confirming reproduction of the oxy-Hb spectrum shown in Fig. 2(a). (e) First derivative curves of the oxy-Hb endmember from Fig. 2(a) (red), of the mean spectrum from panel (d) (black solid line), and of the absorption spectrum of oxy-Hb in bulk (green³⁰) are compared. The error bars indicate the resolution of a spectrometer used to determine column versus spectral peak in Fig. 4(e).

For quantitative evaluation of the projected data cube obtained by the HIP system, a first derivative test is applied to the absorption spectrum of oxy-Hb since local maxima and minima are easily found. Before evaluating the HIP system, the test is applied to the spectra of the original endmember data cube [black curve in Fig. 6(e)] and of a bulk oxy-Hb solution [green in Fig. 6(e)] reported by the Oregon Medical Laser Center (OMLC) elsewhere.³⁰ From this analysis, we confirm that both peak positions in the oxy-Hb absorption from our original are in good agreement with those from the OMLC’s results indicating that the wavelength calibration of our hyperspectral imager is consistent with the OMLCs. A precise reproduction of the unique spectral peaks of the original endmember is essential in building a reference data cube for the calibration and validation of the wavelength scale of the image sensor under test. A first derivative curve of the oxy-Hb absorption spectrum [blue in Fig. 6(e)] extracted from the projected oxy-Hb data cube also shows the local peak positions of the spectrum are well reproduced in the projected data cube. The resemblance of these two derivative curves further confirms that the details of the spectral features near these local maxima are similar to each other. A good agreement here confirms that the wavelength calibration of the imaging sensor was adequate to reproduce the spectral characteristics of the original data cube.

Precise radiometric and spectral information in the HIP-produced data cube as an input allows for the calibration of image sensors. For instance, a subtle discrepancy in the absorption spectrum Fig. 6(d) from the original spectrum from the same region of interest allows for the calculation of spectral responsivity and its spatial distribution across the 2-D image sensor. A systematic analysis of various error sources involving both image acquisition and projection would enable a spatially resolved error analysis of the image senor as well. Further, quantitative analysis with a more complex data cube with spatially varying spectral features with systematic characterizations of error sources will enable complete spatially resolved spectral calibration of the imaging sensors. A rigorous model to quantity system-performance metrics evaluated on the basis of statistical description of measurements, similar to a previous work,³¹ will be beneficial.

4.

Conclusion

An absorption-based wide-field hyperspectral microscope with the aid of SMACC analysis identified the spatial distribution of light-absorbing intracellular oxy-Hb molecules in single erythrocytes. A data cube for oxy-Hb was constructed in 2-D endmember abundance maps along with the endmember spectra across the wavelength in the third dimension. An HIP system employing a pair of LCoS SLMs was demonstrated to reproduce this data cube with spatial and spectral information close to the initial data cube for oxy-Hb distributions. Our approach demonstrates a procedure to generate image data cubes indistinguishable from the real scene of single erythrocytes which can be used for further analysis without repeating the sample preparation. Application of our technique with clinical scenes may eliminate costly repetition of similar clinical data or scenes, allowing valuable clinical data readily accessible to nonclinical laboratories for the test of imaging devices for measurement quality, performance consistency, and interlaboratory comparisons. In particular, these unique data cubes may be used as standards to test spectroscopic images such as those from hyperspectral imaging systems.