1.

Introduction

High-speed and high-resolution imaging techniques have been long sought for material metrology and biological structure observation. Such examples include the inspection of large area subwavelength structures and optical metasurfaces widely used in integrated photonics,^1,2 monitoring fast semiconductor wet etching process,³ microdroplet evaporation dynamics,⁴ observation of live cell morphology, fast dynamics in a large cell population,^5,6 and tracking of high-speed cell motions.^7,8 Conventionally used techniques for material metrology, including scanning electron microscopy (SEM) and atomic force microscopy (AFM), are limited in throughput and speed. In addition, once the structures are encapsulated in dielectric claddings, which are the standard practice for photonic devices, SEM or AFM characterization becomes impractical, as the cladding often obscures the electron imaging contrast between core and cladding and prohibits physical contact of the AFM probe with the structure. On the other hand, the challenge for light microscopy is that many subwavelength structures and living cells share the common feature of being transparent and thin. Under conventional bright-field microscopes, those structures cannot be well resolved due to their weak light absorption. With chemical staining, fluorescence microscopy can be applied to image these structures. However, for certain live cell imaging and most material metrology applications, label-free or noninvasive imaging is favored for minimizing sample perturbation. Furthermore, fluorescence-based imaging techniques are usually slow, thus limiting their use in high-speed imaging applications.

Phase is an intrinsic property of light that carries the object structural information. Quantitative phase microscopy (QPM) can measure phase precisely with high contrast, while being non-invasive and label-free.⁹ For reflective or homogenous transparent samples, the measured phase maps can be transformed into corresponding surface height profiles with nanometer precision. Thus QPM has been increasingly applied to semiconductor wafer defects detection, semiconductors etching monitoring in material science,^10,11 and cell growth, mechanics, and metabolism modeling in biological imaging.^12,13 However, the lateral resolution of plane wave illumination coherent imaging techniques is limited to $\lambda /\mathrm{NA}$ by diffraction. Several computational imaging methods using multiple image recordings have been developed to improve the lateral resolution in coherent imaging, including lensfree microscopy,^14,15 Fourier ptychographic microscopy,^16,17 and synthetic aperture microscopy (SAM).^18–25 Among them, SAM stands out as a simple and robust technique. In SAM, the complex scattered fields, containing different object spatial spectrum sections obtained from different illumination angles, can be retrieved through digital holography and synthesized to obtain an extended object spatial spectrum. Sample rotation,²⁰ galvano-mirror-based beam scanning,^18,26 and illumination source shifting²⁷ have been implemented to realize SAM. To capture multiple illumination angles simultaneously, angle-multiplexed synthetic aperture techniques (such as those based on a vertical cavity surface-emitting laser array,²⁸ subpulses,²⁹ or a spatial light modulator³⁰) and spatially multiplexed synthetic aperture techniques³¹ have also been proposed. Despite all of the advances, the smallest periodic structure resolved by SAM to date was 460 nm³⁰ and the highest imaging speed reported thus far was limited to $<20$ frames per second (fps). Therefore, there is a great need to improve the imaging resolution and speed to meet emerging application demands in material science and biology, including but not limited to profiling large area subwavelength structures and quantifying fast subcellular dynamics.

To address these issues, we propose a light microscopy method, termed high spatial and temporal resolution synthetic aperture phase microscopy (HISTR-SAPM). In HISTR-SAPM, digital micromirror devices (DMDs) are implemented to achieve rapid and efficient angle scanning of the illumination beam, and an off-axis interferometry system is built for measuring complex sample fields. By fully exploring the relationship between captured images under adjacent angles, a synthetic aperture image reconstruction method that precisely models the image formation physics is developed to retrieve high-resolution amplitude and phase images. With a high-speed camera and a CoaXPress data transfer interface, image acquisition speed of $>5\mathrm{kHz}$ is realized. The versatility of HISTR-SAPM is demonstrated via high-speed and high-resolution imaging of subwavelength photonics structures, red blood cells (RBCs), and more complex cells (e.g., 3T3 cells, COS-7 cells, and HeLa cells) and their subcellular dynamics.

2.

Materials and Methods

A sketch of the experimental setup of HISTR-SAPM is shown in Fig. 1(a). A 532-nm laser beam is split into two arms by a $1\times 2$ single-mode fiber coupler with one output arm for sample illumination and the other as the reference beam. In the illumination path, DMD1 (DLP Light Crafter 9000, Texas Instruments Inc.) is placed conjugate to the image plane, and DMD2 (DLP Light Crafter 3000, Texas Instruments Inc.) is placed at the Fourier plane of DMD1. The size of the micromirrors confines the maximum scanning angle to be around 1 deg immediately after the DMD1 plane. To fully utilize the aperture of the objective lens, two 4-f systems, starting from DMD1, consisting of the scanning objective lens (OL1: Zeiss, $40\times /1.3$, oil immersion) and three other lenses, are used to amplify the scan angle range on the sample. After passing through the sample, the scattered light is collected by an imaging objective lens (OL2), which has the same specifications as OL1. After the tube lens, the sample beam is combined with the reference beam through a beam splitter to form an interferogram at the camera plane.

Fig. 1

(a) Schematic of the HISTR-SAPM setup. DMD1 and DMD2, digital micromirror devices; M1 and M2, mirrors; OL1 and OL2, objective lenses; and BS, beam splitter. (b) (i)–(iii) are the raw interferograms under three different illumination angles, and (iv)–(vi) are their corresponding spatial spectra.

A series of binary Lee holograms³² are added to DMD1 for generating different illumination angles on the sample (Note S1 in the Supplemental Materials). However, the binary hologram generates several parasitic diffraction orders besides the desired one. This diffraction noise can be mostly eliminated by a physical annular aperture that is designed to only select the desired diffraction order.³² As the physical annulus is difficult to align and cannot be modified, another approach to get rid of the diffraction noise orders is using a time-multiplexing illumination scheme.³³ However, the 8-bit display mode does not allow for high-speed pattern refresh. To achieve a more flexible filtering scheme without compromising the DMD refresh rate, we added DMD2 at the Fourier plane of DMD1 to make a tunable spatial filter. We synchronized the two DMDs to dynamically select the first diffraction order and block all of the others. Each time when DMD1 displays a different Lee hologram, the corresponding filtering pixels on DMD2 are turned on, as shown in the inset of Fig. 1(a). In this way, only the desired first diffraction order is directed to the subsequent optical system. It interferes with the reference beam and forms an interferogram at the camera plane. Figure 1(b) shows three representative interferograms and their corresponding spatial spectra, measured from a custom-made USAF resolution target as the sample. The first column [(i) and (iv)] corresponds to the pair under normal sample illumination angle; the enlarged interferogram in the inset shows a clean fringe pattern. In the spatial frequency domain, the sample spatial spectrum, enclosed by the dotted circle region, is shifted by ${\mathbf{k}}_{\text{R}\perp }$ due to fringe modulation. The perpendicular subscript indicates the wave vector component perpendicular to the optical axis. The radius of the dotted circle is ${\mathrm{NA}}_{\det }/\lambda$, where ${\mathrm{NA}}_{\det }$ is the numerical aperture (NA) of the imaging objective lens OL2. The next two columns are the interferogram and spatial-spectrum pairs, corresponding to two different oblique sample illuminations. The change of fringe orientation and period reflects the change of the sample illumination angle, and we use ${\mathbf{k}}_{\mathbf{S}\perp }$ to denote the sample illumination beam wavevector, as indicated in panel (v), Figs. 1(b). Galvano-mirrors can also be used for fast angle scanning when paired with a high-speed camera. However, when scanning at high speed, there could be mechanical vibrations that can induce angle-scanning instabilities. Also, it is not easy to simultaneously focus two galvano-mirrors into the specimen plane for scanning angles in two orthogonal directions. Compared to using galvano-mirrors, DMDs do not introduce any mechanical vibrations and allow for a better focusing to the specimen plane. As DMDs provide more flexibility in changing the display patterns, multiple scanning angles can be realized simultaneously by designing a complex displayed pattern, e.g., overlap several hologram patterns.³⁴ This can further enable several folds of imaging speed improvement that is not possible with galvano-mirror-based angle scanning. Therefore, to the best of our knowledge, DMDs are better options than galvano-mirrors in the context of high-speed imaging.

For each HISTR-SAPM image reconstruction, we captured one interferogram under normal illumination angle and 39 interferograms at different oblique sample illuminations, equally spaced in the azimuthal direction, while maintaining a maximum elevation angle of $\sim 40.5\mathrm{deg}$. The spatial spectrum of the USAF resolution target due to normal illumination is shown in the white dotted circle in Fig. 2(a), whereas the dotted gold and orange circle regions show the detected spatial spectrum due to two representative oblique illumination angles. A physical model that fully describes the synthetic reconstruction process is detailed in Note S2 in the Supplemental Materials. After aperture synthesis, an expanded sample spatial spectrum with a radius of ${\mathrm{NA}}_{\det }/\lambda +{\mathrm{NA}}_{\text{scan}}/\lambda$ is achieved. ${\mathrm{NA}}_{\text{scan}}$, determined by the maximum scan angle of the illumination beam, is a fraction of the NA of objective lens OL1. Finally, an inverse Fourier transform is performed over the synthesized spectrum to produce the complex field of the resolution target, from which a high-resolution amplitude map [Fig. 2(b)] and a phase map [Fig. 2(c)] are obtained. In our system, we have determined that ${\mathrm{NA}}_{\det }/\lambda +{\mathrm{NA}}_{\text{scan}}/\lambda \sim 1.76{\mathrm{NA}}_{\det }/\lambda$, from which we inferred a lateral resolution of 233 nm, which is almost two times smaller than the plane wave illumination coherent imaging limit. The lateral resolution, inferred from the edge response of a measured line structure, also agrees with this value (Fig. S1 in the Supplemental Materials). Finally, an inverse Fourier transform is performed to produce the complex sample field, from which a high-resolution amplitude map [Fig. 2(b)] and a phase map [Fig. 2(c)] are obtained. In the resolution target, the smallest structure linewidth is 220 nm (note that our group and element number do not follow the standard 1951 USAF resolution target), which is well-resolved in the aperture synthesized phase map. Since the sample is mostly transparent and very thin, the contrast in the intensity map is not as obvious as in the phase map. We note that (i) our phase imaging quality is not affected by laser speckle (Fig. S2 in the Supplemental Materials), and, (ii) as our spectrum synthesis process does not require much overlapping, we can also significantly reduce the number of illumination angles without compromising image quality to achieve even higher imaging speed (Fig. S3 in the Supplemental Materials). The accuracy of the phase reconstruction is demonstrated by imaging $10\mu \mathrm{m}$ polystyrene beads (Fig. S4 in the Supplemental Materials).

Fig. 2

(a) The spatial spectrum synthesis process in HISTR-SAPM. The dotted circles correspond to the frequency passband in Fig. 1(b). (b), (c) High-resolution amplitude and phase reconstruction, respectively.

3.

Results

In the following, we demonstrate the application of HISTR-SAPM for characterizing subwavelength grating structures, which have been increasingly used in integrated photonic devices. The subwavelength grating structure has an original design, as shown in Fig. 3(a), and its periodic structure under SEM is shown in Fig. 3(b). This structure has a period of 330 nm with a linewidth and gap width of 198 nm and 132 nm, respectively (for sample preparation refer to Note S3 in the Supplemental Materials). As the structure features are much smaller than the coherent imaging diffraction limit, a conventional QPM system is unable to resolve the features to correctly retrieve the height map. For example, Fig. 3(c) shows a height profile measurement of the structure obtained with the conventional QPM, where structures are poorly resolved. In contrast, the height map, reconstructed with HISTR-SAPM, as shown in Fig. 3(d), gives a clear view of the grating structure. An enlarged side view shows each individual grating structure and their height profiles. In Fig. 3(e), we plot the height profile (blue curve) for the structures across the white solid line in Fig. 3(d). An SEM intensity profile of the same sample region is also plotted alongside (orange curve) to show the excellent agreement on the lateral dimension measurement. Note that as the SEM image does not provide the height profile, the dimples across the line structures are not associated with height variations. We have also reconstructed the sample phase map with only 13 sample scanning angles, i.e., 13 ms total image acquisition time when operating the camera (CP90-25-M-72, Optronis) at 1000 fps, and the retrieved height map retains most of the details (Fig. S3 in the Supplemental Materials).

Fig. 3

Imaging of a custom-made subwavelength grating structure: (a) the original design of the structure; (b) a portion of the structure imaged under SEM; (c) the height map retrieved using conventional QPM; (d) the height map reconstructed using HISTR-SAPM; and (e) the line profiles along the white lines in (b) and (d).

Quantifying live cell dynamics is important to understand many physiological processes at the single cell level. For example, RBC membrane height fluctuation maps can be used to derive RBC stiffness parameters for assessing their viability.¹² To demonstrate the high-speed imaging capability of HISTR-SAPM, we have quantified membrane dynamics of human RBCs. A spherocyte, as shown in Fig. 4, was imaged by recording 1000 consecutive interferograms at a frame rate of 1000 fps (the time-lapse video available as Video S1). Only 16 interferograms corresponding to 16 scanning angles were used for each synthetic aperture phase reconstruction; therefore, the effective imaging frame rate of the membrane fluctuations is 63 fps. Time lapse of the height fluctuation for three selected spots is also plotted: the cell center region (A), cell outer region (B), and background (C). From the height fluctuation standard deviation (STD) values, we found that the outer region (B) experiences more fluctuations than the center region (119.9 nm versus 62.4 nm; the values are also much higher than the background STD of 28.5 nm), which is consistent with previous observations.^9,12 This demonstration suggests that our technique can be applied to observe fast nanoscale morphological changes in biological samples.

Fig. 4

RBC membrane height fluctuation over a time period of 1 s (Video S1): (A) cell center region; (B) cell outer region; and (C) background region. Time-lapse video of RBC membrane fluctuation (Video S1, MP4, 5.87 MB [URL: https://doi.org/10.1117/1.AP.2.6.065002.1]).

With the $2\times$ improved spatial resolution and the high imaging contrast, we are also able to image more complexed cells. The phase maps of a COS-7 cell and a HeLa cell under normal illumination [Figs. 5(a) and 5(d)] and after HISTR-SAPM reconstruction [Figs. 5(b) and 5(e)] are shown below. The noise and speckle in the images under normal illumination are mostly removed after reconstruction, making the cell boundaries and subcellar structures more visible (e.g., small organelles in the cytoplasm of the COS-7 cell and nucleus envelope and nucleoli of the HeLa cell). With such a high dynamic range achieved in the synthetic phase map, we also calculated the phase gradient map in Figs. 5(c) and 5(f). Here the interferograms were captured with a faster camera (Fastcam SA-X2, Photron) at a frame rate of 5000 fps, while DMD2 was replaced with the DLP Light Crafter 9000 model to match with the speed of the camera. For each reconstruction, 49 consecutive interferograms corresponding to 49 scanning angles are used, i.e., over 100 fps imaging speed (note that with a reduced number of angles, the imaging speed can be more than doubled to achieve $<5\mathrm{ms}$ temporal resolution).

Fig. 5

Observation of subcellular structures in unlabeled living cells: (a), (d) the phase maps of a COS-7 and a HeLa cell under normal illumination; (b), (e) the phase maps reconstructed with HISTR-SAPM for the cells in (a), (d); and (c), (f) the phase gradient maps obtained from (b), (e).

To demonstrate the capability of imaging fast dynamics of subcellular structures, we exposed 3T3 cells to a drop of 0.5% acetic acid solution and immediately recorded a time lapse of 7500 interferograms for a total of 1.50 s while the illumination angle is scanning. Previous research has shown that exposure to acetic acid can immediately lead to an increase in the refractive index inhomogeneity in the cell nuclear region.³⁵ After processing all of the interferograms, we obtained a time lapse of synthetic phase maps with a temporal resolution of 10 ms (the time-lapse video available as Video S2). As expected in the video, immediately following the exposure to acetic acid, there was a dramatic phase value increase in the cell nuclear region, and the nuclear membrane became more prominent, while the cytoplasm region stayed almost the same. Three representative frames at $t=0$, 0.75, and 1.5 s are shown in Figs. 6(a)–6(c). Also the time-lapse curves of average phase variation in three representative regions [indicated by black boxes in Fig. 6(c)] are plotted in Fig. 6(d), from which we can infer an obvious increase in phase value in the nucleus region over time.

Fig. 6

Observation of living 3T3 cell dynamics after exposure to acetic acid (Video S2). (a)–(c) The representative phase map frames reconstructed with HISTR-SAPM for the 3T3 cell during exposure to acetic acid. (d) The time-lapse curves showing phase evolution in the nucleus, cytoplasm, and background over time. Time-lapse video of 3T3 cell dynamics after exposure to acetic acid (Video S2, MP4, 6.19 MB [URL: https://doi.org/10.1117/1.AP.2.6.065002.2]).

4.

Conclusion

The proposed DMD-based HISTR-SAPM has broken the coherent imaging lateral resolution by a factor of 2 while achieving millisecond level temporal resolution. Using HISTR-SAPM, we have performed surface profiling of material structures that have lateral features as small as 132 nm, quantified RBC membrane fluctuations in millisecond timescale, and observed nucleated cell structures and their dynamic changes during exposure to chemicals. Considering the flexible patterning capability of DMD for generating multiple illumination angles simultaneously and the availability of higher speed cameras (e.g., $>\mathrm{10,000}\mathrm{fps}$), our HISTR-SAPM can be further developed to achieve an imaging speed several times higher. The same system can also be used to map the 3D refractive index distributions of cells by implementing a 3D image reconstruction algorithm. The removal of speckle enables the resolving power that can be compared with white-light diffraction tomography.³⁶ By replacing the detection objective (OL2) with a lower NA one, our system can potentially achieve higher imaging speed compared with Fourier ptychographic microscopy. We envision that this high-speed imaging technique will enable more potential applications in biology studies and material metrology, such as studying live cell motions and interactions with other cells and monitoring material manufacturing process in real time for quality control.