Untrained neural network enhances the resolution of structured illumination microscopy under strong background and noise levels

Yu He; Yunhua Yao; Yilin He; Zhengqi Huang; Dalong Qi; Chonglei Zhang; Xiaoshuai Huang; Kebin Shi; Pengpeng Ding; Chengzhi Jin; Lianzhong Deng; Zhenrong Sun; Xiaocong Yuan; Shian Zhang

doi:10.1117/1.APN.2.4.046005

4 July 2023 Untrained neural network enhances the resolution of structured illumination microscopy under strong background and noise levels

Yu He, Yunhua Yao, Yilin He, Zhengqi Huang, Dalong Qi, Chonglei Zhang, Xiaoshuai Huang, Kebin Shi, Pengpeng Ding, Chengzhi Jin, Lianzhong Deng, Zhenrong Sun, Xiaocong Yuan, Shian Zhang

Author Affiliations +

Advanced Photonics Nexus, Vol. 2, Issue 4, 046005 (July 2023). https://doi.org/10.1117/1.APN.2.4.046005

Abstract

Structured illumination microscopy (SIM) has been widely applied in the superresolution imaging of subcellular dynamics in live cells. Higher spatial resolution is expected for the observation of finer structures. However, further increasing spatial resolution in SIM under the condition of strong background and noise levels remains challenging. Here, we report a method to achieve deep resolution enhancement of SIM by combining an untrained neural network with an alternating direction method of multipliers (ADMM) framework, i.e., ADMM-DRE-SIM. By exploiting the implicit image priors in the neural network and the Hessian prior in the ADMM framework associated with the optical transfer model of SIM, ADMM-DRE-SIM can further realize the spatial frequency extension without the requirement of training datasets. Moreover, an image degradation model containing the convolution with equivalent point spread function of SIM and additional background map is utilized to suppress the strong background while keeping the structure fidelity. Experimental results by imaging tubulins and actins show that ADMM-DRE-SIM can obtain the resolution enhancement by a factor of ∼1.6 compared to conventional SIM, evidencing the promising applications of ADMM-DRE-SIM in superresolution biomedical imaging.

1. Introduction

Utilizing spatial frequency modulation, structured illumination microscopy (SIM) transfers the normally inaccessible high-frequency information into measured low-frequency images with moiré effect, and thus it can extract the superresolution image by spatial frequency recombination, which breaks through the diffraction limit barrier of the optical microscope.¹ Featuring high imaging speed, low excitation power, and high compatibility for various fluorescent labels, SIM has been widely applied in the superresolution imaging of biological specimens, especially the fine structure of organelles, such as actin cytoskeleton,² mitochondria,³ lysosomes,⁴ and endoplasmic reticulum,⁵^,⁶ and also the dynamics of organelles, such as mitochondrial fission,⁷ internalization of clathrin-coated pits (CCPs),⁸ and tubulin and kinesin dynamics.⁹ However, the resolution of SIM with linear illumination mode is only 2 times over that of wide-field microscopy, which reaches about 100 nm, because the maximal frequency of structured illumination pattern is limited by the diffraction. Thus, SIM is inferior to some other superresolution techniques in spatial resolution, such as stimulated emission depletion microscopy,¹⁰ photoactivated localization microscopy,¹¹ or stochastic optical reconstruction microscopy,¹² which can obtain the resolution up to about 10 nm. By now, many methods have been reported to increase the spatial resolution of SIM. For example, fluorescence saturation was utilized to induce nonlinear structured illumination for higher spatial frequency shift.¹³ Similarly, photoswitching proteins were used as the fluorescent labels for nonlinear SIM with low excitation power.¹⁴ With these methods, the nonlinear SIM with a resolution below 50 nm has been realized, but unfortunately it comes at the expense of reducing the imaging speed. In addition, a sparse deconvolution method (sparse-SIM) was proposed to further obtain the resolution enhancement of SIM while keeping the imaging speed by employing the sparsity and continuity priors of biological structures.¹⁵ Without the modification to the hardware device or acquirement mode of SIM, sparse-SIM achieves almost twofold resolution enhancement compared with conventional SIM. However, as an iterative deconvolution method, sparse-SIM is greatly affected by the artificial selection of multiple parameters, which are used to balance the weights of various priors. Therefore, the image processing in sparse-SIM is time-consuming.

In the experimental acquisition of a fluorescent image, strong background and noise are common issues. These issues may be tolerated in normal microscopy analysis but can lead to severe artifacts in the superresolution image reconstruction, which make it difficult to distinguish the real structures and fake artifacts. The strong background mainly results from the out-of-focus fluorescence and cellular autofluorescence, while the strong noise often occurs in the image acquirement by the camera, especially when the fluorescent photon number is limited by the short exposure time or low excitation power. To further improve the resolution of SIM under strong background and noise levels, here we propose a deep resolution enhancement (DRE) method by an untrained neural network associated with alternating direction method of multipliers (ADMM) framework, termed as ADMM-DRE-SIM. An untrained neural network with implicit priors is utilized as a generator for desired resolution-enhanced image, and an image degradation model containing optical transfer of SIM and additional background map provides the criterion for updating the parameters of the neural network. The untrained neural network is quite convenient to obtain the resolution enhancement of SIM because no training datasets are required (it is rather difficult to experimentally acquire the training datasets containing the SIM images and paired resolution-enhanced images). An iterative framework of ADMM embedded with Hessian prior is used to optimize the network by exploiting the intrinsic continuity of biological structures. Based on these special designs, ADMM-DRE-SIM realizes the resolution enhancement of SIM under the strong background and noise while completely preserving the advantage of SIM with high imaging speed. Compared with conventional SIM, a resolution enhancement by a factor of about 1.6 is experimentally demonstrated by imaging tubulins and actins in cells with ADMM-DRE-SIM. Moreover, ADMM-DRE-SIM shows better performance in the fidelity of biological structure compared to other deconvolution methods, which is key for image analysis, and it will provide a powerful tool for the studies of organelle dynamics.

2. Methods

The resolution of the image acquired in wide-field microscopy is restricted by the optical diffraction limit due to the finite aperture of the objective. To exceed the resolution limitation, a structured illumination pattern is utilized to modulate the sample image with spatial frequency shift in SIM. The recorded image in SIM can be formatted in the frequency domain as

Eq. (1)

D (k) = [S (k) \otimes E (k)] O (k) + N (k) = \frac{S}{2} [E (k) - \frac{m}{2} E (k - p) e^{- i ϕ} - \frac{m}{2} E (k + p) e^{i ϕ}] O (k) + N (k),

where

D (k)

is the spatial spectra of recorded image in SIM,

S (k)

is the structured illumination pattern,

E (k)

is the spatial spectra of fluorescent sample,

O (k)

is the optical transfer function (OTF) of the microscopy system,

N (k)

is the noise during the image acquirement,

S

is the max modulation intensity,

m

is the modulation factor,

p

is the modulation frequency vector, and

ϕ

is the modulation phase. The recorded images in SIM are linear combinations of the origin and frequency-shifted components. Normally, the structured illumination patterns with three-step phase shifts in three directions are required to acquire nine images in total, which are then processed to obtain a superresolution image by separating and recombining the frequency components. Since the max frequency shift in linear SIM is close to the bound of OTF, SIM can only acquire an equivalent OTF twice as large as that of wide-field microscopy. The reconstructed SIM image in the spatial domain can be seen as the convolution of fluorescence distribution and equivalent point spread function (PSF) of SIM. The further resolution enhancement from the reconstructed SIM image is similar to deconvolution microscopy,¹⁶ which increases the contrast and resolution of a wide-field image. However, the resolution enhancement from the SIM image is much more complex. The noise amplification in the image reconstruction of SIM seriously hampers deconvolution, which makes it difficult to converge. Considering the strong background and noise levels, this problem becomes tougher because the serious artifacts and distortion contaminate the actual structures. High-fidelity image reconstruction for the resolution enhancement in SIM is still limited by the amplified noise as well as the strong initial background. Under the condition of strong fluorescent background and noise levels, the reconstructed SIM image in the spatial domain can be written as

Eq. (2)

I^{SIM} = I^{sample} \otimes {PSF}^{SIM} + G + N,

where

I^{SIM}

is the reconstructed SIM image;

I^{sample}

is the fluorescence distribution of the sample; and

{PSF}^{SIM}

is the equivalent PSF of SIM, which considers the cut-off frequency bound of the microscope and the frequency shift of structured illumination patterns;

G

is the background distribution; and

N

is the noise from the image acquirement and reconstruction. It should be noted that

G

and

N

correspond to the background and noise after the image reconstruction of SIM. The background occupies the low spatial frequency components. Therefore, the special frequency characteristic allows the extraction of the background to some extent. A wavelet-based background subtraction was proposed to handle this task.¹⁷^,¹⁸ However, the background subtraction in the low-photon images easily causes structural defects because some structures with the low fluorescence intensity are overwhelmed in the background and noise.

To further improve the resolution of SIM under the strong background and noise levels, we adopt a concept of deep image prior (DIP), which indicates that the structure of a generator network itself can capture lots of statistics priors of low-level image for various tasks instead of learning from training datasets.¹⁹ The untrained neural network incorporated with various physical models has shown to be a well-established tool in solving the image reconstruction of computational imaging methods, such as coherent phase imaging,²⁰ digital in-line holography,²¹^,²² ghost imaging,²³ and SIM.²⁴^,²⁵ Here we achieve the further resolution enhancement of SIM by the untrained neural network combining with the background interrupted optical transfer model of SIM. The implicit priors inside the neural network provide the bridge between the available information in the SIM image and the higher frequency information corresponding to the higher resolution. Meanwhile, the optical transfer model containing the diffraction limit in SIM and the background map is employed to supervise the neural network and guarantee the image fidelity with background suppression. Furthermore, an ADMM framework embedded with Hessian prior is used to optimize the parameters of the neural network with high precision, which is different from the previous work with only an untrained neural network.²⁶

The flow chart of ADMM-DRE-SIM is shown in Fig. 1. The original SIM image $x_{0}$ is sent into the neural network with a U-net structure to acquire a resolution-enhanced image $f_{θ} (x_{0})$ . An image degradation model, containing the convolution with equivalent PSF of SIM and the additional background, is then used to provide the estimation of the recorded image in SIM from the original fluorescence distribution of the sample. In addition, a pixel upsampling has been incorporated in the network, and a downsampling process is correspondingly embedded in the image degradation model, which is used to adjust the magnification of the reconstructed image. The estimation can be written as

Eq. (3)

I^{Estimation} = f_{θ} (x_{0}) \otimes {PSF}^{SIM} + m,

where

f_{θ} (.)

is the neural network with the parameter

θ

to be optimized, and

m

is the estimated background map from the SIM image. The equivalent PSF of SIM is estimated from the optical parameters of the SIM system, considering the cut-off frequency bound of the microscope objective and the frequency shift of structured illumination spatial frequency patterns, which can be expressed by the following equation as

Eq. (4)

{PSF}^{SIM} = {[\frac{2 J_{1} (\frac{2 π m_{f} N A \cdot r}{λ})}{\frac{2 π m_{f} N A \cdot r}{λ}}]}^{2},

where

J_{1}

is the first-order Bessel function,

λ

is the fluorescence wavelength, NA is the numerical aperture of the objective lens,

m_{f}

is the modulation factor used to modulate the equivalent NA after SIM reconstruction,

m_{f} = (1 + k_{f} / k_{c})

,

k_{f}

is the structured illumination frequency,

k_{c}

is the cut-off frequency for acquisition, and

r

is the radial space coordinate of object surface. The additional background map is estimated from the input SIM image with wavelet transform.¹⁷^,¹⁸ A “db6” wavelet base is used, decomposing the signal into the seventh level. Once the estimation is optimized to approach the input SIM image

x_{0}

, the resolution-enhanced image can be acquired. An ADMM framework embedded with a Hessian prior is used to iteratively optimize the parameters of the neural network. Generally, the resolution enhancement of SIM can be converted into searching for the optimal parameters of the neural network that satisfy the minimization of the following question, and is given as

Eq. (5)

x^{*} = f_{θ^{*}} (x_{0}), θ^{*} = \underset{θ}{\arg \min} {‖ f_{θ} (x_{0}) \otimes {PSF}^{SIM} + m - x_{0} ‖}_{2}^{2} + ρ R_{Hessian} [f_{θ} (x_{0})],

where

R_{Hessian}

is the Hessian norm,

ρ

is the regularization parameter, and

{‖ . ‖}_{2}

is the

L_{2}

norm. The Hessian matrix can exploit the continuity of biological structures as the prior knowledge and provide a tool to constrain image restoration and suppress noise, which has shown good performance in the image reconstruction of SIM with structural fidelity and artifact suppression.⁴ Here, the Hessian prior is used to constrain the parameter optimization of the neural network. The Hessian matrix can be expressed as

Eq. (6)

R_{Hessian} (I) = \sum_{i}^{n} {‖ D I ‖}_{1} = {‖ D_{x x} I ‖}_{1} + {‖ D_{y y} I ‖}_{1} + 2 {‖ D_{x y} I ‖}_{1},

where

D_{x x}

,

D_{y y}

, and

D_{x y}

denote the discrete second-order derivatives at the pixel

i

of the image

I

along the three axes of horizontal, vertical, and diagonal directions, respectively. Based on the ADMM framework, Eq. (5) can be transferred into the following problem by introducing the auxiliary variable

v

and is written as

Eq. (7)

\underset{θ, v}{\arg \min} {‖ f_{θ} (x_{0}) \otimes {PSF}^{SIM} + m - x_{0} ‖}_{2}^{2} + ρ R_{Hessian} (v), subject to D f_{θ} (x_{0}) = v .

Fig. 1

Flow chart of ADMM-DRE-SIM for realizing the resolution enhancement of SIM.

By further introducing the auxiliary variable $u$ , Eq. (7) can be divided the minimization into the following subproblems and is given as

Eq. (8)

θ^{k + 1} = \underset{θ}{\arg \min} {‖ f_{θ} (x_{0}) \otimes {PSF}^{SIM} + m - x_{0} ‖}_{2}^{2} + β {‖ D f_{θ} (x_{0}) - (v^{k} - \frac{u^{k}}{β}) ‖}_{2}^{2},

Eq. (9)

v^{k + 1} = \underset{v}{\arg \min ρ} R_{Hessian} (v) + β {‖ v - [D f_{θ^{k + 1}} (x_{0}) + \frac{1}{β} u^{k}] ‖}_{2}^{2},

Eq. (10)

u^{k + 1} = u + β [D f_{θ^{k + 1}} (x_{0}) - v^{k + 1}] .

The subproblem in Eq. (8) is similar to the traditional DIP solution by forcing $D f_{θ} (x_{0})$ to approach $v^{k} - u^{k} / β$ with the neural network; here the parameter $θ$ is optimized by a gradient descent method.²⁷^,²⁸ In Eqs. (9) and (10), the second-order derivatives in different directions are separately processed. By the alternate iterative optimization, a resolution-enhanced image with low noise and background removal can be obtained. It is worth mentioning that ADMM-DRE-SIM can optimize the parameters of the network in a self-supervised mode without the training datasets.

3. Results and Discussion

To demonstrate the resolution enhancement of SIM by ADMM-DRE-SIM, a theoretical simulation is conducted. A simulated image containing the curves with width of 32.5 nm is used as the ground truth. The wide-field image is generated by the convolution of the ground truth and the PSF of a microscope with the NA of 1.2 and fluorescence wavelength of 560 nm. Nine images are simulated by sequential structured illumination and PSF convolution based on the ground truth. A fairSIM algorithm is utilized to recover the superresolution SIM image.²⁹ Then, the SIM image is further processed by ADMM-DRE-SIM to obtain a resolution-enhanced image. All the image processing is performed on a server with Intel i9-10920X, 192 GB of RAM and NVIDIA GeForce RTX3090. The simulation results are shown in Fig. 2, and the corresponding spatial frequency spectra and the intensity profiles along the labeled lines are also provided. As can be seen, only the low-frequency components are preserved in the wide-field image due to the limited OTF of the microscope, which lead to low spatial resolution, and the curves along the labeled line are seriously blurred. The SIM image has a frequency bound twice that of the wide-field image, and the intensity profile of the curves can be distinguished with difficulty. However, the ADMM-DRE-SIM image has a greatly extended frequency bound, which is close to the ground truth, and the adjacent curves can be clearly distinguished.

Fig. 2

Simulation results for the resolution enhancement of SIM by ADMM-DRE-SIM. (a) The ground truth, (b) wide-field, (c) SIM, and (d) ADMM-DRE-SIM images of the simulated structures, associated with the corresponding frequency spectra and intensity distributions along the labeled lines in the insets. The insets are the enlarged views of the selected areas with yellow squares.

In actual experiments, the background and noise are inevitable, and thus the recorded images with structured illumination in SIM will be contaminated. Although SIM can suppress the background to some extent, heavy noise will cause vague artifacts during SIM reconstruction. As for the resolution improvement in noisy conditions, the noise will mislead the optimization direction in iterative calculations. The heavier the noise is, the more effort the algorithm takes to cover the noise. This leads to a bias between the reconstructed result and the real structure, which finally results in serious artifacts. Besides, the more resolution enhancement is conducted, the more sensitive to noise the reconstructed result is. The SIM images involving background and various noise levels are processed to test the resolution enhancement performance by ADMM-DRE-SIM. The simulated results are shown in Fig. 3; here, a uniform background combined with Gaussian noise $σ = 0.1$ , 0.3, 0.5, and Poisson noise mixed with Gaussian noise $σ = 0.5$ are, respectively, added in the recorded image with structured illumination in SIM. For comparison, some traditional deconvolution algorithms are also utilized to process the SIM image, including Wiener deconvolution,³⁰ Richardson–Lucy total variation (RLTV) deconvolution,³¹ and Hessian deconvolution.³² The SIM images are also reconstructed by fairSIM, associated with the resolution-enhanced images by various traditional algorithms and ADMM-DRE-SIM, as shown in Fig. 3(b). Obviously, ADMM-DRE-SIM is able to improve the resolution of SIM and simultaneously suppress the background and noise. In contrast, other deconvolution algorithms bring serious artifacts with slight resolution enhancement. In addition, structural similarity values of the images processed by Wiener, RLTV, Hessian deconvolution and ADMM-DRE-SIM in the condition of Poisson noise mixed with Gaussian noise $σ = 0.5$ are calculated to be 0.07, 0.10, 0.09, and 0.68, respectively. In a word, ADMM-DRE-SIM shows good performance in the resistance to the background and noise during the image processing of resolution enhancement, which is very useful in processing the experimental SIM images.

Fig. 3

Simulation results for the effects of background and noise levels on the resolution enhancement of SIM by various algorithms. (a) The ground truth, wide-field, SIM, and ADMM-DRE-SIM images of the simulated structures without the background and noise. (b) The SIM, Wiener deconvolution, RLTV deconvolution, Hessian deconvolution, and ADMM-DRE-SIM images with a uniform background combined with Gaussian noise $σ = 0.1$ , 0.3, 0.5, and Poisson noise mixed with Gaussian noise $σ = 0.5$ .

Furthermore, two experiments are performed by imaging the intracellular structures with a commercial SIM system (Nikon, N-SIM) to further demonstrate the resolution enhancement capability of ADMM-DRE-SIM. The exposure time is set as 200 ms with illumination power of about 22.9 mW. First, the tubulins in mouse embryonic fibroblast (MEF) cells are investigated, as shown in Fig. 4(a). Tubulin is part of the cytoskeleton of cells and has a filamentous distribution with large space. Compared to SIM, ADMM-DRE-SIM provides finer structural details of the tubulins, and the overlapped fibers can be distinguished more clearly. Meanwhile, the intensity profiles along the labeled line also intuitively show the resolution enhancement. To quantitatively determine the resolution enhancement, a Fourier ring correlation (FRC) method is utilized to characterize the spatial resolution, and the calculated results are given in Fig. 4(b). The resolution is improved from 145 nm of SIM to 82 nm by ADMM-DRE-SIM, which indicates that ADMM-DRE-SIM is an effective technique to improve the resolution of SIM. Second, the actins in NIH/3T3 cells are further investigated, as shown in Fig. 4(c). Different from the tubulins, the distribution of actins in cells is much denser, and the fiber is thinner. Thus, the images of actins show a stronger background due to the out of focus fluorescence, which bring the difficulty for further resolution enhancement. Similarly, ADMM-DRE-SIM shows clearer spatial structure of actins compared with SIM. Importantly, ADMM-DRE-SIM can greatly suppress the artifacts of the SIM image due to the calculation error of the background in the image reconstruction. According to the calculated results of FRC in Fig. 4(d), the resolution of ADMM-DRE-SIM is enhanced to 85 nm from 140 nm of SIM. Based on our experimental observations in Fig. 4, ADMM-DRE-SIM can realize the resolution enhancement by a factor of about 1.6.

Fig. 4

Experimental results for the resolution enhancement of SIM by ADMM-DRE-SIM. (a) The wide-field, SIM, and ADMM-DRE-SIM images of tubulins in MEF cells and (c) actins in NIH/3T3 cells, together with the corresponding intensity distributions along the labeled lines in the insets. The FRCs of the SIM and ADMM-DRE-SIM images for (b) tubulins and (d) actins.

To specifically exhibit the superiority of ADMM-DRE-SIM in suppressing the background and noise, the SIM images of tubulins with strong background or low signal-to-noise ratio (SNR) are separately collected and processed in experiment. As shown in Fig. 5(a), the tubulin fibers in the SIM image are seriously contaminated by the background, although SIM has the function of optical sectioning.³³ The strong background results from imperfect fluorescence staining during sample preparation. The resolution-enhanced images by conventional deconvolution algorithms of RL, RLTV, and Hessian are all affected by the background and therefore contain severe artifacts. On the contrary, ADMM-DRE-SIM can eliminate the background and keep the fiber structures with enhanced resolution because the image degradation model in ADMM-DRE-SIM has separated the background. And the sparse deconvolution has similar performance. As shown in Fig. 5(b), the SIM image has quite a low SNR due to the recorded nine images, with structured illumination being acquired at high speed with a short exposure time (0.5 ms),⁴ which results in a great challenge for further resolution enhancement. In the same way, the noise is greatly amplified using conventional deconvolution algorithms to process the SIM image with low SNR, and the artifacts are very serious and almost confused with the tubulin fibers. In contrast, ADMM-DRE-SIM succeeds in suppressing the artifacts and acquiring a resolution-enhanced image with high-fidelity structures.

Fig. 5

Experimental results for the effects of the strong background and low SNR on the resolution enhancement of SIM by various algorithms. (a) SIM image of tubulins with strong background. Left, the stitched image with the SIM image on the top and the ADMM-DRE-SIM image on the bottom; right, the enlarged SIM image marked with the yellow square and the images processed by RL, RLTV, Hessian, sparse deconvolution, and ADMM-DRE-SIM algorithms, respectively. (b) SIM image of tubulins with low SNR. The image arrangement is the same as (a).

To verify the superiority of ADMM-DRE-SIM in structural fidelity, the SIM image of tubulins with strong background⁶ is processed by ADMM-DRE-SIM and modified ADMM-DRE-SIM; here the modified ADMM-DRE-SIM means that the background estimation is directly subtracted from the input SIM image, and the image degradation model has only the convolution of PSF of SIM. The SIM image in Fig. 6(a) has the strong background, and the background map can be estimated by the wavelet transform, as shown in Fig. 6(b). The background subtracted SIM image has obvious discontinuity, as shown in Fig. 6(c). As shown in Figs. 6(f) and 6(g), ADMM-DRE-SIM and the modified version both can achieve the resolution enhancement with sharper profiles compared with the SIM image. The direct subtraction of the estimated background map leads to the accidental removal of the useful structures, as shown in Fig. 6(e) and therefore it is very difficult to recover the missing structures during the image processing of resolution enhancement without the additional information, as shown in Fig. 6(f). However, ADMM-DRE-SIM addresses this issue by inserting the background estimation into the image degradation model instead of the direct subtraction, where the accidental removal of the structures can be eliminated during the iterative optimization of the network for the resolution enhancement. The resolution-enhanced image in Fig. 6(g) by ADMM-DRE-SIM shows the higher fidelity with fewer missing structures compared with the image processed by modified ADMM-DRE-SIM with direct subtraction of background estimation in Fig. 6(f). By comparison, it is obvious that ADMM-DRE-SIM has the stronger robustness and can preserve the structural information while removing the background and noise.

Fig. 6

Comparison of ADMM-DRE-SIM and modified ADMM-DRE-SIM in structural fidelity. (a) SIM image of tubulins; (b) estimated background map; (c) image by directly subtracting the estimated background map from the SIM image. (d)–(g) The SIM, background subtracted SIM, modified ADMM-DRE-SIM, and ADMM-DRE-SIM images for the selected area marked with the yellow square, respectively.

For the superresolution imaging of small intracellular structures, the pixel size of the camera is also an important factor for the final imaging performance. The smaller pixel size can contribute to the higher resolution bound, while it also leads to the lower SNR, and therefore the pixel size of the camera used in superresolution imaging is usually a few micrometers. However, the pixel effect sometimes results in a discrete profile of the superresolution structures. To acquire a resolution-enhanced image of the finer structures with smooth profile, a $2 \times$ upsampling is embedded in the front of the neural network. In this way, a $2 \times$ enlarged superresolution image with more details can be acquired. The CCPs, which play an important role in transporting proteins between organelles, have ring shapes, and it is difficult to be recognized in wide-field microscopy due to its small size. To demonstrate the performance of ADMM-DRE-SIM with upsampling, the CCPs are first imaged by SIM,⁶ and then processed by ADMM-DRE-SIM with and without $2 \times$ upsampling, respectively. As shown in Fig. 7(a), some CCPs show the ring shape in SIM, while some others have the ball shape because of the nonuniform size. By the image processing of ADMM-DRE-SIM, the background is suppressed and the ring shape of most CCPs can be visualized through the resolution enhancement. As can be seen in the enlarged images of the selected area in Figs. 7(b) and 7(c), ADMM-DRE-SIM can extract the ring shape from the ball shape of the SIM image, and ADMM-DRE-SIM with $2 \times$ upsampling can further provide a smoother and finer structure, which will be very helpful for the analysis of fine structures and functions.

Fig. 7

Experimental results for the resolution enhancement of SIM by ADMM-DRE-SIM with $2 \times$ upsampling. (a) The stitched image of CCPs with the SIM image on the left and the ADMM-DRE-SIM image on the right. (b) and (c) The SIM, ADMM-DRE-SIM, and $2 \times$ upsampling ADMM-DRE-SIM images for the selected areas in panel (a), associated with the intensity distributions along the labeled lines.

As shown above, ADMM-DRE-SIM shows a powerful ability to transfer the resolution-limited SIM image to the resolution-enhanced image by bridging the low and high frequencies with the implicit priors from the network and Hessian continuity. Meanwhile, an inner image degradation model containing the equivalent PSF of SIM and background map offers the resistance to the strong background. It should be pointed out that the image degradation model is utilized in each iteration during the optimization to maintain the structural integrity because a simple subtraction of the background estimation may lead to the structural defects, especially under the low SNR condition. ADMM-DRE-SIM not only realizes the resolution enhancement of SIM but also retains the advantages of SIM involving high imaging speed, low photo-damage, and wide applications for various fluorescent labels. Therefore, ADMM-DRE-SIM provides a well-established tool to capture the high-speed dynamics of finer structures, which will have important applications in many biological areas, such as organelle dynamics³⁴ and cell metabolism.³⁵ However, ADMM-DRE-SIM has also some limitations. The imaging degradation model is based on the premise of uniform PSF across the whole image, which is an ideal situation and not very consistent with actual experiments; thus the bias may induce uneven resolution enhancement. This issue can be addressed by employing space-variation PSF. In addition, the SIM images in this work are reconstructed by fairSIM, which is a classic algorithm without the priors, and so the effect of resolution enhancement by ADMM-DRE-SIM is obvious due to the use of implicit priors. If the SIM images are reconstructed by TVSIM³⁶ or Hessian-SIM,⁴ which have already exploited the priors, the resolution enhancement ability of ADMM-DRE-SIM will be slightly reduced.

4. Conclusions

In summary, we have developed an ADMM-DRE-SIM to further improve the resolution of SIM by an untrained neural network optimized with an ADMM framework. By incorporating an image degradation model containing equivalent PSF of SIM and additional background map, ADMM-DRE-SIM can significantly improve the resolution of SIM even under the condition of strong background and noise levels. A resolution enhancement by a factor of about 1.6 was experimentally demonstrated by the superresolution imaging of tubulins and actins in cells. Possessing the resolution enhancement ability while maintaining the intrinsic advantages of SIM, ADMM-DRE-SIM can provide a flexible and robust tool for studying the high-speed dynamics of biological fine structures, which will greatly promote the development of biomedical imaging. Furthermore, by inserting the imaging degradation model of three-dimensional (3D) SIM³⁷ into the framework of ADMM-DRE-SIM, 3D resolution enhancement can also be achieved. Additionally, ADMM-DRE-SIM is not limited to the resolution enhancement of SIM. The superresolution microscopy images acquired by some other techniques can also be further processed to obtain the higher resolution once the corresponding equivalent PSF is provided, for example, confocal spinning-disk,³⁸ two-photon,³⁹ and expansion microscopy.⁴⁰

Data and Code Availability

The data and code presented in this study are available from the corresponding author upon request.

Acknowledgments

This work was supported by the National Natural Science Foundation of China (Grant Nos. 12274129, 12274139, 12074121, 92150301, 62105101, 62175066, and 12034008) and the Science and Technology Commission of Shanghai Municipality (Grant Nos. 21XD1400900, 20ZR1417100, and 21JM0010700). The authors declare no competing financial interests.

References

1.

M. G. Gustafsson, “Surpassing the lateral resolution limit by a factor of two using structured illumination microscopy,” J. Microsc., 198 82 –87 https://doi.org/10.1046/j.1365-2818.2000.00710.x JMICAR 0022-2720 (2000). Google Scholar

2.

S. Q. Hu et al., “Structured illumination microscopy reveals focal adhesions are composed of linear subunits,” Cytoskeleton, 72 (5), 235 –245 https://doi.org/10.1002/cm.21223 (2015). Google Scholar

3.

I. S. Opstad et al., “Multi-color imaging of sub-mitochondria structures in living cells using structured illumination microscopy,” Nanophotonics, 7 (5), 935 –947 https://doi.org/10.1515/nanoph-2017-0112 (2018). Google Scholar

4.

X. Huang et al., “Fast, long-term, super-resolution imaging with Hessian structured illumination microscopy,” Nat. Biotechnol., 36 (5), 451 –459 https://doi.org/10.1038/nbt.4115 NABIF9 1087-0156 (2018). Google Scholar

5.

A. Markwirth et al., “Video-rate multi-color structured illumination microscopy with simultaneous real-time reconstruction,” Nat. Commun., 10 4315 https://doi.org/10.1038/s41467-019-12165-x NCAOBW 2041-1723 (2019). Google Scholar

6.

C. Qiao et al., “Evaluation and development of deep neural networks for image super-resolution in optical microscopy,” Nat. Methods, 18 (2), 194 –202 https://doi.org/10.1038/s41592-020-01048-5 1548-7091 (2021). Google Scholar

7.

Q. Chen et al., “Quantitative analysis of interactive behavior of mitochondria and lysosomes using structured illumination microscopy,” Biomaterials, 250 120059 https://doi.org/10.1016/j.biomaterials.2020.120059 BIMADU 0142-9612 (2020). Google Scholar

8.

D. Li et al., “Extended-resolution structured illumination imaging of endocytic and cytoskeletal dynamics,” Science, 349 aab3500 https://doi.org/10.1126/science.aab3500 SCIEAS 0036-8075 (2015). Google Scholar

9.

P. Kner et al., “Super-resolution video microscopy of live cells by structured illumination,” Nat. Methods, 6 (5), 339 –342 https://doi.org/10.1038/nmeth.1324 1548-7091 (2009). Google Scholar

10.

W. Hell and J. Wichmann, “Breaking the diffraction resolution limit by stimulated emission: stimulated-emission-depletion fluorescence microscopy,” Opt. Lett., 19 (11), 780 –782 https://doi.org/10.1364/OL.19.000780 OPLEDP 0146-9592 (1994). Google Scholar

11.

S. Manley et al., “High-density mapping of single-molecule trajectories with photoactivated localization microscopy,” Nat. Methods, 5 (2), 155 –157 https://doi.org/10.1038/nmeth.1176 1548-7091 (2008). Google Scholar

12.

M. J. Rust, M. Bates and X. Zhuang, “Sub-diffraction-limit imaging by stochastic optical reconstruction microscopy (STORM),” Nat. Methods, 3 (10), 793 –796 https://doi.org/10.1038/nmeth929 1548-7091 (2006). Google Scholar

13.

M. G. Gustafsson, “Nonlinear structured-illumination microscopy: wide-field fluorescence imaging with theoretically unlimited resolution,” Proc. Natl. Acad. Sci. U. S. A., 102 (37), 13081 –13086 https://doi.org/10.1073/pnas.0406877102 (2005). Google Scholar

14.

E. H. Rego et al., “Nonlinear structured-illumination microscopy with a photoswitchable protein reveals cellular structures at 50-nm resolution,” Proc. Natl. Acad. Sci. U. S. A., 109 (3), E135 –E143 https://doi.org/10.1073/pnas.1107547108 (2012). Google Scholar

15.

W. Zhao et al., “Sparse deconvolution improves the resolution of live-cell super-resolution fluorescence microscopy,” Nat. Biotechnol., 40 (4), 606 –617 https://doi.org/10.1038/s41587-021-01092-2 NABIF9 1087-0156 (2022). Google Scholar

16.

J. B. Sibarita, “Deconvolution microscopy,” Adv. Biochem. Eng./Biotechnol., 95 201 –243 https://doi.org/10.1007/b102215 ABEBDZ 0724-6145 (2005). Google Scholar

17.

M. Hupfel et al., “Wavelet-based background and noise subtraction for fluorescence microscopy images,” Biomed. Opt. Express, 12 (2), 969 –980 https://doi.org/10.1364/BOE.413181 BOEICL 2156-7085 (2021). Google Scholar

18.

C. M. Galloway, E. C. Le Ru and P. G. Etchegoin, “An iterative algorithm for background removal in spectroscopy by wavelet transforms,” Appl. Spectrosc., 63 (12), 1370 –1376 https://doi.org/10.1366/000370209790108905 APSPA4 0003-7028 (2009). Google Scholar

19.

D. Ulyanov, A. Vedaldi and V. Lempitsky, “Deep image prior,” in Proc. IEEE Conf. Comput. Vis. and Pattern Recognit., 9446 –9454 (2018). https://doi.org/10.1109/CVPR.2018.00984 Google Scholar

20.

F. Wang et al., “Phase imaging with an untrained neural network,” Light: Sci. Appl., 9 (1), 77 https://doi.org/10.1038/s41377-020-0302-3 (2020). Google Scholar

21.

H. Li et al., “Deep DIH: single-shot digital in-line holography reconstruction by deep learning,” IEEE Access, 8 202648 –202659 https://doi.org/10.1109/ACCESS.2020.3036380 (2020). Google Scholar

22.

C. Bai et al., “Dual-wavelength in-line digital holography with untrained deep neural networks,” Photonics Res., 9 (12), 2501 –2510 https://doi.org/10.1364/PRJ.441054 (2021). Google Scholar

23.

S. Liu et al., “Computational ghost imaging based on an untrained neural network,” Opt. Laser Eng., 147 106744 https://doi.org/10.1016/j.optlaseng.2021.106744 (2021). Google Scholar

24.

Z. Burns and Z. Liu, “Untrained, physics-informed neural networks for structured illumination microscopy,” Opt. Express, 31 8714 –8724 https://doi.org/10.1364/OE.476781 OPEXFF 1094-4087 (2023). Google Scholar

25.

X. Liu et al., “Reconstruction of structured illumination microscopy with an untrained neural network,” Opt. Commun., 537 129431 https://doi.org/10.1016/j.optcom.2023.129431 (2023). Google Scholar

26.

Y. He et al., “Surpassing the resolution limitation of structured illumination microscopy by an untrained neural network,” Bio. Opt. Express, 14 (1), 106 –117 https://doi.org/10.1364/BOE.479621 BOEICL 2156-7085 (2023). Google Scholar

27.

D. P. Kingma and J. Ba, “Adam: a method for stochastic optimization,” (2014). Google Scholar

28.

P. Cascarano et al., “Combining weighted total variation and deep image prior for natural and medical image restoration via ADMM,” in 21st Int. Conf. Computational Sci. and its Appl. (ICCSA), 39 –46 (2021). https://doi.org/10.1109/ICCSA54496.2021.00016 Google Scholar

29.

M. Muller et al., “Open-source image reconstruction of super-resolution structured illumination microscopy data in ImageJ,” Nat. Commun., 7 10980 https://doi.org/10.1038/ncomms10980 NCAOBW 2041-1723 (2016). Google Scholar

30.

E. Sekko, G. Thomas and A. Boukrouche, “A deconvolution technique using optimal Wiener filtering and regularization,” Signal Process., 72 23 –32 https://doi.org/10.1016/S0165-1684(98)00161-3 (1999). Google Scholar

31.

N. Dey et al., “Richardson-Lucy algorithm with total variation regularization for 3D confocal microscope deconvolution,” Microsc. Res. Tech., 69 260 –266 https://doi.org/10.1002/jemt.20294 MRTEEO 1059-910X (2006). Google Scholar

32.

S. Lefkimmiatis, A. Bourquard and M. Unser, “Hessian-based norm regularization for image restoration with biomedical applications,” IEEE Trans. Image Process., 21 (3), 983 –995 https://doi.org/10.1109/TIP.2011.2168232 IIPRE4 1057-7149 (2012). Google Scholar

33.

J. Mertz, “Optical sectioning microscopy with planar or structured illumination,” Nat. Methods, 8 (10), 811 –819 https://doi.org/10.1038/nmeth.1709 1548-7091 (2011). Google Scholar

34.

D. S. Dong et al., “Super-resolution fluorescence-assisted diffraction computational tomography reveals the three-dimensional landscape of the cellular organelle interactome,” Light-Sci. Appl., 9 (1), 15 https://doi.org/10.1038/s41377-020-0249-4 (2020). Google Scholar

35.

N. J. Dolman et al., “Tools and techniques to measure mitophagy using fluorescence microscopy,” Autophagy, 9 (11), 1653 –1662 https://doi.org/10.4161/auto.24001 (2013). Google Scholar

36.

K. Chu et al., “Image reconstruction for structured-illumination microscopy with low signal level,” Opt. Express, 22 (7), 8687 –8702 https://doi.org/10.1364/OE.22.008687 OPEXFF 1094-4087 (2014). Google Scholar

37.

L. Shao et al., “Super-resolution 3D microscopy of live whole cells using structured illumination,” Nat. Methods, 8 (12), 1044 –1046 https://doi.org/10.1038/nmeth.1734 1548-7091 (2011). Google Scholar

38.

O. Schulz et al., “Resolution doubling in fluorescence microscopy with confocal spinning-disk image scanning microscopy,” Proc. Natl. Acad. Sci. U. S. A., 110 (52), 21000 –21005 https://doi.org/10.1073/pnas.1315858110 (2013). Google Scholar

39.

W. Zong et al., “Fast high-resolution miniature two-photon microscopy for brain imaging in freely behaving mice,” Nat. Methods, 14 (7), 713 –719 https://doi.org/10.1038/nmeth.4305 1548-7091 (2017). Google Scholar

40.

D. E. Sun et al., “Click-ExM enables expansion microscopy for all biomolecules,” Nat. Methods, 18 (1), 107 –113 https://doi.org/10.1038/s41592-020-01005-2 1548-7091 (2021). Google Scholar

Biography

Yu He is a PhD student studying at the State Key Laboratory of Precision Spectroscopy, East China Normal University (ECNU) under the supervisions of Prof. Shian Zhang. His research focuses on high-speed super-resolution microscopy.

Yunhua Yao is an associate professor from the State Key Laboratory of Precision Spectroscopy at ECNU. He obtained his PhD in Optics from ECNU in 2018. His current research interest focuses on high-speed super-resolution microscopy and ultrafast optical imaging.

Yilin He is a PhD student studying at the State Key Laboratory of Precision Spectroscopy, ECNU under the supervisions of Prof. Shian Zhang. His research focuses on high-speed super-resolution microscopy.

Zhengqi Huang is a PhD student studying at the State Key Laboratory of Precision Spectroscopy, ECNU under the supervisions of Prof. Shian Zhang. His research focuses on high-speed super-resolution microscopy.

Dalong Qi is a young professor from the State Key Laboratory of Precision Spectroscopy at ECNU. He obtained his PhD in Optics from ECNU in 2017. His current research interest focuses on ultrafast optical and electronic imaging techniques and their applications.

Chonglei Zhang is a professor from Nanophotonics Research Center at Shenzhen University. He obtained his PhD in Physics from Nankai University in 2007. His current research interest focuses on super resolution microscopy and surface plasmon resonance.

Xiaoshuai Huang received his bachelor’s degree in science from Wuhan University in 2013 and his PhD from Peking University in 2018. He is an assistant professor at Peking University. From 2018 to 2020, he was trained as a postdoctoral research fellow at Peking University. His research is focused on super-resolution microscopy and cell biology.

Kebin Shi is a professor from State Key Laboratory for Mesoscopic Physics at Peking University. He obtained his PhD from the Pennsylvania State University. His current research interest focuses on nonlinear photonics and biomedical imaging.

Pengpeng Ding is a PhD student studying at the State Key Laboratory of Precision Spectroscopy, ECNU under the supervisions of Prof. Shian Zhang. His research focuses on ultrafast optical imaging.

Chengzhi Jin is a PhD student studying at the State Key Laboratory of Precision Spectroscopy, ECNU under the supervisions of Prof. Shian Zhang. His research focuses on ultrafast optical imaging.

Lianzhong Deng is an associate professor from the State Key Laboratory of Precision Spectroscopy at ECNU. He obtained his PhD in Optics from ECNU in 2008. His current research interest focuses on light field manipulation and their applications.

Zhenrong Sun is a professor of the State Key Laboratory of Precision Spectroscopy at ECNU. He obtained his PhD in Physics from ECNU in 2007. His current research interest focuses on ultrafast dynamics of clusters and ultrafast optical imaging.

Xiaocong Yuan is a professor and the director of Nanophotonics Research Center at Shenzhen University. He obtained his PhD in Physics from King’s College London in 1994. His current research interest focuses on optical manipulation, high-sensitivity sensor, super-resolution microscopy and surface-enhanced Raman spectroscopy.

Shian Zhang is a professor and the deputy director of the State Key Laboratory of Precision Spectroscopy at ECNU. He obtained his PhD in Optics from ECNU in 2006. His current research interest focuses on ultrafast optical imaging, high-speed super-resolution microscopy and light field manipulation.

CC BY: © The Authors. Published by SPIE and CLP under a Creative Commons Attribution 4.0 International License. Distribution or reproduction of this work in whole or in part requires full attribution of the original publication, including its DOI.

Citation Download Citation

Yu He, Yunhua Yao, Yilin He, Zhengqi Huang, Dalong Qi, Chonglei Zhang, Xiaoshuai Huang, Kebin Shi, Pengpeng Ding, Chengzhi Jin, Lianzhong Deng, Zhenrong Sun, Xiaocong Yuan, and Shian Zhang "Untrained neural network enhances the resolution of structured illumination microscopy under strong background and noise levels," Advanced Photonics Nexus 2(4), 046005 (4 July 2023). https://doi.org/10.1117/1.APN.2.4.046005

Received: 10 February 2023; Accepted: 14 June 2023; Published: 4 July 2023

Access the abstract

JOURNAL ARTICLE
10 PAGES

DOWNLOAD PAPER SAVE TO MY LIBRARY

GET CITATION

KEYWORDS

Resolution enhancement technologies

Image resolution

Image enhancement

Neural networks

Image processing

Background noise

Image restoration

1.

Introduction

2.

Methods

Eq. (1)

Eq. (2)

Eq. (3)

Eq. (4)

Eq. (5)

Eq. (6)

Eq. (7)

Fig. 1

Eq. (8)

Eq. (9)

Eq. (10)

3.

Results and Discussion

Fig. 2

Fig. 3

Fig. 4

Fig. 5

Fig. 6

Fig. 7

4.

Conclusions

Data and Code Availability

Acknowledgments

References

Biography

Show All Keywords

Keywords/Phrases

Search In:

Publication Years