2.1.

Software Structure and Acquisition Workflow

The acquisition program is built on Pycro-Manager (v0.18.1). Pycro-Manager is a Python package that communicates with Micro-Manager, which handles hardware control via Micro-Manager’s device adaptors, such as the laser scanning module OpenScan. The fast data transfer and translation layer between Python and Java in Pycro-Manager allows Java libraries to be called as if they were in Python. Because Micro-Manager already supports a large variety of microscopy hardware, integrating Pycro-Manager allows for customized acquisitions to be programmed in Python, enabling analytical Python packages to be easily inserted into the acquisition pipeline in a single Python environment.

The acquisition program is written in Jupyter Notebook. The program first deploys a fast prescan at low magnification (e.g., $4\times$). A position list covering the whole slide area is automatically generated according to the slide size, field of view size, and image pixel size. After the acquisition, the tiles are stitched according to the position list using the grid/collection plugin³⁴ from ImageJ²⁶ accessed from Python via PyImageJ. During the acquisition, software autofocus is applied (see Appendix B for details of the autofocus algorithm), and the focus position is recorded for each position. This is necessary because a tissue section mounted on a glass slide can have variations in terrain heights, which leads to varying focal planes at different locations. Background tiles are automatically recorded and averaged to produce a background image, which is used for white balancing and illumination correction. This scan can be done relatively quickly because of the large field of view (FOV) of each frame at low magnification. A $4\times$ magnification objective is used for the low-magnification scan because it provides a good balance between the resolution of details needed to identify some tissue features and a wide FOV. The stitched prescan image is then exported in a BioFormats compatible pyramidal format (OME TIFF) with slide metadata (e.g., $XY$ pixel size, objective magnification) by calling QuPath via its command line interface from Jupyter Notebook.

The user can then make annotations by opening the low-magnification prescan image in QuPath. A QuPath script is used to generate acquisition tiles, shown in an overlay on the image, covering the annotated areas according to the FOV size at 20× magnification for BF and LSM. Alternatively, the annotations can be generated automatically by DL-based detection models (details covered in the following section). The acquisition program then uses the annotation(s) to perform a selective acquisition at a higher magnification with BF and LSM using the positions saved in the files. The LSM control is handled by an in-house software-OpenScan,³⁵ a Micro-Manager device library that handles the generation of scanning waveforms, the control of scanning galvanometer, the scanning resolution, and the photomultiplier tube (PMT). The interactions of the software used in our system are illustrated in Fig. 2.

Fig. 2

Architecture of the application. The acquisition hardware is controlled via Pycro-Manager, which interacts with Micro-manager and translates the Java libraries into Python. QuPath is called for annotation handling and reading/outputting annotation files.

The focus of each tile at $20\times$ BF is first roughly determined by interpolating the focus map recorded in the 4× prescan and then fine-tuned using the same autofocus algorithm with a narrower searching range during the 20× BF acquisition. The focus of each tile is again recorded and the focus map is interpolated and sampled to generate a more finely detailed focus map to be used for the LSM modality (the $20\times$ LSM and BF FOV sizes can be slightly different, and there will be a $Z$ offset between modalities). For multiphoton LSM or SHG, a $z$-stack is often beneficial due to the optical sectioning property of multiphoton microscopy resulting in incomplete coverage of the full depth of the tissue.⁹ The step size and range of the $z$-stack can be configured according to the sample thickness; the following experiments used a $5\text{-}\mu \mathrm{m}$ step size with three steps. With the center of the tissue at each location roughly determined in the $20\times$ BF scan, the number of slices in the $z$-stack of LSM can be reduced such that it only covers the depth around the center of the tissue at each location.

2.2.

Multimodal Imaging System

The imaging system is designed to couple BF and multiphoton LSM light paths with the ability to switch between the two modalities via simple shutter control. A Tsunami Ti:Sapphire laser (Spectra-Physics, Santa Clara, California) tuned to 800 nm, with a pulse length of $\sim 100\mathrm{fs}$, is directed through a Pockels cell (ConOptics, Danbury, Connecticut), half and quarter waveplates (ThorLabs, Newton, New Jersey), a beam expander (ThorLabs), a 3-mm galvanometer driven mirror pair (Cambridge, Bedford, Massachusetts), a scan ($f=75\mathrm{mm}$) and tube ($f=250\mathrm{mm}$) lens pair (ThorLabs), and a dichroic mirror (Semrock, Rochester, New York) and is focused by a $20\times /0.75$ NA air objective (Nikon, Melville, New York). SHG light is collected in the forward direction with a variable NA condenser (Olympus, Lombard, Illinois) and filtered with a 680-nm short-pass filter (Semrock) and an interference filter centered at 400 nm with a full-width at half-maximum bandwidth of 10 nm (ThorLabs). The back aperture of the condenser lens is focused onto a H7422-40P GaAsP PMT (Hamamatsu, Hamamatsu, Japan) using a secondary collection lens ($f=150\mathrm{mm}$, ThorLabs). The signal from the PMT is amplified with a C7319 integrating amplifier (Hamamatsu) and sampled with an analog data acquisition device NI PIXe-6356 (National Instruments, Austin, Texas). The galvanometer is controlled by an analog data output device NI PXIe-6738 (National Instruments, Austin, Texas). Timing among the galvo scanners, signal acquisition, and motorized stage positioning is achieved using our custom software called OpenScan in conjunction with the open-source software Micro-Manager (v2.0).²⁸

The Rapid Automated Modular Microscope system (Applied Scientific Instrumentation, Eugene, Oregon) serves as our microscope base, and ASI motorized translation stages are used for $x$, $y$, and $z$ motion control. BF images are captured with the same system using a MCWHL2 white LED lamp (ThorLabs) and both the $20\times /0.75$ NA objective and a $4\times /0.13$ NA objective (Nikon). White light from this lamp travels through the condenser directed by a short-pass dichroic mirror with a cutoff at 670 nm (Semrock). The white light is passed through the first dichroic, focused on an RGB camera (QICAM Fast 1394, Qimaging, Surrey, BC, Canada) with a collection lens ($f=230\mathrm{mm}$, ThorLabs). The image data transfer is handled by OpenScan/Micro-Manager. An objective slider (ASI) is used for easily switching between $4\times$ and $20\times$ objectives and a $z$-oriented motorized arm ($f$-stage, ASI) for optimizing the amount of light collected by the condenser for each imaging modality and objective position. All components of the laser light focusing and collection are contained in a blackout box as shown by the dashed line in Fig. 3.

Fig. 3

Optical schematic of the laser and BF light path. The 800-nm laser light is passed through an electro-optic modulator (Pockels Cell) and a half-wave plate followed by a quarter-wave plate ($\lambda /2$, $\lambda /4$). The polarized light is focused by a 75-mm focal length scan lens and a 250-mm focal length tube lens to the back aperture of a Nikon $20\times /0.75$ NA objective lens. Light from the sample is collected by an adjustable NA condenser lens and passed through a 670-nm dichroic mirror (FF670), a 680-nm short-pass filter (680 SP), and a 400-nm interference filter (400/10) and is focused by a 150-mm focal length collection lens to a Hamamatsu 7422-40P photomultiplier tube (PMT). White light is generated by an LED lamp and passed through the condenser and the sample and collected by either a Nikon $4\times /0.13$ NA objective or the $20\times$ objective. The light passes through a 720-nm dichroic mirror (FF 720) and is focused by a 230-mm focal length tube lens onto a Qimaging QICAM Fast 1394 camera (RGB Camera).

2.3.

Automatic ROIs Detection with Deep Neural Networks

In addition to acquiring annotated regions defined by user-entered annotations, our system can generate annotations automatically via DL-based detection models.

Convolutional neural networks (CNNs) have demonstrated state-of-the-art performance in many vision tasks and have been extensively used in computational histopathology for tasks such as tumor detection, gland segmentation, cell segmentation, and cell classification.^36–39 A CNN differs from traditional machine learning methods in that the feature extractor is parameterized by layers of convolutions with learnable filters and nonlinear functions. The parameters of the filters are learned during the training, whereas most of the traditional machine learning models for vision tasks use handcrafted features, such as SIFT.⁴⁰ With a sufficient amount of training data available, CNNs usually outperform traditional machine learning models and show better generalizability to unseen data. In our design, we use ResNet⁴¹ as the CNN backbone for building the detector. ResNet uses residual connections that allow for more convolution layers to be used in the network without suffering from gradient vanishing, and it can alleviate overfitting by providing “shortcuts” for certain information to skip redundant convolutional layers. The training settings can be found in Appendix A.1 The trained CNN are enabled in the acquisition workflow. The low-magnification scan is processed by the CNN, and the coordinates of positive areas are generated and saved in a position list. The position list is then used by the acquisition program for high magnification BF and SHG imaging.

2.4.

Deep Learning-Based Run-Time Image Restoration

Performing laser scanning imaging, such as SHG, can be time-consuming due to the point-wise scanning only collecting a single pixel at a time. The imaging time is further extended if a $z$-stack is required to cover the sample thickness. A complete scan of a tissue section could take several days to complete. Thus solutions that enable faster scanning while maintaining image quality are desirable. The image quality is mainly determined by the SNR and scanning resolution. Generally, the SNR of LSM is proportional to the square root of the number of photons (which is proportional to the dwelling time of each scanning point). The scanning resolution of the images is determined by the number of scanning points along each dimension (apart from optical resolution determined by the NA of the objective). Ideally, the scanning resolution should match the optical resolution such that Nyquist sampling for the optical signals is met. This means that a lower scanning resolution might undersample the optical signals and a higher scanning resolution, i.e., more scanning points along each dimension, would be necessary to increase the image resolution. Due to LSM deploying point-to-point scanning, the scanning time increases quadratically with the scanning resolution. Fortunately, recent advances in DL-based image restoration and enhancement techniques have brought new solutions to address this problem. In our system, two types of DL-based image enhancement models are implemented to shorten the scanning time while preserving the image quality, including an SSD model and a SISR model. With a successfully trained model enabled during the acquisition, the scanning procedures can be performed an order of magnitude faster while maintaining image quality comparable to that of the original scan.

2.4.1.

Self-supervised denoising model

The signals that form an image are considered to be $\mathit{x}=\mathit{s}+\mathit{n}$, where $\mathit{s}$ and $\mathit{n}$ are drawn from two the joint distributions:

Eq. (1)

$$p(\mathit{s},\mathit{n})=p(\mathit{s})p(\mathit{n}|\mathit{s}).$$

Within a given radius $\mathbf{R}$ of $i$, the signal value ${\mathit{s}}_i$ conditionally depends on the observation of signals at other locations within the radius (i.e., $p({\mathit{s}}_i)\ne p({\mathit{s}}_i|{\mathit{s}}_j)$ s. t. $\{{\mathit{s}}_j\in \mathbf{R},j\ne i\}$) but is independent of the noise at other locations (i.e., $p({\mathit{s}}_i)=p({\mathit{s}}_i|{\mathit{n}}_j)$ s. t. $\{{\mathit{n}}_j\in \mathbf{R},j\ne i\}$), meaning that the distribution of the noise conditioning on the signal factorizes as

Eq. (2)

$$p(\mathit{n}|\mathit{s})=\prod _{i}p({\mathit{n}}_i|{\mathit{s}}_i).$$

Under these assumptions, the expectation of the pixel value at $i$ is estimated by a blind spot network that observes the surrounding region of the pixel $i$.⁴² Because $\mathbb{E}[{\mathit{x}}_i]=\mathbb{E}[{\mathit{s}}_i]+\mathbb{E}[{\mathit{n}}_i]=\mathbb{E}[{\mathit{s}}_i]$ (assuming the noise is zero mean, i.e., $\mathbb{E}[{\mathit{n}}_i]=0$), the estimation is free of noise.⁴² This suggests that the model can estimate the clean image even without observing clean images in the training phase. Such methods are known as self-supervised learning (SSl) denoising.⁴³ Models such as Noise2Void have been successfully deployed for denoising microscopic data and tested for the cases of Poisson noise and Gaussian noise.⁴²

We used the same idea but modified the original implementation of Noise2Void by making use of random dropout and stochastic forward pass similar to Monte Carlo dropout.⁴⁴ In the original training process of Noise2Void, subregions are randomly cropped from the image. In each subregion, a randomly selected pixel is replaced by another randomly selected pixel. The network is then trained to predict the original pixel value at the replaced location. In the inference, the predictions are then obtained at each location using a sliding window. In our training scheme, we applied random dropout on the input image with a small dropout rate $p$ that creates random blind spots in the input image. The neural network was then trained to predict the missing values at the blind spots. The loss was only evaluated at the blind spot locations to prevent the network from overfitting, which could result in an identity mapping. At inference, a number of $k$ dropout operations with the same dropout rate $p$ were applied to the input image, and $k$ outputs were obtained. Only the pixel values at the blind spot locations in each output were kept. The final output was then computed by averaging the $k$ outputs, similar to the stochastic forward pass used in Bayesian neural networks. We used small values $<0.1$ for the dropout rate $p$ to ensure that the created blind spots were sparse enough such that each blind spot had enough surrounding pixels observed by the network to achieve a good estimation of the missing pixels. Also the values of $k$ need to be set inversely proportional to $p$ such that the probability of creating a blind spot at each pixel location at least once is close to 1. The training and inference schemes are illustrated in Fig. 4.

Fig. 4

SSD training and inference. During training, blind spots are created throughout the input images. The model is trained to predict the missing values at the blind spots using an image-to-image translation network. Loss is computed only at blind spot locations. At inference, blind spots are created for the input image multiple times, and the outputs at corresponding blind spot locations are averaged to produce the final denoised image.

2.4.2.

Supervised single-image super-resolution model

The goal of SISR models is to estimate the HR image from a given LR image. Let $\mathit{l}$ and $\mathit{h}$ denote an LR image and its corresponding HR image, respectively. The relation between $\mathit{l}$ and $\mathit{h}$ can be written as $\mathit{l}=\mathit{h}*\mathit{f}$, where $*$ denotes a convolution operation and $\mathit{f}$ denotes the blurring operator. In microscopy, the inverse problem $\mathit{l}*{\mathit{f}}^{-1}=\mathit{h}$ can be intractable due to the difficulty of finding the true point spread function (blurring operator) and solving the deconvolution itself.⁴⁵ Recently, CNN-based solutions have been proposed to estimate the inverse operation and have achieved state-of-the-art performances.⁴⁶ Most of the CNN-based solutions rely on training on a large amount of LR–HR image pairs, with the difference measurements between the network outputs and ground-truth HR images being minimized. Recently, generative adversarial networks (GANs) have been introduced to enable the CNN to generate HR images with higher visual quality.⁴⁷ Moreover, similar example-based networks can also be applied to the image denoising problem, with the network being trained on noisy-clean image pairs.^48,49

Details of the training specifics can be found in Appendix A.2. The CNN backbone of the models is similar to the image-to-image translation model used in Ref. 50, but with only one input channel (Fig. 5). Although the denoising model can be trained solely using noisy images, the SISR model needs LR/HR image pairs to train. For the SISR model, the input images can be collected at both a low scanning resolution and fast scanning rate (fewer pixels along each dimension and a lower SNR), whereas the “ground truth” images can be acquired at a high scanning resolution with a slow scan rate (more pixels along each dimension and a high SNR). The SISR model can then achieve SISR and denoising simultaneously. Once the models are trained, the user can enable the models during the LSM acquisition for run-time image resolution enhancement and denoising.

Fig. 5

U-Net that maps for image-to-image translation. The network can be trained on image pairs consisting of noisy/clean image pairs for denoising and LR/HR image pairs for training the SISR model. The network can also be used for SSD that uses only noisy images. The expressions in the format of $N\times N\times N$ denote the number of channels × feature width × feature height for the convolutional layers.

To enable the generation of HR and high-SNR images with sharper details and better perceptual quality, we incorporate GANs and perceptual loss for training the image-to-image translation network. GANs have been used in many inverse problems, such as SISR,⁵¹ image inpainting,⁵² and style transfer.⁵³ The discriminator of the implemented GAN consists of five convolutional layers, each followed by a ReLU activation function. The output of the last convolutional layer is processed by an average-pooling layer and fed to a linear layer that produces a single prediction. The discriminator serves as a surrogate to push the generator (the image-to-image translation network) to generate outputs that are indistinguishable by the discriminator.

In addition to GANs, another frequently used approach to increase the visual quality of the generated images is perceptual loss.⁵⁴ Perceptual loss is usually measured by a pretrained CNN in which the features extracted by the CNN are compared between the generated image and the ground truth image. The CNN is pretrained on a third-party dataset (usually ImageNet), and the weights are frozen during the training process of the image-to-image translation network. Thus the features extracted by the CNN can be treated as static latent representations of the input images with structural and semantic meanings. Measuring the distances between the generated image and ground truth image in the latent space provides complementary information that is not available from a pixel-wise loss, such as mean-absolute error (L1 loss) between pair of pixels in the two images. In our implementation, we used ImageNet pretrained VGG16 to extract latent features,⁵⁵ as suggested in Ref. 54. Consider a dataset $\mathcal{D}={\{{\mathbf{x}}_i,{\mathbf{y}}_i\}}_{i=1}^N$, where $N$ equals the number of input-target image pairs. The optimization objective of network training is written as

Eq. (3)

$$G*=\arg \underset{G}{\min }\frac{1}{N}\sum _{i=1}^N{\Vert G({\mathbf{x}}_i)-{\mathbf{y}}_i\Vert }_1+{\lambda }_p{\Vert V_{\mathrm{VGG}}(G({\mathbf{x}}_i))-V_{\mathrm{VGG}}({\mathbf{y}}_i)\Vert }_2+{\lambda }_g\arg \underset{D}{\max }{\mathcal{L}}_{\mathrm{GAN}}(G,D),$$

Eq. (4)

$${\mathcal{L}}_{\mathrm{GAN}}(G,D)=\frac{1}{N}\log D({\mathbf{y}}_i)+\log (1-D(G({\mathbf{x}}_i))),$$

where $G$ is the generator parameterized by an image-to-image translation network, $D$ is the discriminator described above, and $V_{\mathrm{VGG}}$ is a pretrained VGG16 network. ${\lambda }_p$ is the weight for the perceptual loss, and ${\lambda }_g$ is the weight for the adversarial loss.

The resulting training pipeline that optimizes all three kinds of loss functions is illustrated in Fig. 6. Similar concepts have been implemented for autofluorescence-harmonic microscopy.⁵⁶ Our major contribution lies in integrating the methods into a run-time analysis workflow along with the acquisition. This system would also facilitate the collection of paired image data of different resolutions and different modalities for training such DL models. We evaluated the efficacy of the image-to-image translation network for denoising and resolution enhancement of SHG images, with results presented in the next section.

Fig. 6

Training procedure of the image-to-image translation network with three types of loss functions. (a) Pixel-wise L1 loss is computed between pixel pairs in the output image and target image. (b) Perceptual loss is measured as the mean-square error between the latent features extracted by a pretrained VGG16 of the output image and target image. (c) A discriminator is trained on par with the generator and pushes the generator to generate outputs that are hard to be distinguished by the discriminator.

3.1.

Samples and Datasets

The samples used for imaging were eight TMA slides bought from US Biomax,⁵⁷ BBS14011, PA2072, PA961e, PA485, PA802, PA2081b, HPan-Ade120Sur, and Hpan-Ade170Sur.

The TMAs contain a total of 769 TMA cores of pancreatic cancer (pancreatic ductal adenocarcinoma), chronic pancreatitis, and normal pancreas tissue. Cores were generally 2 mm in diameter and $4\text{-}\mu \mathrm{m}$ thick. PA2081b, containing a mixture of the three types of cores, was used to collect data with annotations generated automatically by the machine learning detector.

This TMA slide was also used for testing the run-time LSM image denoising and resolution enhance models. The rest of the slides were used for collecting BF and SHG data from manually drawn annotations and the data were used to train the DNNs. Further details regarding each TMA slide can be found in Ref. 57. The hardware of the machine used to control the imaging system is as follows: CPU: Intel^® Core™ i9-10900X CPU 3.70 GHz (10 cores), RAM: 32 GB, GPU: NVIDIA RTX A4000 (16 GB).

3.2.

Selective Acquisition in Multiscale and Multimodal

3.2.1.

Manual annotation

Once the slide is mounted on the stage, the rapid $4\times$ scan function is executed. Image tiles are automatically white-balanced, flat-field corrected, stitched, and exported as a pyramidal OME.TIFF^33,58 files. To make annotations manually, the user opens the $4\times$ image in QuPath and draw annotations with QuPath annotation tools. Annotations with arbitrary closed shapes are supported. The system uses a set of QuPath scripts (written in Groovy) that can convert the annotations into stage position lists that can be read by the acquisition program. The position list is automatically passed to the acquisition program when the $20\times$ and SHG acquisition workflows are executed. Multiple position lists are generated and executed in a loop for multiple annotations on a slide. The acquisition procedure and results of this functionality are shown in Fig. 7.

Fig. 7

Selective acquisition using our system. A rapid BF scan is first conducted at $4\times$. Annotations are then created manually in QuPath or using the machine learning model trained for target detection. The annotations are then converted into position list files to be read by the acquisition program for BF and SHG acquisition at $20\times$. The scale bar for the bottom images is $200\mu \mathrm{m}$.

3.3.

Run-Time Image Restoration

The trained denoising model and resolution enhancement model are enabled during the acquisition by specifying them as the image processing hook function in the Pycro-Anager Acquisition API. The image processing hook function provides access to the data-stream as data is being acquired. This allows the data to be modified, analyzed, and diverted to customized visualization and saving on-the-fly. For evaluating the SSD model and SISR model, the models are enabled with a fast scanning pixel rate of 0.5 MHz at a $256\times 256$ scanning resolution. The processed images are compared with the ground truth images collected using a slow scanning pixel rate of 0.1 MHz at a $512\times 512$ scanning resolution. Note that the same scanning rates and scanning resolutions are used to collect the training data for model training. The SSD model is trained using SHG images collected at 0.5 MHz at a $256\times 256$ scanning resolution, whereas the SISR model is trained to map SHG images collected at 0.1 MHz at a $256\times 256$ scanning resolution to images collected at 0.1 MHz at a $512\times 512$ scanning resolution.

Four metrics are used to evaluate the similarity between the network outputs and the ground truth: peak-signal-to-noise ratio (PSNR), structural similarity (SSIM),⁵⁹ Fréchet inception distance (FID),⁶⁰ and CurveAlign collagen fiber statistics.⁶¹ PSNR is a pixel-wise metric that measures the pair-wise distances of pixels in the same locations of two images. SSIM and FID are perception-based metrics that quantify the visual similarity between images based on measurements derived from the whole images or regions instead of individual pixels. SSIM follows an explicitly defined formula, whereas FID is based on features computed from pretrained Inception networks.⁶² FID is shown to have the ability to produce measurements that align well with visual observations of human, and it is currently the standard metric to assess the quality of images generated by GANs.⁶³

CurveAlign is a popular tool box for characterizing collagen fiber topography, such as measuring collagen fiber density, and fiber alignment coefficients.⁶¹ We computed the absolute differences between the collagen fiber alignment coefficient and collagen fiber density calculated by CurveAlign between the processed image and ground truth image. The absolute difference for each image is than divided by the maximum value of alignment coefficient and density in the testing set, respectively, resulting in absolute error ratios for alignment coefficient and density. The ratios are then averaged across the testing set. This metric can be seen as another type of image-level metric with collagen-specific domain knowledge, selected due to collagen structure being the biologically relevant focus of the SHG imaging.

The outputs of the supervised method (SISR network) and self-supervised method are compared with several traditional denoising methods, including median filter, wavelet-based denoising,⁶⁴ and total variation (TV)-based denoising.⁶⁵ After performing denoising, the resulting images are upscaled using bicubic interpolation to match the size of the ground truth images collected using a slower scanning rate and a higher scanning resolution. Note that the SISR network performs denoising and upscaling simultaneously in a single network and bicubic upscaling is not needed. Details of the quantitative evaluation of the outputs are summarized in Table 1. Some representative outputs generated by the baselines and DL-based methods are shown in Fig. 8.

Table 1

Comparison of different denoising and upscaling methods.

Fig. 8

Denoising of SHG images: (a) noisy and LR image collected using a faster scanning rate (0.5 MHz) at a $256\times 256$ scanning resolution; (b) median filter (radius = 1.5) followed by bicubic upscaling; (c) wavelet-based denoising followed by bicubic upscaling; (d) TV-based denoising followed by bicubic upscaling; (e) SSD followed by bicubic upscaling; (f) supervised denoising using the SISR network without GAN and perceptual loss; (g) supervised denoising using the SISR network with GAN and perceptual loss; and (h) clean and HR image collected using a slow scanning rate (0.1 MHz) at a $512\times 512$ scanning resolution.

The evaluation results show that for SHG image denoising, SSD compares favorably to traditional methods. For the SISR model, the network processed images better resemble the ground truth images with better PSNR, SSIM, and FID score and more similar collagen fiber alignment coefficient and density. The results also suggest that the addition of GAN and perceptual loss increases the visual quality of the generated images, with a smaller FID score and a higher SSIM.

The computation time of the SISR model is around $0.2\mathrm{s}/\text{frame}$, and the scanning time for each frame is reduced from $\sim 3$ to $\sim 0.15\mathrm{s}$ (a breakdown of run times can be found in Appendix C, Table 2). The superior performance in image restoration quality and the resulting acquisition acceleration demonstrate the high efficacy of DL-based methods for run-time LSM image processing.

5.1.

Detection Model

The CNN backbone used to train the detection models is ResNet18.⁴¹ The optimizer is Adam⁶⁷ with a learning rate of 0.0002 and cosine annealing learning scheduler⁶⁸ without warm restart. The batch size for the supervised model is 64. The loss function is cross-entropy loss and binary cross-entropy loss, respectively. The number of training epochs is 100.

5.2.

Runtime Image Enhancement Model

The image-to-image translation CNN backbone is described in Ref. 50, without the discriminator network, with the input and output channel being set to 1 for grayscale LSM images. The optimizer is Adam⁶⁷ with a learning rate of 0.0002 and cosine annealing learning scheduler⁶⁸ without warm restart. The batch size is 32 and the loss function is mean square error. The number of training epoch is 100. The input dropout rate is $p=0.1$ and the number of passes of the stochastic forward pass at inference is $1/p\times 64$.