2.1.

Overview

We propose an unsupervised method for performing the SR of clinical CT to the $\mu \mathrm{CT}$-scale, using unpaired clinical $\mathrm{CT}-\mu \mathrm{CT}$ images for training. We call our method SR-CycleGAN, since the structure of SR-CycleGAN is based on CycleGAN. The novelty of SR-CycleGAN consists of three aspects: (1) a network for SR, where the image-to-image translation networks of conventional CycleGAN were replaced by SR networks. The output SR image size is $2^k$-times ($k\in \mathbb{N}$) larger than the input LR image. (2) A loss function named MMSR loss, which ensures that the output SR image has the same structure as that of the input LR image. (3) An optimized network structure for reducing training time and achieving better quantitative/qualitative results.

For training, our method requires clinical CT images and $\mu \mathrm{CT}$ images. Inputs of the network are 2D CT images (LR images) cropped from clinical CT volumes. Outputs are corresponding SR images. It is noteworthy that the height and width of SR images are $2^k$-times ($k\in \mathbb{N}$) larger than those of the LR image.

2.2.

Conventional CycleGAN

This section explains conventional CycleGAN to better understand our SR-CycleGAN. CycleGAN³⁴ is an unsupervised image-to-image translation method based on deep generative models. It can learn to translate an image from a source domain $\mathbb{X}$ to a target domain $\mathbb{Y}$ in the absence of paired examples. The mathematical idea of CycleGAN is to obtain a generator $G_1$: $\mathbb{X}\to \mathbb{Y}$ and another generator $G_2$: $\mathbb{Y}\to \mathbb{X}$. At the training stage of CycleGAN, the generators $G_1$ and $G_2$ are trained simultaneously, and a loss named cycle-consistency loss is adopted to maintain cycle-consistency $G_2(G_1(\mathit{x}))\approx \mathit{x}$ and $G_1(G_2(\mathit{y}))\approx \mathit{y}$. Here, $\mathit{x}$ and $\mathit{y}$ are the images from domain $\mathbb{X}$ and domain $\mathbb{Y}$, respectively. The cycle-consistency loss is formulated as

The combination of adversarial losses and cycle-consistency loss is used for the unpaired image-to-image translation in CycleGAN.

2.3.

SR-CycleGAN

The conventional CycleGAN is not designed for SR. Since CycleGAN is an image-to-image translation network, output and input images are of the same size. However, in performing the SR of a given image, the output image’s size is larger than the input image, since the output image’s resolution is higher than that of the input. Furthermore, CycleGAN faces problems such as providing diverse outputs.³⁵ In the SR of medical images, we desire an output image that has the same anatomical structures as the input image. The SR result of a bronchus should still have the shape of a bronchus. Due to such constraints, we propose an SR network based on CycleGAN, and we named our method SR-CycleGAN. The structures of CycleGAN and SR-CycleGAN are shown in Fig. 2. Here, the input size and output size of CycleGAN are the same, but the output size is larger than the input in SR-CycleGAN.

Fig. 2

Structure comparison of (a) conventional CycleGAN and (b) SR-CycleGAN (our method). Conventional CycleGAN is an image-to-image translation network, where both its input and output are $256\times 256\text{pixels}$. Our method is an SR network. Its input size is $32\times 32\text{pixels}$, where its output size is $256\times 256\text{pixels}$.

2.3.1.

Network structure of SR-CycleGAN

The specific network structure of SR-CycleGAN is shown in Fig. 3. As shown in Fig. 3(a), we modified conventional CycleGAN’s image-to-image translation neural network (generator) $G_1$ to an SR neural network by removing downblocks/upblocks (definitions of downblocks/upblocks are given in Fig. 3) and adding pixel-shuffling layers. In conventional CycleGAN, the input and output of $G_1$ are of the same size. We input an image with a size of $n\times n$ pixels into $G_1$ of CycleGAN. Then we obtained the same-sized image of $n\times n$ pixels as output. On the other hand, by inputting an image with a size of $n\times n$ pixels into $G_1$ of SR-CycleGAN, we obtained an image of $2^{k}n\times 2^{k}n$ ($k\in \mathbb{N}$) pixels as output. The original network structure of generator $G_1$ has three “downblocks” at the network’s beginning, as shown in Fig. 3. Each downblock contains a convolution layer that scales down the image to 1/2 of its original size, following a batch normalization layer and an activation layer. If we input an image of $32\times 32\text{pixels}$ into three downblocks, we would obtain feature maps of $4\times 4\text{pixels}$. Such small feature maps would wash away the spatial features of the given image. Therefore, we remove the downblocks of the generator $G_1$. Upblocks consist of deconvolution layers that scale up the feature maps to their original size in generator $G_1$ of CycleGAN. Since we remove the downblocks in SR-CycleGAN, the feature maps are no longer scaled-down, and thus we also remove the upblocks in SR-CycleGAN. Finally, SR-CycleGAN is an SR network. Thus, we need to scale up feature maps at the end of the network to obtain the SR image. Use of a sub-pixel shuffling layer has been proven to reduce computational complexity, save computing time, and perform significantly better than using a deconvolution layer in SR operation.³⁶ Therefore, we add sub-pixel shuffling layers at the end of the network for scaling up feature maps to obtain the SR image as shown in Fig. 3(a).

Fig. 3

Modification from CycleGAN to SR-CycleGAN. The modifications of $G_1$ are as follows. (1) Removal of downblocks to maintain spatial information of the input image as shown in (a). (2) Removal of upblocks because feature maps no longer need them for scaling up as shown in (b). (3) Addition of sub-pixel shuffling layers at the end of the network for scaling up feature maps to the SR image. $G_2$ is a generator that shrinks an input image of $256\times 256\text{pixels}$ into an image of $32\times 32\text{pixels}$. We added three downsample blocks (downblocks) to generator $G_2$. The specific structure of each block is shown in (c).

In SR-CycleGAN, generator $G_2$ is an inverse function of generator $G_1$. Since generator $G_1$ scales up an input image to an SR image, we modified the generator $G_2$ to scale down an HR image to an LR image. In conventional CycleGAN, an image with a size of $2^{k}n\times 2^{k}n$ ($k\in \mathbb{N}$) pixels is input into $G_2$, and an image of the same size is produced as output. On the other hand, in generator $G_2$ of SR-CycleGAN, we obtain an image of $n\times n$ as output from an input image of size $2^{k}n\times 2^{k}n$ ($k\in \mathbb{N}$). We added downblocks consisting of downsampling layers at the end of generator $G_2$ to scale down the feature maps, as shown in Fig. 3(b).

2.3.2.

Multi-modality super-resolution loss in SR-CycleGAN

There are two important factors in the SR of clinical CT images. One is anatomical structure, and the other is intensity distribution. Here, we explain the relationship between anatomical structure and intensity distribution. Structures such as arteries, bronchi, and alveoli are anatomical structures. Intensity distribution describes how a certain tissue has a certain intensity (grayscale). The intensity of clinical CT is described by the Hounsfield scale, and a specific substance such as bone has a specific intensity of $+300$ to $+1900$.³⁷ On the other hand, the intensity of $\mu \mathrm{CT}$ changes with every scan, so the intensity of a specific substance varies slightly at each time of scan.

The same anatomical structures have totally different intensity distributions between clinical CT and $\mu \mathrm{CT}$. For instance, in clinical CT images, the intensities of blood vessels and bronchus walls are around 0 and $-500$ Hounsfield units (H.U.). In $\mu \mathrm{CT}$ images, the intensities of blood vessels and bronchus walls are around 15,000 and 11,000 in the scanner used in our experiments. The intensity distribution of $\mu \mathrm{CT}$ focuses on a range of about [2000, 15,000] as shown in Fig 4(b), while the intensity of a lung’s clinical CT is distributed relatively uniformly in the range $[-\mathrm{1000,500}]$ as shown in Fig. 4(a). Even if we normalize the intensities of both $\mu \mathrm{CT}$ and clinical CT to the range $[-\mathrm{1,1}]$, the histograms of the two intensity distributions are still very different.

Fig. 4

The intensity distribution of clinical CT image and $\mu \mathrm{CT}$ image. Intensity of clinical CT is described by the Hounsfield scale, and a specific substance such as bone has a specific intensity of $+300\sim +1900$.³⁷ The intensity of $\mu \mathrm{CT}$ is not described by the Hounsfield scale, and a specific substance’s intensity varies slightly at each time of scan. An example of a clinical CT image and its intensity distribution is shown in (a). An example of a $\mu \mathrm{CT}$ image and its intensity distribution is shown in (b). Histogram at right side: $x$ axis is the intensity value of a particular pixel, while y-axis is the percentage of corresponding intensity. For the blue curve of the graph (a), around 0 H.U. on the $x$ axis, the $y$ axis is around 0.11. This implies that the number of voxels with an intensity of $-100\sim 0$ H.U. of clinical CT is around 11% of the total number of voxels. It is noteworthy that for clinical CT, we count the number of voxels by every one hundred, but since the intensity range of $\mu \mathrm{CT}$ is huge, we count the number of voxels here by every one thousand. The histograms illustrate that the intensity distributions of clinical and $\mu \mathrm{CT}$ are very different, which is one reason why CycleGAN without the proposed MMSR loss failed to perform SR of clinical CT using $\mu \mathrm{CT}$ images.

For the SR of medical images, a drastic change in image appearances may mislead clinicians. We need anatomical structures such as blood vessels and bronchi in clinical CT images (LR image) to maintain their original size and shape after SR. In addition, we have to ensure that the intensity distribution of the clinical CT’s SR result stays close to that of the original clinical CT image.

The loss function used in conventional CycleGAN does not ensure that input LR and output SR images have the same anatomical structures and intensity distribution. If we only modify the network structure of CycleGAN as shown in Sec. 2.3.1, the modified network outputs SR images with totally different intensity and anatomical structures from the input LR image. The objective of conventional CycleGAN is to output images close to the target domain instead of the source domain. In clinical CT image SR, the source domain is the LR domain (clinical CT) and the target domain is the SR domain ($\mu \mathrm{CT}$). Therefore, CycleGAN with conventional loss terms outputs SR images with no similarity to the input LR image. Loss terms that guarantee that the output SR image has the same anatomical structures and intensity distribution as the input LR image are desired.

We propose a novel loss function named MMSR loss as shown in Fig. 5. The MMSR loss contains the following terms: (1) structural similarity (SSIM) loss, (2) downsample loss, and (3) upsample loss. As shown in Fig. 5, the downsample loss and upsample loss ensure that the SR image has a similar intensity distribution to that of the input LR image, and the SSIM loss ensures that the SR image has similar anatomical structures to those of the input LR image. Consequently, we use the MMSR loss to train SR-CycleGAN.

Fig. 5

Illustration of proposed loss terms. SSIM loss and downsample loss between input clinical CT image $\mathit{x}$ and output SR image ${\mathit{x}}^{\mathrm{SR}}$ are shown in (a). We use the average pooling function $f_{\downarrow }()$ to downsample ${\mathit{x}}^{\mathrm{SR}}$ to the same size of input $\mathit{x}$. Then we calculate the SSIM loss and downsample loss of $\mathit{x}$ and $f_{\downarrow }({\mathit{x}}^{\mathrm{SR}})$. These losses are calculated to optimize the parameters in generator $G_1$. SSIM loss and upsample loss between input $\mu \mathrm{CT}$ image $\mathit{y}$ and output downsample image ${\mathit{y}}^{\mathrm{LR}}$ are shown in (b). We use the nearest upsampling function $f_{\uparrow }()$ to upsample the generated clinical CT-like low-resolution image ${\mathit{y}}^{\mathrm{LR}}$. Then we calculate the SSIM loss and downsample loss of $\mathit{y}$ and $f_{\uparrow }({\mathit{y}}^{\mathrm{LR}})$. These losses are calculated to optimize the parameters in generator $G_2$.

SSIM loss

The first loss term we propose is named SSIM loss. SSIM³⁸ is an indicator that evaluates the structure similarity of two images. SSIM between two images is defined as

Eq. (3)

$$\mathrm{SSIM}(\mathit{a},\mathit{b})=\frac{({\mu }_{\mathit{a}}{\mu }_{\mathit{b}}+C_1)(2{\sigma }_{\mathit{a}\mathit{b}}+C_2)}{({\mu }_{\mathit{a}}^2+{\mu }_{\mathit{b}}^2+C_1)({\sigma }_{\mathit{a}}^2+{\sigma }_{\mathit{b}}^2+C_2)},$$

where ${\mu }_{\mathit{a}}$ and ${\mu }_{\mathit{b}}$ are the average intensity of given images $\mathit{a}$ and $\mathit{b}$, respectively. ${\sigma }_{\mathit{a}}$ and ${\sigma }_{\mathit{b}}$ are the variance of given images $\mathit{a}$ and $\mathit{b}$, respectively. ${\sigma }_{\mathit{ab}}$ is the covariance of given images $\mathit{a}$ and $\mathit{b}$. $C_1$ and $C_2$ are constant numbers included to avoid instability. Based on this equation, we set the loss term named SSIM loss as

Eq. (4)

$${\mathcal{L}}_{\mathrm{S}}(\mathit{x},{\mathit{x}}^{\mathrm{SR}})={\mathbb{E}}_{\mathit{x}\sim \mathbb{X}}[1-\mathrm{SSIM}(\mathit{x},f_{\downarrow }({\mathit{x}}^{\mathrm{SR}}))],$$

where $\mathit{x}$ is an input clinical CT image, ${\mathit{x}}^{\mathrm{SR}}$ is the SR image, $\mathbb{X}$ is the domain of clinical CT images, and $f_{\downarrow }()$ is the average pooling³⁹ function. Average pooling calculates the average value for patches of a feature map and uses it to create a downsampled (pooled) feature map.⁴⁰ $f_{\downarrow }()$ rescales a given image to $1/n$ $(n\in \mathbb{R})$ of its original size by width and height. We use $1-\mathrm{SSIM}(\mathit{x},f_{\downarrow }({\mathit{x}}^{\mathrm{SR}}))$ as the basis of this loss term, since we desire the SSIM of $\mathit{x}$ and $f_{\downarrow }({\mathit{x}}^{\mathrm{SR}})$ to be close to 1.

Downsample loss

To prevent a change of intensity in the CT image after SR, we propose another loss term named the downsample loss, which is written as

Eq. (5)

$${\mathcal{L}}_{\mathrm{D}}(\mathit{x},{\mathit{x}}^{\mathrm{SR}})={\mathbb{E}}_{\mathit{x}\sim \mathbb{X}}{\Vert (\mathit{x},f_{\downarrow }({\mathit{x}}^{\mathrm{SR}}))\Vert }_2^2,$$

where ${\Vert ·\Vert }_2^2$ is the square of the $l^2$-norm, $\mathit{x}$ is the input clinical CT (LR) image, and ${\mathit{x}}^{\mathrm{SR}}$ is the SR image. We call this the downsample loss because it is calculated using the downsampled SR image $f_{\downarrow }({\mathit{x}}^{\mathrm{SR}})$ and the input LR image $\mathit{x}$. Since the downsample loss calculates the pixel-wise loss between the SR and LR images, this loss can prevent the SR image ${\mathit{x}}^{\mathrm{SR}}$ from deforming and changing of its intensity in relation to the LR image.

Upsample loss

The third proposed loss term is named upsample loss. As shown in Fig. 5(b), in SR-CycleGAN, there is another generator $G_2$ that can translate a given $\mu \mathrm{CT}$ image $\mathit{y}$ into a clinical CT-like image ${\mathit{y}}^{\mathrm{LR}}$. By the same principle as downsample loss, to prevent a change in the intensity between $\mathit{y}$ and ${\mathit{y}}^{\mathrm{LR}}$, the upsample loss is formulated as

Eq. (6)

$${\mathcal{L}}_{\mathrm{U}}(\mathit{y},{\mathit{y}}^{\mathrm{LR}})={\mathbb{E}}_{\mathit{y}\sim \mathbb{Y}}{\Vert (\mathit{y},f_{\uparrow }({\mathit{y}}^{\mathrm{LR}}))\Vert }_2^2,$$

where $f_{\uparrow }()$ is the nearest upsampling function. The nearest upsampling function selects the value of the nearest pixels of a feature map, and then assigns this value to new pixels to create an upsampled feature map. $f_{\uparrow }()$ rescales a given image to $k$ ($k\in \mathbb{R}$) times its original size by width and height, and $\mathbb{Y}$ is the domain of $\mu \mathrm{CT}$ images $\mathit{y}$. We call this the upsample loss because it is calculated from the $l^2$ norm between the upsampled fake clinical CT $f_{\uparrow }({\mathit{y}}^{\mathrm{LR}})$ and the original $\mu \mathrm{CT}$ $\mathit{y}$.

Adding MMSR loss in SR-CycleGAN

The MMSR loss consists of SSIM loss, downsample loss, and upsample loss. The MMSR loss is formulated as

Eq. (7)

$$\begin{gathered} {\mathcal{L}}_{\mathrm{MMSR}}(\mathit{x},\mathit{y},{\mathit{y}}^{\mathrm{LR}},{\mathit{x}}^{\mathrm{SR}})={\lambda }_1{\mathcal{L}}_{\mathrm{S}}(\mathit{x},f_{\downarrow }{\mathit{x}}^{\mathrm{SR}})) \\ +{\lambda }_2{\mathcal{L}}_{\mathrm{S}}(\mathit{y},f_{\uparrow }({\mathit{y}}^{\mathrm{LR}})) \\ +{\lambda }_3{\mathcal{L}}_{\mathrm{D}}(\mathit{x},f_{\downarrow }({\mathit{x}}^{\mathrm{SR}})) \\ +{\lambda }_4{\mathcal{L}}_{\mathrm{U}}(\mathit{y},f_{\uparrow }({\mathit{y}}^{\mathrm{LR}})), \end{gathered}$$

where ${\mathcal{L}}_{\mathrm{S}}(\mathit{x},f_{\downarrow }({\mathit{x}}^{\mathrm{SR}}))$ is the SSIM loss between the input clinical image $\mathit{x}$ and the output SR image ${\mathit{x}}^{\mathrm{SR}}$. ${\mathcal{L}}_{\mathrm{S}}(\mathit{y},f_{\uparrow }({\mathit{y}}^{\mathrm{LR}}))$ is the SSIM loss between the $\mu \mathrm{CT}$ image $\mathit{y}$ and the generated clinical CT-like image ${\mathit{y}}^{\mathrm{LR}}$. ${\mathcal{L}}_{\mathrm{D}}(\mathit{x},f_{\downarrow }({\mathit{x}}^{\mathrm{SR}}))$ is the downsample loss of $\mathit{x}$ and ${\mathit{x}}^{\mathrm{SR}}$. ${\mathcal{L}}_{\mathrm{U}}(\mathit{y},f_{\uparrow }({\mathit{y}}^{\mathrm{LR}}))$ is the upsample loss of $\mathit{y}$ and ${\mathit{y}}^{\mathrm{LR}}$. $f_{\downarrow }()$ is the average pooling function that scales up a given image. $f_{\uparrow }()$ is the nearest upsampling function that scales down a given image. ${\lambda }_1$, ${\lambda }_2$, ${\lambda }_3$, and ${\lambda }_4$ are weights. We add the proposed MMSR loss as an additional loss term into the proposed SR-CycleGAN. We formulate the total loss function of SR-CycleGAN as

Eq. (8)

$$\begin{gathered} {\mathcal{L}}_{\text{Total}}={\lambda }_1{\mathcal{L}}_{\mathrm{S}}(\mathit{x},f_{\downarrow }({\mathit{x}}^{\mathrm{SR}})) \\ +{\lambda }_2{\mathcal{L}}_{\mathrm{S}}(\mathit{y},f_{\uparrow }({\mathit{y}}^{\mathrm{LR}})) \\ +{\lambda }_3{\mathcal{L}}_{\mathrm{D}}(\mathit{x},f_{\downarrow }({\mathit{x}}^{\mathrm{SR}})) \\ +{\lambda }_4{\mathcal{L}}_{\mathrm{U}}(\mathit{y},f_{\uparrow }({\mathit{y}}^{\mathrm{LR}})) \\ +{\lambda }_5{\mathcal{L}}_{\mathrm{GAN}}({\mathit{x}}^{\mathrm{SR}},\mathit{y}) \\ +{\lambda }_6{\mathcal{L}}_{\mathrm{GAN}}({\mathit{y}}^{\mathrm{LR}},\mathit{x}) \\ +{\lambda }_7{\mathcal{L}}_{\mathrm{cyc}}(\mathit{x},G_2({\mathit{x}}^{\mathrm{SR}}),\mathit{y},G_1({\mathit{y}}^{\mathrm{LR}})), \end{gathered}$$

where ${\mathcal{L}}_{\mathrm{GAN}}({\mathit{x}}^{\mathrm{SR}},\mathit{y})$ and ${\mathcal{L}}_{\mathrm{GAN}}({\mathit{y}}^{\mathrm{LR}},\mathit{x})$ are GAN loss, and ${\mathcal{L}}_{\mathrm{cyc}}(\mathit{x},G_2({\mathit{x}}^{\mathrm{SR}}),\mathit{y},G_1({\mathit{y}}^{\mathrm{LR}}))$ is cycle-consistency loss proposed in the conventional CycleGAN described in Sec. 2.2. ${\lambda }_5$, ${\lambda }_6$, and ${\lambda }_7$ are weights. By adding the MMSR loss to CycleGAN, we successfully performed the SR of clinical CT of lung cancer patients to the $\mu \mathrm{CT}$ level, while conventional CycleGAN failed to perform SR.

2.4.

Training and Inference of SR-CycleGAN

In the training phase, the input of generator $G_1$ is a clinical CT image with the size of $n\times n$ pixels. We denote the clinical CT image as $\mathit{x}$. The generator $G_1$ generates an SR image ${\mathit{x}}^{\mathrm{SR}}=G_1(\mathit{x})$ with a size of $2^{k}n\times 2^{k}n$ pixels. On the other hand, a $\mu \mathrm{CT}$ image $\mathit{y}$ with a size of $2^{k}n\times 2^{k}n$ pixels is input into the generator $G_2$. The generator $G_2$ generates a clinical CT-like image ${\mathit{y}}^{\mathrm{LR}}=G_2(\mathit{y})$ of $n\times n$ pixels from the $\mu \mathrm{CT}$ image $\mathit{y}$ of $2^{k}n\times 2^{k}n$ pixels. The loss of the entire SR-CycleGAN is calculated from $\mathit{x}$, ${\mathit{x}}^{\mathrm{SR}}$, $\mathit{y}$, and ${\mathit{y}}^{\mathrm{LR}}$. Then the loss is used for to optimize the network.

For inference, we only use the trained generator $G_1$. We extracted images of size $n\times n$ pixels from clinical CT and input them into the trained network $G_1$. The output is SR images of size $2^{k}n\times 2^{k}n$ pixels.

3.1.

Datasets

In our experiments, we newly built a dataset containing ten $\mu \mathrm{CT}$ volumes and eight clinical CT volumes. The clinical CT volumes were scanned by a clinical CT scanner (SOMATOM Definition Flash, Siemens Inc., Munich, Germany). The resolution of the clinical CT volumes was $0.625\times 0.625\times 0.6{\mathrm{mm}}^3/\text{voxel}$. The size of the clinical CT volumes was $512\times 512\times 435\sim 554\text{voxels}$. The $\mu \mathrm{CT}$ volumes were scanned by a $\mu \mathrm{CT}$ scanner (inspeXio SMX-90 CT Plus, Shimadzu Inc., Kyoto, Japan) as shown in Fig. 6(a). The lung cancer specimens were fixed by Heitzman’s method⁴¹ as shown in Fig. 6(b). Lung specimens were scanned at isotropic resolutions of $42\sim 52\times 42\sim 52\times 42\sim 52\mu {\mathrm{m}}^3/\text{voxel}$. The size of the $\mu \mathrm{CT}$ volumes was $1024\times 1024\times 545\sim 983\text{voxels}$. We trained SR-CycleGAN using five clinical CT volumes and five corresponding $\mu \mathrm{CT}$ volumes of lung cancer specimens. We evaluated the SR-CycleGAN qualitatively on three clinical CT volumes and quantitatively on five $\mu \mathrm{CT}$ volumes. These clinical and $\mu \mathrm{CT}$ volumes were not used for training.

Fig. 6

Our $\mu \mathrm{CT}$ scanner and a sample of lung specimen. $\mu \mathrm{CT}$ scanner (inspeXio SMX-90 CT Plus, Shimadzu Inc., Kyoto, Japan) is shown in (a). Resected lung cancer specimen from human lung cancer patient is shown in (b).

3.2.

Preprocessing

Chest clinical CT images have various tissues outside the lungs that are not appropriate for our experiments, such as bones, muscles, esophagus, etc. We first segmented lung regions from clinical CT chest images. We conducted region growing⁴² to obtain a coarse segmentation mask of the lung and performed morphological operations to fill the holes in the coarse segmentation mask.

$\mu \mathrm{CT}$ images also require a target region restriction. In our experiments, lung specimens were placed in a plastic cylinder and put into the $\mu \mathrm{CT}$ scanner for scanning. Therefore, parts of the plastic cylinder are shown in the $\mu \mathrm{CT}$ images. Since the plastic cylinder is not suitable for our experiment, we manually cropped lung regions from the $\mu \mathrm{CT}$ images, and only used the lung regions for the experiment.

In addition, normalization of the intensities of both clinical CT and $\mu \mathrm{CT}$ images was required. We normalized both the intensity of $\mu \mathrm{CT}$ and clinical CT to the range $[-\mathrm{1,1}]$. In clinical CT, the intensity of a tissue is represented using the Hounsfield scale, with water having a value of 0 H.U., tissues denser than water having positive values, and tissues less dense than water having negative values.⁴³ In $\mu \mathrm{CT}$, the intensity is not represented by Hounsfield scale. The intensity range of the clinical CT volume was about 3500 H.U. (intensity of air is around $-1000$ H.U. and intensity of bone is around 2500 H.U.), but the scale of the $\mu \mathrm{CT}$ volume was about 16,000 (intensity of air is around $-1000$ to 0, and cancer is around 15,000). For clinical CT, we normalized the intensity in this way: For intensity larger than 2500 H.U. (larger than the bone intensity), we set the intensity to 2500 H.U. We also set voxels that have intensity smaller than $-1000$ H.U. to $-1000$ H.U. For $\mu \mathrm{CT}$, we set voxels that have intensity higher than 15,000 (higher than cancer) to 15,000 and set voxels that have intensity smaller than 0 to 0. Finally, the intensities of both clinical CT and $\mu \mathrm{CT}$ images were compressed to $[-\mathrm{1,1}]$.

3.3.

Parameter Settings

3.3.3.

Evaluation methods

For qualitative evaluation, we utilized three clinical CT volumes. We cropped clinical CT images of size $32\times 32\text{pixels}$ from three clinical CT volumes and input the clinical CT images into generator $G_1$ of trained SR-CycleGAN. Then, we obtained SR images of size $256\times 256\text{pixels}$. For demonstrating the effectiveness of network modification and MMSR loss of SR-CycleGAN, we compared SR-CycleGAN with conventional CycleGAN. Since input and output of CycleGAN is of the same size, CycleGAN could not be applied directly for SR. Therefore, we add upblocks into CycleGAN’s generator $G_1$ to ensure output of $G_1$ is eight times larger than input (by width and height). We name this CycleGAN as “CycleGAN with upblocks.” We also conducted ablation experiments to verify the effectiveness of network modification.

For quantitative evaluation, we proposed a novel quantitative evaluation method. In previous supervised SR studies,⁴⁵ quantitative evaluations were often conducted by comparing the output SR image with its HR counterpart. Therefore, paired LR images (clinical CT images) and HR images ($\mu \mathrm{CT}$ images) were required for quantitative evaluations. Since we could not obtain paired clinical $\mathrm{CT}/\mu \mathrm{CT}$ images, we conducted an alternative approach: First, we used bicubic interpolation³¹ to downsample $\mu \mathrm{CT}$ images to 1/8 of their original size to simulate clinical CT images (In image processing, bicubic interpolation is used for interpolating data points on a 2D regular grid. Bicubic interpolation considers 16 pixels ($4\times 4$) around the pixel to be interpolated and calculates a weighted addition of these 16 pixels as the new pixel.). For a given $\mu \mathrm{CT}$ image of $256\times 256\text{pixels}$, we performed bicubic downsampling of the $\mu \mathrm{CT}$ image to obtain an image size of $32\times 32\text{pixels}$ and then input it into trained $G_1$ to obtain a $256\times 256\text{pixel}$ SR output. We compared the SR output with the original $\mu \mathrm{CT}$ images using evaluation metrics such as peak signal-noise ratio (PSNR).⁴⁶ It is noteworthy that $G_1$ is trained by clinical CT and $\mu \mathrm{CT}$ images as explained in Sec. 3.3.1. We used five $\mu \mathrm{CT}$ cases of 1544 images for quantitative evaluation.

We compared the following networks. Network1: CycleGAN with upblocks (no MMSR loss, no network modification, only upblocks for a larger output image). Network2: CycleGAN with network modification (sub-pixel shuffling layers but no MMSR loss). Network3: SR-CycleGAN with downblocks (with MMSR loss, no sub-pixel shuffling layers). Network4: Proposed SR-CycleGAN (with MMSR loss and sub-pixel shuffling).

We also quantitatively evaluated how sub-pixel shuffling layers reduce training time. Before adding sub-pixel shuffling layers in generator $G_1$, we used upblocks to upscale the feature maps to a larger size. Figure 7 shows a comparison of $G_1$ with/without pixel-shuffling layers. We used 2000 patches cropped from clinical CT images of $32\times 32\text{pixels}$ and 2000 patches cropped from $\mu \mathrm{CT}$ images of SR-CycleGAN for training.

Fig. 7

To prove that the sub-pixel shuffling layers actually reduce computing time, we performed experiments on two kinds of generator $G_1$: (a) generator $G_1$ with upblocks and (b) generator $G_1$ with sub-pixel shuffling layers. We extracted 2000 patches for training on Nividia Tesla V100 (32 GB memory). (a) needed 491 s for training in each epoch, while (b) needed 353 s in each epoch.

3.4.

Comparison of Results

SR results of SR-CycleGAN were compared with CycleGAN with upblocks in Fig. 8. Furthermore, for evaluating the effectiveness of removing downblocks and introducing sub-pixel shuffling layers, we also evaluated SR-CycleGAN with/without removing downblocks and with/without sub-pixel shuffling layers as shown in Fig. 9.

Fig. 8

SR results of clinical CT images from one case. Rows (1) and (3) are images cropped from blood vessels and lung field region. Rows (2) and (4) are images cropped from the bronchus and blood vessels region. Column (a) are original clinical CT images. Column (b) and (c) are results of “CycleGAN with upblocks” and our method, respectively. We can obtain that SR-CycleGAN output reliable SR results, while CycleGAN with upblocks (no MMSR loss, no network modification, only upblocks for larger output image) output results that do not have similarity with the input image. As pointed by red arrows, blood vessels and bronchus in SR images of CycleGAN with upblocks severely deformed or disappeared, while blood vessels and bronchus in SR-CycleGAN’s SR images have sharp edges and same shape as in LR images.

Fig. 9

Comparison of SR-CycleGAN before/after removing downblocks and adding sub-pixel shuffling layers. Rows (1), (2), and (3) are CT images of the bronchus and blood vessel region. Row (4) has CT images of the tumor and bronchus region. Column (a) are clinical CT images. Column (b) and (c) are results of “SR-CycleGAN with downblocks” and “SR-CycleGAN with sub-pixel shuffling” respectively. After removing downblocks and adding sub-pixel shuffling layers, SR-CycleGAN performed better qualitatively. As indicated by the red arrows, results of SR-CycleGAN with downblocks (SR-CycleGAN with downblocks and before adding sub-pixel shuffling layers) have many artifacts, and the edges of the bronchus and blood vessels look discontinuous. On the other hand, these defects do not appear in the results of SR-CycleGAN.

3.4.1.

Qualitative evaluation

We show the cropped part of the SR images obtained by the SR-CycleGAN in Fig. 8(c). The results of CycleGAN with upblocks are shown in Fig. 8(b). In SR results of SR-CycleGAN, lung anatomies, such as the bronchus, appear more clearly than the original clinical CT images as indicated by red arrows in Fig. 8(c). CycleGAN with upblocks (no network modification except adding upblocks and no MMSR loss) only produced results that have no similarity with the input LR image (clinical CT image). Important anatomical structures such as the blood vessels and bronchus disappeared, as indicated by red arrows in Fig. 8(b). The results demonstrate that the proposed SR-CycleGAN is suitable for SR of clinical CT images.

The results of “SR-CycleGAN with downblocks”⁴⁷ (SR-CycleGAN with MMSR loss but without network modification) are shown in Fig. 9(b), which seems noisy, and the edge of the blood vessel and bronchus has many artifacts indicated by red arrows. The results of SR-CycleGAN are shown in Fig. 9(c), which is clearer and noiseless compared with Fig. 9(b).

To observe SR results from a larger scale, we illustrate both clinical CT images of the whole lung region and images cropped from the lung region before and after SR in Fig. 10.

Fig. 10

To observe SR results from larger scale, this image illustrates both clinical CT images of whole lung region and images cropped from lung region before and after SR. (a) CT image extracted from axial axis. (b) Another CT image extracted from axial axis. In (a) and (b), edges of arteries and bronchus (red arrows) are smoother and clearer after SR.

3.4.2.

Quantitative evaluation

The SR results and quantitative evaluation results are shown in Fig. 11 and Table 2. We used PSNR and SSIM⁴⁶ for quantitative evaluation. Table 2 shows that the proposed SR-CycleGAN performed quantitatively better than other methods, with the highest PSNR and SSIM.

Fig. 11

Qualitative results of SR-CycleGAN. SR results of SR-CycleGAN with downblocks are shown in (b), with PSNR of 16.48 dB. SR results of SR-CycleGAN with sub-pixel shuffling layer and without downblocks are shown in (c), with PSNR of 17.71 dB. Column (a) and (d) are LR images and corresponding HR images, respectively. We used bicubic downsampling³¹ to rescale $\mu \mathrm{CT}$ images (HR image) to 1/8 of their original sizes to simulate clinical CT images (LR images), and then input the downsampled image into trained SR-CycleGAN’s generator $G_1$. It is noteworthy that the higher PSNR indicates a better result.

Table 2

Quantitative evaluation of our methods. Network1: CycleGAN with upblocks (no MMSR loss, no network modification, only upblocks for larger output image). Network2: CycleGAN with network modification (sub-pixel shuffling layers but no MMSR loss). Network3: SR-CycleGAN with downblocks (with MMSR loss, no sub-pixel shuffling layers). Network4: proposed SR-CycleGAN (with MMSR loss and sub-pixel shuffling). Bold values are the highest.

	Network1	Network2	Network3	SR-CycleGAN (our method)
PSNR	13.64	15.39	16.48	17.71
SSIM	0.05	0.37	0.44	0.54

We also evaluated how sub-pixel shuffling layers reduce training time. SR-CycleGAN without sub-pixel shuffling layers needs 491 s for training per epoch (2000 patches per epoch). After replacing upblocks with sub-pixel shuffling layers, the entire network needs 353 s for training per epoch. Thus, training time was significantly reduced. The network was trained on Nvidia Tesla V100 (32 GB memory).

3.5.

Ablation Studies

For accessing the effectiveness of different components of our method, we performed ablation studies. On top of baseline (CycleGAN with upblocks), we progressively added network modification and the MMSR loss function. Further, to clear effectiveness of each component of MMSR loss, we also analyzed each term in MMSR loss separately. Experiments showed that our method with all proposed components performed best quantitatively and qualitatively.

3.5.1.

Effectiveness of network modification

We first analyzed the effect of network modification. As network modification, we removed downblocks and added pixel-shuffling layers to a conventional CycleGAN’s generator $G_1$. Network modification avoided the need to encode the input image into smaller feature maps, thus preserving spatial information while performing SR. Additionally, it also reduced training and referencing time. With network modification, PSNR increased by 1.75 dB and SSIM increased by 0.32 compared to the baseline (CycleGAN with upblocks). The qualitative results of baseline and baseline with network modification are shown as condition A and condition C, respectively, in Fig. 12; images of the latter were qualitatively better than those of the former. Quantitative results of network modification are shown in Table 3. In Table 3, the PSNR and SSIM score of condition C (baseline with network modification) are higher than those of condition A (baseline). Therefore, network modification is required in our method.

Fig. 12

Qualitative results of ablation studies. We chose five combinations of each proposed component and illustrate the qualitative results of each combination in this figure. The method with all components (our method) achieved the highest PSNR and SSIM score. A, C, H, I, and M (our method) correspond to the “condition” column of Table 3. Upper: whole images. Lower: zoom-in on the regions in the red boxes for better comparison.

Table 3

Ablation studies and quantitative results. SSIM loss 1 is LS(x,f↓(xSR)) and SSIM loss 2 is LS(y,f↑(yLR)). Applying network modification and all loss terms simultaneously obtains the highest PSNR and SSIM means such a component is not utilized, and F0FC means such a component is utilized. Bold values are the highest.

Condition	Network modification	Upsample loss	Downsample loss	SSIM loss 1	SSIM loss 2	PSNR	SSIM
A	—	—	—	—	—	13.64	0.05
B	—	✓	✓	✓	✓	16.48	0.44
C	✓	—	—	—	—	15.39	0.37
D	✓	✓	—	—	—	15.83	0.40
E	✓	—	✓	—	—	16.53	0.42
F	✓	—	—	✓	—	15.13	0.27
G	✓	—	—	—	✓	14.18	0.25
H	✓	✓	✓	—	—	15.92	0.40
I	✓	—	—	✓	✓	16.92	0.49
J	—	✓	✓	✓	✓	16.48	0.44
K	✓	—	✓	✓	✓	15.78	0.46
L	✓	✓	—	✓	✓	17.70	0.50
M (our method)	✓	✓	✓	✓	✓	17.71	0.54

3.6.

Comparison with Recent Baselines

We compared our method with three recent SR methods. We first compared our method with a recent unsupervised baseline named CinCGAN.³² CinCGAN first utilizes cycle-in-cycle network structure to map a noisy and blurry LR image to a noise-free LR image. Then the noise-free LR image is upsampled with a pre-trained deep SR model. CinCGAN is trained with LR-HR images in an end-to-end manner. The trained CinCGAN is used for performing SR of a given LR image.³² We also compared our method with a newly proposed SOTA unsupervised SR method named pseudo-SR,⁴⁹ and a widely used supervised SR method named ESRGAN.²³ Pseudo-SR is an SR method consists of an unpaired kernel/noise correction network and a pseudo-paired SR network. The correction network removes noise and adjusts the blurring kernel⁵⁰ of the input LR image. Then the pseudo-paired SR network upscales the corrected clean LR image.⁴⁹ ESRGAN is a supervised SR method utilizing newly proposed loss terms such as adversarial loss and perceptual loss, and the residual-in-residual dense block into SR network.⁵¹ We did not have paired clinical CT (LR) and $\mu \mathrm{CT}$ (HR) images. Therefore, we trained ESRGAN with unpaired LR-HR images. The results of our method and these recent baselines were shown in Fig. 13. As shown in the red boxes in Fig. 13, our method output SR images close to the HR images (ground truth). Recent SR baselines output SR images quite different from the HR images (ground truth). The PSNR and SSIM of our method were the highest among all methods, as shown in Table 4. We also compared our method’s inference time, training time, and parameter size with recent baselines in Table 5. As shown in the Table 5, training time for one epoch was the shortest with our method, and the number of network parameters was the smallest.

Fig. 13

Qualitative comparison between our method and recent baselines on clinical $\mathrm{CT}-\mu \mathrm{CT}$ dataset. We compared our method with a recent supervised baseline (ESRGAN⁵¹) and two recent unsupervised baselines (pseudo-SR⁴⁹ and CinCGAN).³² Our method output convincing SR results, while recent SR baselines output SR images quite different from the HR images (ground truth). Upper: whole images. Lower: zoom-in on regions in the red boxes for better comparison.

Table 4

Quantitative comparison between our method and recent baselines. Our method has the highest PSNR and SSIM score. These results were computed on the clinical CT−μCT dataset. Bold values are the highest.

Table 5

Comparison of training time, inference time and number of parameters between our method and recent baselines. Our method has the shortest average training time and the fewest parameters compared to recent SR baselines ESRGAN,51 pseudo-SR,49 and CinCGAN.32 Bold values are the highest.

3.7.

Experimental Results on COVID-19 Lung CT Segmentation Challenge—2020 Dataset

We also performed an experiment with an additional benchmark CT dataset to examine whether our method could perform SR of commonly used medical images (such as CT images). We chose the COVID-19 Lung CT Segmentation Challenge—2020 dataset.⁵² This dataset has 249 cases collected from patients of different hospitals, countries, ages, and genders. Here, 199 cases were for training and 50 cases were for testing. We chose $4\times$ SR (width and length of an output image are four-times those of an input image). Input LR image size was $48\times 48\text{pixels}$, and output SR image size was $192\times 192\text{pixels}$. We compared our method with recent baselines: unsupervised SR methods CinCGAN³² and pseudo-SR,⁴⁹ and a supervised method ESRGAN.⁵¹ Qualitative results are shown in Fig. 14, and quantitative results are shown in Table 6. Our method outperformed these recent baselines quantitatively as shown in Table 6. It could output clear images and reconstruct important anatomical structures such as vessels and bronchi. Results of recent baselines are blurred (CinCGAN and pseudo-SR) or unreasonable (ESRGAN) in Fig. 14. The experimental results prove that our method is effective on commonly used medical images.

Fig. 14

Experimental result on COVID-19 Lung CT Lesion Segmentation Challenge—2020 dataset.⁵² We compared our method with ESRGAN,⁵¹ pseudo-SR,⁴⁹ and CinCGAN.³² It is noteworthy that because our method is trained with unpaired LR-HR images pairs, we also train ESRGAN with unpaired LR-HR images. ESRGAN output unreasonable results. Pseudo-SR and CinCGAN output blurry and noisy results. On the other hand, our method output convincing results. Upper: whole images from the axial axis. Lower: zoom-in on regions in the red boxes for better comparison.

Table 6

Quantitative comparison between our method and recent baselines on the COVID-19 Lung CT Segmentation Challenge—2020 dataset.52 Bold values are the highest.

4.1.

Unsupervised SR of Clinical CT Utilizing $\mu \mathrm{CT}$ Data

To the best of our knowledge, our method is the first method to perform SR on clinical CT to the $\mu \mathrm{CT}$ scale without a corresponding HR image as ground truth. The method is also the first to perform SR of clinical CT utilizing $\mu \mathrm{CT}$ data. MMSR loss and modification of networks enabled SR-CycleGAN to perform SR by forcing SR images to have the same anatomical structures as the input clinical CT (LR) images. We believe MMSR loss is more important than network modification, since in Fig. 8(b), CycleGAN with upblocks (no MMSR loss, no network modification, only upblocks for larger output image) output results that do not have similarity with the input images. As shown in Fig. 9(b), SR-CycleGAN with downblocks (with MMSR loss, no network modification) performed SR of clinical CT images. However, these results were not as good as SR-CycLeGAN with sub-pixel shuffling (with both MMSR loss and network modification) in Fig. 9(c). MMSR loss enabled SR of clinical CT images, and modification of the network enhanced the qualitative and quantitative results.

4.2.

Effect of Hyperparameter Adjustment

We performed further experiments to address the effect of different hyperparameters on the final result. Specifically, we changed the number of Resblocks, the convolution kernel size, and the patch size for training. We showed the number of Resblocks, the convolution kernel size, and the patch size utilized in our method in Fig. 15. First, we changed the number of Resblocks. The number of Resblocks in generator $G_1$ of our method was 9. Since we built our method based on CycleGAN, whose numbers of Resblocks were 6 (for small patches) and 9 (for large patches), we performed an experiment with a smaller number of Resblocks 6. In addition, since the difference between 9 (number of Resblocks in our method) and 6 (the smaller number of Resblocks) was 3, we further performed an experiment with a larger number of Resblocks of $9+3=12$. Furthermore, we performed an experiment with a larger or smaller convolution kernel. The first Conv+BN+ReLU block in generator $G_1$ of our method utilized a convolution kernel of size $3\times 3$; the second Conv+BN+ReLU block utilized a convolution kernel of size $7\times 7$. We changed the first Conv+BN+ReLU block’s convolutional kernel size to $7\times 7$ to test the effect of a larger convolution kernel. Correspondingly, we changed the second Conv+BN+ReLU block’s convolutional kernel size to $3\times 3$ to test the effect of a smaller convolution kernel. The patch size for training was also adjusted. The input patch size in our method was $32\times 32\text{pixels}$. We tried using smaller ($24\times 24\text{pixels}$) and larger ($48\times 48\text{pixels}$) patch sizes to investigate the impact of patch size on the results.

Fig. 15

Hyperparameters of our method’s generator $G_1$. The first Conv+BN+ReLU block uses a convolution kernel of size $3\times 3$ and the second Conv+BN+ReLU block uses a convolution kernel of size $7\times 7$. Input patch size is $32\times 32\text{pixels}$ and output size is $256\times 256\text{pixels}$. Number of Resblocks is 9.

Table 7 shows that using 9 Resblocks, $3\times 3$ and $7\times 7$ convolution kernel sizes, and $32\times 32\text{pixels}$ patch size led to the highest PSNR and SSIM score. Using either more or fewer Resblocks, larger or smaller convolution kernel size, or larger or smaller patch size resulted in a lower PSNR and SSIM score. Qualitative results of different hyperparameters were similar, as shown in Fig. 16. It is obvious that the parts enclosed in the red boxes in Fig. 16 do not have significant differences. In conclusion, the experimental results show that our method’s number of Resblocks, convolution kernel sizes, and patch size resulted in the best quantitative result as shown in Table 7. Additionally, the number of Resblocks, convolution kernel sizes, and patch size do not have much effect on the qualitative results as shown in Fig. 16.

Table 7

Different hyperparameters result in different experimental results. Experimental results showed that using nine Resblocks, 3×3 and 7×7 convolution kernel sizes, and 32×32  pixels patch size results in the best PSNR and SSIM score. The red characters in each condition indicate its difference with condition 1. Bold values are the highest.

Fig. 16

Results of different hyperparameter settings. We performed an experiment with generator $G_1$ with different numbers of Resblocks, different convolution kernel sizes, and different patch sizes. Table 7 gives detailed parameters. We zoom in on the regions in the red boxes for a better comparison.

4.3.

Novelty of Our Method and Difference from Recent CT SR Methods

Our method has three novel components: (1) a lightweight network equipped with sub-pixel shuffling layers,³⁶ (2) novel loss terms named upsample and downsample losses, and (3) a novel loss term named SSIM loss. We modified components (1), (2), and (3) in applying them to our task. We added component (1) in CycleGAN to apply component (1) in unsupervised scenarios. Although components (2) and (3) have been used as loss terms in some SR methods,⁵³ they were never used to measure the similarities of different-size images (e.g., one image size of $32\times 32$ and another of $128\times 128$). We modified components (2) and (3) to measure the similarities of differently sized images and utilized the similarities as loss terms to optimize our proposed network. No existing CT SR method utilizes components (1), (2), and (3) at the same time. By combining components (1), (2), and (3) in our method, we successfully implemented unsupervised SR with a relatively lightweight network. As a result, our method successfully achieved SR on a clinical $\mathrm{CT}-\mu \mathrm{CT}$ dataset, which cannot be attained by recent CT SR methods.

Here, we compare the MMSR loss with other loss terms proposed in previous methods, and discuss about the necessity of the MMSR loss. A relevant work named GAN-CIRCLE²⁹ used adversarial loss, cycle-consistency loss, identity loss, and joint sparsifying transform loss to indirectly promote the consistency between input LR and output SR image. In contrast, our method imposes the MMSR loss to directly constrain input LR and output SR images have higher SSIM and pixel-wise similarity. In our newly built clinical $\mathrm{CT}-\mu \mathrm{CT}$ dataset, LR and HR images have huge intensity and structural difference. Therefore, if we train SR methods without directly constraints between input LR and output SR images on our clinical $\mathrm{CT}-\mu \mathrm{CT}$ dataset, the trained network tends to output SR images that is totally different from input LR images, such as results of pseudo-SR in Fig. 13. In contrast, using the MMSR loss, our method obtained satisfying qualitative and quantitative results. Another relevant network named CinCGAN³² uses modified identity loss and modified TV loss to ensure SR network’s output has higher pixel-wise similarity with input. However, CinCGAN only calculates the modified identity loss between input LR and output SR image. On the other hand, our method calculates MMSR loss from (1) input LR and output SR image and (2) HR image and corresponding synthesized LR image. Moreover, our MMSR loss is proposed based on two evaluation metrics: MSE and SSIM. Our method showed better performance than CinCGAN on MSE-based (PSNR) and SSIM-based evaluation metrics.

We can further differentiate our method from recent supervised and unsupervised CT SR methods. Recent supervised CT SR methods, such as ESRGAN for CT SR,⁵⁴ require pairs of LR-HR images for training. In contrast, our method does not need any paired LR-HR images for training. Some image denoising methods could be applied in SR.⁵⁵ GAN with network-in-network structure embed with skip connection naming deep convolutional generative adversarial network (DCSWGAN)²⁰ was proved to be effective in CT image denoising. The generator of DCSWGAN consists of convolutional blocks, and each convolutional block consists of convolutional layer, bias, and leaky rectified linear unit, which is similar to our method’s generator $G_1$. The generator of DCSWGAN uses a cascade structure containing two subnetworks, one is a feature extraction network, the other is a reconstruction network. In contrast, our method only uses one network for SR. A disadvantage of DCSWGAN is that it still needs paired images for training. You et al. proposed an unsupervised SR method for CT and MRI images named GAN-CIRCLE,^29,56 and further applied to bone micro structure reconstruction⁵⁷ and brain MRI reconstruction.⁵⁸ GAN-CIRCLE performed $2\times$ SR (resolution of output SR image is two times of input LR image). On the other hand, we desire an $8\times$ SR method which performs SR of clinical CT images to $\mu \mathrm{CT}$ scale. Our method achieved 8× SR (SR from $32\times 32\text{pixels}$ to $256\times 256\text{pixels}$). Moreover, unsupervised SR methods such as CinCGAN³² and GAN-CIRCLE²⁹ can only perform SR between images of the same modality (e.g., LR MRI images to HR MRI images); consequently, the LR and HR images do not have huge differences aside from resolution. Therefore, recent SR methods performed poorly on our clinical $\mathrm{CT}-\mu \mathrm{CT}$ dataset, since our HR ($\mu \mathrm{CT}$) and LR (clinical CT) images are from totally different modalities.

4.4.

Analysis of Parameter Selection of Loss Terms

Here, we analyze the parameter selection of each loss term and discuss how assigning weights to each loss term leads to the best results. The overall loss function is composed of three terms: (1) SSIM loss, (2) downsample loss, and (3) upsample loss. Various combinations of loss terms lead to different quantitative results, as shown in Table 3. Table 3 shows that each loss function contributes to the final result. SSIM loss (containing two loss terms) brings the highest PSNR and SSIM score improvement. While the method is already equipped with SSIM loss, downsample loss and upsample loss can still improve PSNR and SSIM score slightly. Therefore, we believe that a higher weight of SSIM loss together with smaller weights of downsample loss and upsample loss brings the highest PSNR and SSIM score.

4.5.

Effect of Downblocks in SR-CycleGAN

We performed experiments to verify the effectiveness of removing downblocks and adding pixel-shuffling layers in generator $G_1$. As shown in Fig. 9, the SR results obtained by generator $G_1$ with downblocks and without pixel-shuffling layers [Fig. 17(a)] look blurred and noisy, while the SR results obtained by generator $G_1$ without downblocks and with sub-pixel shuffling layers [Fig. 17(b)] look clearer. This is because downblocks scale down the input images to a smaller size. Input images have $32\times 32\text{pixels}$; downblocks scale down the input images into feature maps of $4\times 4\text{pixels}$, and such small feature maps destroy spatial information in the input image. Furthermore, generator $G_1$ with downblocks [Fig. 17(a)] is deeper than generaor $G_1$ without downblocks [Fig. 17(b)]. Previous research affirmed that deeper stages of neural networks are more semantic but spatially coarser.⁵⁹ Thus, the shape of essential anatomical structures such as the bronchus are likely to deform in the SR result, as shown in Fig. 9(b).

Fig. 17

Two kinds of generator $G_1$ structures. We performed experiments on (a) $G_1$ with downblocks and (b) $G_1$ without downblocks but with added sub-pixel shuffling layers; (b) performed qualitatively and quantitatively better than (a). (c) Detailed compositions of each block.

4.6.

Effect of Reducing Computing Time Using Sub-Pixel Shuffling Layers

The sub-pixel shuffling layers were proved to shorten computing time, compared with upblocks.³⁶ We replaced upblocks with sub-pixel shuffling layers in the proposed SR-CycleGAN. In Fig. 7, two kinds of network structures for generator $G_1$ are compared. The experimental results show that training time was significantly reduced from 491 to 353 s for training per epoch (2000 patches). For handling large-scale networks, such as CycleGAN, reducing computing time is an important issue. Introducing sub-pixel shuffling layers saved computing resources without loss of accuracy.

4.7.

Difficulty of Quantitative Evaluation

In conventional SR methods, quantitative evaluation is typically conducted by comparing SR and HR image pairs. However, it is infeasible to obtain such pairs between clinical CT and $\mu \mathrm{CT}$ images, as mentioned in Sec. 1. To perform quantitative evaluation, we used downsampled $\mu \mathrm{CT}$ images instead of clinical CT images. We input the downsampled $\mu \mathrm{CT}$ image into trained generator $G_1$ and then obtained the SR result of downsampled $\mu \mathrm{CT}$ from $G_1$. Next, we compared the SR result with the original $\mu \mathrm{CT}$ images. We used PSNR to compare the SR image and the original $\mu \mathrm{CT}$ image. Since $\mu \mathrm{CT}$ images and clinical CT images have the same anatomical structures (bronchi and arteries), downsampled $\mu \mathrm{CT}$ images can simulate clinical CT images to a certain extent.

However, downsampled $\mu \mathrm{CT}$ images cannot simulate clinical CT images perfectly because the imaging conditions of $\mu \mathrm{CT}$ and clinical CT are different. For a specific tissues such as the bronchus in clinical CT, intensity is around $-500$ to 200 H.U. On the other hand, the intensity of the bronchus in $\mu \mathrm{CT}$ is around 6000 to 14,000 H.U. Furthermore, lung specimens for scanning $\mu \mathrm{CT}$ images are resected from part of the lung, so the $\mu \mathrm{CT}$ images of lung specimens do not contain anatomical information of the whole lung. Hence, we cannot simulate clinical CT perfectly by downsampling $\mu \mathrm{CT}$ images to the clinical CT scale. Therefore, in the future, we plan to propose a new evaluation matrix for the evaluation of SR-CycleGAN.