2.1

Learning Complex Deformable Motion with a GAN model

The proposed GAN architecture is illustrated in Fig. 1. The proposed approach leveraged the concept of Partial Angle Reconstruction (PAR) combined with spatial transformer modules, previously used in CBCT motion compensation [4]. A complete, motion-free, CBCT projection dataset, with a total of 360 projections, is reconstructed into N_α = 12 PAR volumes of 512 × 512 × 128 voxels, each containing the backprojection from 30 consecutive projection views. Those PARs are the input to the generator network, which outputs N_α sets of 36 × 36 × 12 B-spline coefficients that serve as a lower dimensionality representation for each of the N_α motion vector fields (MVFs) representing the simulated 4D deformable motion of the volume. Dense MVFs, with size equal to that of the PAR volumes, are then generated via B-spline interpolation. The set of dense MVFs and the original motion-free PARs are then input to a spatial transformer module that applies the simulated deformation to each of the PARs and add them together to obtain a final motion-corrupted volume. In the resulting architecture PARs are considered static, effectively assuming a piecewise constant temporal motion trajectory. A discriminator network was used to provide a GAN loss discriminating between real and simulated motion-corrupted volumes. The architecture of each of the components is discussed below.

Figure 1.

Schematic depiction of the realistic motion simulation framework, based on a deep generative adversarial network model. Motion-free simulated CBCT projections are divided into N_α = 12 groups of consecutive projections that generate a set of N_α. The generator network, based on the U-NET architecture, receives at input the set of static PARs and a random perturbation of the latent space. The sparse 4D MVF output by the generator induces the deformation into the individual PARs that were added together into the final motion-corrupted volume. For GAN training, a discriminator receives as input the generated simulated motion volumes and samples of the real motion-corrupted volumes from the training dataset.

Generator: The generator featured a non-symmetrical 3D UNet-like [10] structure with a 3-stage encoding branch, and 2- stage decoding branch. The encoding branch received the motion-free PARs as the input and extracted into the latent space features associated to structural content of the image associated with motion characteristics. This way, the input, motionfree PARs act as the condition variable of the conditional GAN architecture. Each stage on the encoder branch included a set of two 3×3×3 convolution layers, batch normalization, leaky ReLU activation, and a final 2x max pooling layer.

The set of latent space features were combined with a Gaussian random perturbation field to generate random, distinct, motion patterns for a given input condition during both training and inference time. The set of latent features and random perturbation entered the decoder branch, with 2 stages implementing a 3×3×3 convolution layer, a batch normalization layer, and leaky ReLU activation, followed by a 3×3×3 transposed convolution for up-sampling of the feature maps. Skip connections were placed between equivalent levels of the encoder and decoder branches. The output of the decoder is input to three branches implementing a cascade of two 3×3×3 convolution layers, with leaky ReLU activations, that generate the B-spline coefficients for the directional components of the MVFs in the antero-posterior (AP), lateral (LAT), and superior-inferior (SI) directions.

Discriminator: The discriminator acts on motion-corrupted CBCT volumes to predict whether the input comes from a real or simulated instance. During training, the Binary Cross Entropy (BCE) loss was calculated for the simulated and real datasets, and the total loss was defined as the average of both.

In the proposed model, the discriminator featured a cascade of 5 convolution layers (4×4×4 kernel), followed by batch normalization, leaky ReLU activation, and a dropout layer (0.2 dropout). The final fully connected layer (with sigmoid activation) acted on the flattened set of features.

2.2

Data Generation and Motion Model

For this proof-of-concept study, training and validation data were generated from 70 Multi-detector CT (MDCT) abdominal datasets from the TCIA Lymph Node Abdomen collection. 60 distinct MDCT instances were used for training, 5 for validation, and 5 for testing. For each source MDCT volume, we randomly selected a subvolume of 128 mm length at a random longitudinal position within the abdomen. The subvolume was then forward projected using a high-fidelity CBCT model with a geometry with source-to-detector distance of 1200 mm, and source-to-axis distance of 785 mm. The detector was modeled as a flat panel with 576 × 440 pixels (0.616 mm isotropic pixel size).

Motion corrupted datasets were obtained by inducing deformable motion during forward projection. The simulated motion field followed a cosine temporal trajectory with random frequency between 0.75 – 1.25 cycles per scan and random phase. Spatial distribution of motion amplitude was modelled as an elliptical field with maximum amplitude (randomly set between 10 and 25 mm) at the center, and randomly placed at a soft-tissue region of the volume. The amplitude faded following a Gaussian decay curve that reached zero at the ellipse axis length, randomly chosen from 200 to 300 mm in the medial-lateral (LAT) direction and between 100 and 150 mm in the antero-posterior (AP) direction. Motion amplitude was kept constant across slices. Motion direction was randomly chosen, allocating between 60% and 80% of motion to the SI direction and the rest to the AP direction, with no lateral motion.

To avoid unrealistic large motion of the spine region, the center of the spine was detected in the volume and a cylindrical motion-exclusion mask with 100 mm radius was defined. The mask performed a smooth transition from one to zero and multiplied the motion field, to minimize the motion in the spine, as illustrated in Fig 2.

Figure 2.

Example training case. Deformable motion was induced following with a motion field with randomly selected central location and amplitude. In this proof-of-concept study, the spinal region was automatically detected and an exclusion mask with smooth boundaries was defined (see cyan delineation). The motion field was attenuated to preserve the spine region quasi-static.

The motion-corrupted datasets featured 3 distinct properties that were used for validation of the GAN capability for inference of consistent motion instances: i) the induced motion was composed of diffeomorphic deformations; ii) the spine region remained nearly static for all training instances; and, iii) the majority of the motion was allocated to the SI direction with the rest in the AP direction.

Motion-corrupted datasets were reconstructed into volumetric grids of 512 × 512 × 128 voxels with 0.5 × 0.5 × 0.5 mm³ voxel size, and motion-free cases were reconstructed into 12 PARs with equivalent parameters. The PARs were downsampled to 128 × 128 × 32 voxels for input to the generator.

2.3

Network training and validation studies

A total of 720 motion-corrupted volumes based on 60 anatomical instances were included in the training set, and 15 instances from 5 separate anatomical structures were used for validation. Each dataset contained a motion-free collection and a motion-corrupted collection. During training, a sample with N_α = 12 PARs from the motion-free collection was randomly selected as input to the generator, while one sample from motion-corrupted collection was input to the discriminator. Training was performed with an unbalanced scheme in which the generator is updated every 1 batch while the discriminator is updated every 2 batches for 100 epochs, with a batch size of 12. We used the ADAM optimizer for both generator and discriminator with learning rates of 10⁻⁵ and 10⁻⁴, respectively. BCE Loss was selected as objective function to be maximized by generator while minimized by discriminator.

To validate the results, we used a test set of 15 samples based on 5 anatomical structures not seen by the network. For testing, static PARs were input to the generator together with the Gaussian random perturbation. Validity of the generated motion fields was validated via measurements of diffeomorphism, based on the determinants of the Jacobian of the deformation, and metrics of average displacement at regions of maximum motion and regions static in the training set. Furthermore, directional components of the synthetic motion were evaluated in comparison with underlying trends in the training dataset.