1.1

Motivation

Mobile C-Arm devices are an integral part of modern surgical procedures. In addition to 2D X-Ray projection images used for guidance, modern systems allow to accurately verify the placement of tools and implants via 3D imaging. One major limitation of this modality are metal artifacts. Originating from simplifications in the reconstruction model and inconsistencies in the measured data, these image artifacts appear as bright and dark streaks, local distortions of CT values, and shadowing. As they emerge especially around metal objects, they obstruct the implant placement verification and thus drastically decrease the diagnostic value during the surgical intervention.

1.2

Existing MAR Approaches

Most modern Metal Artifact Reduction (MAR) methods reduce metal artifacts by inpainting the metal traces in projection domain. These metal traces are obtained by forward projection of the volumetric segmentation of metal objects. Inpainting in this context refers to the process of substituting pixel values to remove their contribution to the reconstruction image. Over the last decades, multiple approaches for inpainting have been proposed starting with simple linear or polynomial interpolation,^{1, 2} frequency domain interpolation,³ using wavelets⁴ or with the help of machine learning.⁵ Regardless of how elaborated the inpainting approach is designed, a faulty estimation of the metal mask can sabotage the effectiveness of said approaches, reduce the level of detail around the corrected metal, remove relevant anatomy, and even introduce new artifacts.⁶

1.3

Related work

To enhance the mask quality, a shape-model based estimation of the metallic objects has been proposed.⁷ A known object’s outline is registered to a coarse volumetric metal segmentation to refine its shape. However, this approach is not generally applicable, as prior knowledge about the shape of the metal is usually not available. The segmentation of metal in image domain is prone to error mainly because of the metal artifacts. The alterations of CT values around the depicted metal prohibit a purely value-based approach such as the traditional global thresholding. By including structural information, Convolutional Neural Networks (CNN) have proven to be successful in many medical segmentation scenarios. Recent advances in known operator integration have facilitated the end-to-end training of cross-domain architectures. A derivable backprojection operator can be embedded into the model and used with gradient backpropagation for supervised learning.⁸ The observation that metal artifacts originate during the domain transformation suggests that neural network architectures can benefit from projection domain information. Multi-domain approaches have successfully been applied in Metal Artifact Avoidance (MAA) to estimate metallic objects from very few given projections.⁹ However, the objective is fundamentally different compared to MAR. For MAA, a rough distribution of metal is sufficient to adapt the trajectory, whereas high detail masks are desirable for MAR. Furthermore, the system matrix based reconstruction from⁹ cannot be applied to MAR segmentation due to the larger number of input projections. A dual domain network topology has also been demonstrated for direct, learned MAR.¹⁰ The authors report significant improvements over other single domain MAR approaches.

1.4

Planned Contributions

In this work, a novel cross-domain segmentation network which yields both raw rotational and reconstruction domain information is presented. It is compared to a CNN approach applied to artifact-corrupted reconstruction images and a simple threshold-based method.

2.1

Data

The cross-domain architecture proposed in this work requires training data, where the input consists of X-Ray projection images and the target metal mask is a volume-shaped binary array. It is crucial that the simulated projection images exhibit all physical effects which are relevant for the formation of metal artifacts. To model said effects, a simulation framework was derived from the DeepDRR method described in.¹¹ However, the provided pre-trained weights for material segmentation and scatter estimation could not be used as they did not generalize well on the raw data used in this project.

A total of 11 cone-beam CT volumes were acquired from 5 different specimens from a human spine cadaver study of the lumbar and thoracic region using a Siemens Cios Spin System.^* To realistically model the shapes of metal objects, a library of 3D models of surgical screws, plates, k-wires, and towers are available from Nuvasive, San Diego, California. A set of metal objects is realistically positioned relative to the skeletal structures in the base volume using the 3D modelling suite Blender.^† The position and orientation of each object is stored and considered during the simulation.

2.2

Optimal Threshold-Based Method

As current product-grade MAR methods utilize a threshold-based segmentation, it serves as a baseline here. To obtain a best-case estimate of such a global thresholding method, the optimal threshold is calculated for each sample. As an optimality criterion, the dice similarity coefficient (DSC) is used. Equation 1 shows the dice similarity defined on the binarized volume X and the binary label Y.

A threshold is defined as optimal when the binarization it generates maximises the DSC metric. For each simulated CT volume, an optimal threshold is determined by testing 1000 values over the range of its histogram values.

2.3

Image Domain CNN Method

To obtain a standardized segmentation benchmark, the self-configuring framework nnUNet is used.¹² nnUNet automatically configures all relevant parameters of the popular architecture U-Net and adapts it to the so-called dataset fingerprint. The user can choose between a 2D, a 3D high-resolution, and a 3D low-resolution U-Net layout. For this project, the 3D low-resolution (3d-lowres) U-Net model was chosen. The training of the other models was omitted, as the cross-domain method produces lower resolution masks and the results are compared on this lower resolution anyway. This model contains 6 mio. trainable parameters and is applied patch-wise on cube-shaped sub-volumes of side-length 128. The model is trained on CT volumes reconstructed from the simulated projection data. To predict volumes of side-length 256, the model is evaluated 27 times as the patches are strided by half a patch-size.

The performance of the 3d-lowres architecture was evaluated using 5-fold cross validation. The simulated X-Ray images were reconstructed using filtered backprojection to serve as training input to this method. The data splitting, pre-processing and evaluation using the DSC metric was handled by the framework.

2.4

Cross-Domain Architecture

Inspired by the success of the 3D U-Net architecture, an encoder-decoder layout with skip-connections is used.¹³ To enable end-to-end training with input and labels from different domains, a derivable backprojector is integrated into the network. This operator is implemented as the PyroNN filtered backprojection layer with a non-trainable Ram-Lak filter.⁸

As seen in figure 1, this domain transform happens during the skip-connection step, such that the encoder is applied to projection domain and the decoder to reconstruction domain. The shapes of the tensors are indicated next to each stage of the architecture, whilst the number of computed feature-maps is shown above each block symbolizing a layer’s output. At each stage of the network, the stack of projections is downsampled both in the number of projections and their resolution.

Figure 1.

Cross-domain network architecture with an encoder in projection domain and a decoder in image/reconstruction domain. The backprojection operators embedded on the skip-connections reconstruct volumes from the input projection domain feature maps and stack them along the channel axis (blue).

To facilitate the reconstruction of differently sized volumes, the projection matrices and the resolution configurations are adapted accordingly. The convolutional layers with kernels of side length three use zero-padding to generate similarly sized feature maps. At the skip-connections, these feature maps are reconstructed individually and the resulting volume-shaped activations are stacked along the channel axis.

2.5

Cross-Domain Training Procedure

3.1

Quantitative

The cross-validation results and the evaluation of the optimal threshold-based method are shown in Table 1. Our method achieves a dice similarity of 0.87 ± 0.05 averaged over the different data splits. The reference method applied in reconstruction domain evaluates to a DSC of 0.86 ± 0.03 across all cross-validation runs. Note, that the data splits are not identical between our method and the reference image domain CNN. The best possible segmentation using a per-sample threshold was evaluated for the entire simulated dataset. Overall, this method achieves a DSC metric of 0.45 ± 0.21.

Table 1.

Results of Model Evaluation (Dice Similarity)

Apart from the segmentation performance, other attributes of the compared methods are displayed in Table 2. The inference time of the reference image domain CNN method is about eleven times longer than our method. The auto-configuring framework nnUNet trains the models for about 97h which is more than six times longer than our methods, which are trained 15h on average. The cross-domain network requires about four times more GPU memory during training and inference.

Table 2.

Additional Method Attributes

3.2

Qualitative

To intuitively compare the methods performance, an unseen sample is segmented using the three compared methods (Figure 2). From the resulting volumetric metal masks, a slice is selected and cropped to a region of interest showing two screws with tulips fixated to connecting rods. For reference, the corresponding reconstruction is added. As the screws’ main axes align with the acquisition trajectory, heavy artefacts are visible. The thresholding method c) greatly underestimates the ground truth mask b). The image domain CNN and the cross-domain segmentation network produce similar looking masks, which, compared to the label, yield a dice similarity of 0.88 and 0.90 respectively.

Figure 2.

Cropped slice from a volumetric metal segmentation. a) shows the corresponding artifact-corrupted reconstruction, b) is the ground truth, c) depicts the best-possible threshold-based segmentation, d) shows the result of the reference image domain CNN, and e) shows our methods prediction.