2.1.1.

Leakage between training and testing, in transitioning from AI to medical imaging AI

Understanding the nature of medical images and the clinical context in which they are used is a necessary prerequisite for any successful application of an AI algorithm for diagnostic or prognostic medical purposes. How the training models should be built and how the data will be used in that process can be a main source of confusion and error and subsequent bias as well.³⁵ For instance, in a diagnostic learning task, a CT scan with many images could be erroneously perceived and then evaluated on a 2D slice-by-slice analysis based on an inaccurate assumption of independence between slices from the same patient while they are typically correlated. As such, slices from the same patient may be included in both training and testing sets, thereby yielding overly optimistic testing results due to this data leakage. Similarly, in the case of prognostic studies for predicting death or ventilation need, it is sometime unclear whether images from the same patient at different time points were mistakenly used for both training and testing.³⁶

2.1.2.

Feature/model selection and multiple testing

In traditional machine learning models based on feature engineering, feature selection is an essential process in the building of computational tools for the AI interpretation of medical images. This process is prone to statistical bias and potentially overly optimistic results if not done properly; for example, if the entire dataset was first used for feature selection and then partitioned into apparently non-overlapping subsets for training and testing, the test results would be optimistically biased.³⁷ Proper methods should be used to avoid such pitfalls, such as the use of nested cross validation or other resampling or independent testing methods in algorithm development and evaluation.³⁸ Although the new generation of deep learning algorithms bypasses the feature selection process by embedding the data representation as part of the algorithm training itself [e.g., the convolutional layers in the convolutional neural networks (CNNs)], proper safeguards should still be applied when tuning a large number of hyperparameters in this case (i.e., model selection).

Another commonly encountered issue is false positive findings due to multiple testing, i.e., when multiple hypotheses are tested without appropriate statistical adjustment, a truly null hypothesis may be found significant due purely to chance. An infamous problem in this area was cited in the analysis of fMRI data from an image of a dead Atlantic salmon. Using standard acquisition, preprocessing, and analysis techniques, the authors showed that active voxel clusters could be observed in the dead salmon’s brain when using uncorrected statistical thresholds for multiple comparisons, which led to such false positives, i.e., active voxels clusters in dead fish.³⁹ To avoid such pitfalls, appropriate statistical methods must be prespecified in a study to control the type I error rate.^40,41

2.1.3.

Erroneous labeling of the clinical task

This arises when there is no clarity in the description of the clinical problem.²⁹ For instance, a clinical problem might be posed as a diagnostic problem for differentiating between COVID-19 cases and control cases. In this scenario, the control cohort might be healthy individuals or might be individuals with other lung diseases such as pneumonia (in which presentation of the disease on the medical image might mimic that of COVID-19). Those two different control cohorts will yield a wide range of classification performances and, thus, confusion on the ultimate clinical use of the AI.

In another context, confusion can arise in the purpose of the AI algorithm. Is the AI algorithm for identifying which images have COVID-19 presentations (classification task) or is the AI algorithm for localizing regions of COVID-19 involvement within an image (segmentation task)? This can also manifest in the performance evaluation. These clinical tasks will require different evaluation metrics and clarification of sensitivity and/or specificity, especially when imbalances arise in the dataset.

2.1.4.

Mismatches between training data and clinical task

An AI algorithm trained in a non-medical context may not transfer well to medical imaging AI, in part due to restrictive deidentification or HIPAA concerns, which may limit some of the data availability. For instance, the required data may be fragmented or distributed across multiple systems. This is in addition to issues related to the need for proper data curation processes to ensure that the necessary quality is available. Different hospitals may apply their CT protocols differently such as variations in image acquisition systems and protocols (i.e., slice thickness and reconstruction) indicating the need for harmonization. Otherwise, the process may be subject to the “garbage in garbage out” pitfall.

On the other hand, transfer learning,^42,43 with fine tuning and feature extraction, has proven to be quite useful in AI of medical imaging, especially for tasks related to image analysis such as segmentation, denoising, and classification (leading to diagnosis and other tasks). Therefore, to mitigate the problem of mismatch between the intended training process and the clinical task, there needs to be transparent communication between the AI and the medical imaging domain experts.

2.1.5.

Role of continuous learning

Several dynamic algorithms such as reinforcement learning tend to continuously update their training by exploring the environment, which could work well in a safe setting such as a board game but may have legal and ethical ramifications in the real world (e.g., self-driving accidents) and especially in the context of medicine, where life or death decisions are being made. The traditional approach to “continuous” learning is implemented through manually controlled periodic updates to “locked” algorithms.⁴⁴ Recent technology advances in continuous learning AI have the potential to provide fully automatic updates by a computer with the promise of adapting continuously to changes in local data.⁴⁵ However, how to assess such an adaptive AI system to ensure safety and effectiveness in the real world is an open question, and there are several ongoing research efforts and discussions on this topic across academia and regulatory agencies to ensure that the deployed AI algorithm is still correctly performing the originally intended task.^44–46 This process may require an AI monitoring program and periodic evaluations over the lifetime of the algorithm.

2.2.1.

Application of pretrained models

As machine learning methods are increasingly developed for medical imaging applications, there are numerous models made available by investigators either publicly or through collaborations. However, direct applications of off-the-shelf models are often unsuccessful. Needless to say, models trained for a specific task, e.g., detecting a certain disease on a certain imaging modality, are not suitable for being directly applied to a different task, because useful features for performing tasks can be vastly different. Moreover, even when the task in question remains the same, directly applying off-the-shelf models developed on one dataset often fails to generalize their performance when applied to another dataset. There are several reasons for the lack of robustness across different datasets for machine learning models in medical imaging applications, including differences in imaging acquisition protocols, scanners, patient populations, and image preprocessing methods. Features extracted by machine learning models, both human-engineered features and automatically learned features by deep learning, are often sensitive to these differences between the source domain and the target domain. Therefore, it is good practice to preprocess the images in the target domain the same way as for the development set, harmonize differences in images resulting from image acquisition and population characteristics, and fine-tune the pretrained model on a subset of the images in the target domain when appropriate.

Another significant issue is cross-institution generalizability of deep learning CNNs, i.e., a CNN trained with data from one clinical site or institution is found to have a significant drop in performance on data from another site. This has been frequently reported in the recent literature, for instances, on the classification of abnormal chest radiographs,⁴⁷ on the diagnosis of pneumonia,⁴⁸ and on the detection of COVID-19 from chest radiographs.⁴⁹ In addition to the possible reasons discussed in the previous paragraph, a particular issue found by these authors is that CNNs have readily learned features specific to sites (such as metal tokens placed on patients at a specific site) rather than the pathology information in the image. As such, proper testing procedures that mimic real-world implementation are crucial before any AI clinical deployment.⁵⁰

2.2.2.

Use of texture analysis software packages

Often off-the-shelf texture analysis software packages are used to quantify the variation perceived in a medical image, the output of which is subsequently related to disease states using AI algorithms.⁵¹ Assessment of texture provides quantitative values of pattern intensity, spatial frequency content (i.e., coarseness), and directionality.⁵² Caution is needed, however, since, as noted by Foy et al.,⁵¹ algorithm parameters, such as those used for normalization of pixel values prior to processing, can vary between software packages that are calculating the same mathematical feature. An example is in the calculation of texture features from gray-level co-occurrence matrices (GLCM), in which GLCM parameters may be unique to the specific software package. Entropy, for example, involves the calculation of a logarithm, which might be to the base $e$, the base 2, or the base 10, with each yielding different output texture values of entropy.⁵¹ There have been attempts to standardize the calculation of radiomic features.⁵³ It is important to understand both the mathematical formulation of the radiomic feature as well as the software package implementation to achieve the intended results as recommended by the Image Biomarker Standardization Initiative.⁵³

2.2.3.

Overfitting and bias in development of AI systems

The novice developer of AI should be careful to avoid overfitting and bias in AI algorithms, often caused by limited datasets in terms of number, distribution, and collection sites of cases.⁵⁴ While even independent testing using cases from a single institution may yield promising performance levels, the AI algorithm may not be assumed to be generalizable until it is tested on an independent multi-institutional dataset with results given in terms of performance in the specific task including its variation (reproducibility and repeatability) as well as assessment of whether or not the algorithms are biased. Sharing of data in such instances may be challenging; therefore, the availability of public resources such as the Medical Imaging Data and Resource Center (MIDRC) or the utilization of emerging methods based on federated learning would be helpful in this process.⁵⁵

2.3.1.

Observer variability of COVID-19

Observer variability refers to the concept of different readers assessing the same input image and producing different results.^56,57 While not uncommon in medical imaging, the degree to which observer variability may impact an AI model either during training or evaluation is dependent on the imaging task.^58–62 For example, in COVID-19 detection problems, the standard ground truth identifying disease positivity is RT-PCR, which may have less variability compared with imaging.^63,64 Alternatively, medical image segmentation tasks generally utilize a ground truth mask delineated by expert radiologists. Several studies have analyzed observer variability for tasks related to COVID-19 assessment, including disease detection, diagnosis, segmentation, and prognosis, including detection of abnormalities on CXRs and segmentation of disease on CT scans, even in other diagnostic tests such as RT-PCR.^63–69 Methods that can alleviate complications related to observer variability include statistical resampling techniques, such as bootstrapping or imbalance correction such as synthetic minority over-sampling technique and using appropriate evaluation metrics such as precision–recall and receiver operating characteristic analyses.^70,71 Furthermore, implementation of clearly defined task-relevant scores such as CO-RADS can standardize evaluations and reduce the impact of such interobserver variability.^72,72

2.3.2.

Understanding COVID-19 imaging protocols

Most diseases have existed for many years and thus have undergone extensive investigation of determining appropriate imaging protocols for detection, diagnosis, and prognosis evaluations. However, protocols for COVID-19 patient imaging have rapidly changed over the past year as researchers investigated optimal practices for a variety of situational evaluations.⁷³ Further, different radiological societies may have inconsistent or conflicting recommendations based on regional and national standards.^74–80 The effect of these phenomena is twofold. (1) Existing COVID-19 imaging datasets may contain images that would not be appropriate for general application due to outdated image acquisition protocols and thus should not be included in AI training without careful curation, especially if acquired externally to the region or institution in which the AI system will be utilized. (2) Task selection for an AI system should be performed based on local practice, which in turn should be based on radiological society or WHO recommendations. For example, most imaging societies do not recommend the use of CT for COVID-19 screening due to the increased patient dose, with likely no information gain compared with RT-PCR and chest radiography; thus AI algorithms trained for COVID-19 detection may prioritize detection on CXR rather than CT for widespread implementation. While these studies may still provide some value, AI algorithms should generally be developed with appropriate task selection and data curation that is relevant with current image acquisition guidelines for their intended application.

2.3.3.

Biased COVID-19 datasets and models

One of the most common obstacles in training unbiased medical imaging AI systems is insufficient relevant patient imaging data.⁸¹ This obstacle is exacerbated due to the brief existence of COVID-19 as the total pipeline of image acquisition, curation, and labeling for ML usage can be time-consuming. During the pandemic, several teams have attempted to utilize publicly available datasets of both COVID-19 and non-COVID-19 patient images to develop generalized models for clinical use.³⁵ Thus two key problems currently are posed: (1) data, particularly private data, are generally skewed such that the amount of COVID-19 positive data is notably smaller than COVID-19 negative data and (2) the use of publicly available datasets can lead to a high risk of model bias, either due to a lack of information (such as consistent confirmation of COVID-19 diagnosis through RT-PCR and/or other clinical tests) or through crossover of images between databases leading to biased evaluations.⁶⁴ Further, some public datasets include images not in DICOM format (e.g., images scrubbed from published papers). Thus it is critically important to understand the source of the imaging data and evaluate appropriately. For example, evaluation should be performed on completely independent private (when possible) and public datasets, such as the image testing data in the MIDRC.

3.1.1.

COVID-19 detection/diagnosis

Although RT-PCR is the reference standard for COVID-19 diagnosis, molecular testing may not be readily available or may lead to false negative results for patients under investigation of COVID-19, and thus under these circumstances, CXR/CT can be critically useful in the diagnosis of the disease. CXR/CT in COVID-19 diagnosis has moderate sensitivity and specificity⁹¹ in current radiological practice possibly because of clinicians’ limited experience in reading COVID-19 cases. AI/ML models can be trained using these images with RT-PCR as the reference standard. These trained models can be applied to patients to make rapid diagnosis when only imaging data are available.^92–94 AI/ML models have been shown to be able to help clinicians differentiate COVID-19 induced pneumonia from other types of pneumonia, including common and unique radiographic features differentiation.²⁶ Thus AI/ML models including both deep learning and human-engineered radiomics have the potential to aid clinicians with improved diagnostic performance.

3.1.2.

COVID-19 severity assessment/prognosis

Baseline pulmonary CXR/CT imaging characteristics can be used to assess the severity/extent of disease.⁹⁵ A severity score derived from lung images can be used for triaging the patient to decide home discharge, regular hospital admission, or intensive care unit admission.⁹⁶ The imaging characteristics can be used for prognosis assessment⁹⁷ and evaluation of the extent of disease,¹¹ and it has potential to predict the length of hospital stay.⁹⁸

Although COVID-19 is seen as a disease that primarily affects the lungs, it may also damage other organs, including heart^93,99 and brain,^93,99–102 and many long-term COVID-19 effects are still unknown. The COVID-19 patients have to be closely monitored even after their recovery, when imaging and AI may play a pivotal role.

3.1.3.

Response to treatment

Baseline pulmonary CXR/CT imaging characteristics can also be used for patient management to determine the therapeutic treatment plan. The temporal changes can be used to monitor the patients’ response to the treatment and to inform the decision for discharge.⁹⁷ Lung ultrasound has also been suggested as the imaging tool for COVID-19 pandemic in disease triage, diagnosis, prognosis, severity assessment, and monitoring the treatment response.¹³