Impact of different study populations on reader behavior and performance metrics: initial results

Brandon D. Gallas; Etta Pisano; Elodia Cole; Kyle Myers

doi:10.1117/12.2255977

Translator Disclaimer

10 March 2017 Impact of different study populations on reader behavior and performance metrics: initial results

Brandon D. Gallas, Etta Pisano, Elodia Cole, Kyle Myers

Author Affiliations +

Proceedings Volume 10136, Medical Imaging 2017: Image Perception, Observer Performance, and Technology Assessment; 101360A (2017) https://doi.org/10.1117/12.2255977
Event: SPIE Medical Imaging, 2017, Orlando, Florida, United States

Abstract

The FDA recently completed a study on design methodologies surrounding the Validation of Imaging Premarket Evaluation and Regulation called VIPER. VIPER consisted of five large reader sub-studies to compare the impact of different study populations on reader behavior as seen by sensitivity, specificity, and AUC, the area under the ROC curve (receiver operating characteristic curve). The study investigated different prevalence levels and two kinds of sampling of non-cancer patients: a screening population and a challenge population. The VIPER study compared full-field digital mammography (FFDM) to screenfilm mammography (SFM) for women with heterogeneously dense or extremely dense breasts. All cases and corresponding images were sampled from Digital Mammographic Imaging Screening Trial (DMIST) archives. There were 20 readers (American Board Certified radiologists) for each sub-study, and instead of every reader reading every case (fully-crossed study), readers and cases were split into groups to reduce reader workload and the total number of observations (split-plot study). For data collection, readers first decided whether or not they would recall a patient. Following that decision, they provided an ROC score for how close or far that patient was from the recall decision threshold. Performance results for FFDM show that as prevalence increases to 50%, there is a moderate increase in sensitivity and decrease in specificity, whereas AUC is mainly flat. Regarding precision, the statistical efficiency (ratio of variances) of sensitivity and specificity relative to AUC are 0.66 at best and decrease with prevalence. Analyses comparing modalities and the study populations (screening vs. challenge) are still ongoing.

Conference Presentation

Citation Download Citation

Brandon D. Gallas, Etta Pisano, Elodia Cole, and Kyle Myers "Impact of different study populations on reader behavior and performance metrics: initial results", Proc. SPIE 10136, Medical Imaging 2017: Image Perception, Observer Performance, and Technology Assessment, 101360A (10 March 2017); https://doi.org/10.1117/12.2255977

ACCESS THE FULL ARTICLE