Efficiently calculating ROC curves, AUC, and uncertainty from 2AFC studies with finite samples

Dylan H. Shekter; Frank W. Samuelson

doi:10.1117/12.2550601

16 March 2020 Efficiently calculating ROC curves, AUC, and uncertainty from 2AFC studies with finite samples

Proceedings Volume 11316, Medical Imaging 2020: Image Perception, Observer Performance, and Technology Assessment; 113160L (2020) https://doi.org/10.1117/12.2550601
Event: SPIE Medical Imaging, 2020, Houston, Texas, United States

Abstract

Two-alternative forced-choice (2AFC) reader studies are useful for evaluating medical imaging devices because humans can rapidly make direct comparisons with high precision leading to low variability in study results. We propose a method for estimating the receiver operating characteristic (ROC) curve, reader performance (area under the ROC curve, AUC), and uncertainty on AUC from a series of 2AFC trials on a finite data set. Our method greatly reduces the number of 2AFC comparisons required by using an algorithm created for sorting, in this case Merge Sort. By altering the algorithm to work in discrete layers, we can make unbiased estimates as the study proceeds. Because the merging is pre-planned with a tree structure, we can use a Hanley-McNeil approximation to predict the reduction in variance in AUC from performing more 2AFC comparisons. The algorithm is also altered to increase the amount of time between the reader seeing the same image repeatedly thus decreasing potential learning. We compare our method with that of Massanes and Brankov (2016).

Conference Presentation

Citation Download Citation

Dylan H. Shekter and Frank W. Samuelson "Efficiently calculating ROC curves, AUC, and uncertainty from 2AFC studies with finite samples", Proc. SPIE 11316, Medical Imaging 2020: Image Perception, Observer Performance, and Technology Assessment, 113160L (16 March 2020); https://doi.org/10.1117/12.2550601

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE

Includes PDF, HTML & Video, when available