Jackknife free-response ROC methodology

Dev Prasad Chakraborty; Kevin S. Berbaum

doi:10.1117/12.533319

4 May 2004 Jackknife free-response ROC methodology

Dev Prasad Chakraborty, Kevin S. Berbaum

Proceedings Volume 5372, Medical Imaging 2004: Image Perception, Observer Performance, and Technology Assessment; (2004) https://doi.org/10.1117/12.533319
Event: Medical Imaging 2004, 2004, San Diego, California, United States

Abstract

Although ROC analysis is the accepted methodology for evaluation of diagnostic imaging systems, it has some serious shortcomings. By contrast, FROC methodology allows the observer to report multiple abnormalities per case, and uses the location of reported abnormalities to improve the measurement. Because ROC methodology has no way to allow multiple responses or use the location information, its statistical power will suffer. The FROC method has not enjoyed widespread acceptance because of concern about whether responses made to the same case can be treated as independent. We propose a new jackknife FROC method (JAFROC) that does not make the independence assumption. The new method combines elements of FROC and the Dorfman-Berbaum-Metz (DBM) multi-reader ROC methods. To compare the JAFROC method to an earlier free-response method (alternative free-response or AFROC method), and to the DBM method, which uses conventional ROC scoring, we developed a model for generating simulated FROC detection and location data. The simulation model is quite general and can be used to evaluate any method for analysis of multiple-response detection-and-localization data. It allowed us to examine null hypothesis (NH) behavior and statistical power of analytic methods. We found that AFROC analysis did not pass the NH test, being unduly conservative. Both the JAFROC method and the DBM passed the NH test, but JAFROC had more statistical power than the DBM method. The results of this comparison suggests that future studies of diagnostic performance may enjoy improved statistical power or reduced sample size requirements through the use of the JAFROC method.

Citation Download Citation

Dev Prasad Chakraborty and Kevin S. Berbaum "Jackknife free-response ROC methodology", Proc. SPIE 5372, Medical Imaging 2004: Image Perception, Observer Performance, and Technology Assessment, (4 May 2004); https://doi.org/10.1117/12.533319

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE

Includes PDF, HTML & Video, when available