JAFROC analysis revisited: figure-of-merit considerations for human observer studies

D. P. Chakraborty; Hong-Jun Yoon

doi:10.1117/12.810859

Translator Disclaimer

12 March 2009 JAFROC analysis revisited: figure-of-merit considerations for human observer studies

D. P. Chakraborty, Hong-Jun Yoon

Author Affiliations +

Proceedings Volume 7263, Medical Imaging 2009: Image Perception, Observer Performance, and Technology Assessment; 72630T (2009) https://doi.org/10.1117/12.810859
Event: SPIE Medical Imaging, 2009, Lake Buena Vista (Orlando Area), Florida, United States

Abstract

Jackknife alternative free-response receiver operating characteristic (JAFROC) is a method for measuring human observer performance in localization tasks. JAFROC is being increasingly used to evaluate imaging modalities because it has been shown to have greater statistical power than conventional receiver operating characteristic (ROC) analysis, which neglects location information. JAFROC neglects the non-lesion localization marks ("false positives") on abnormal images. JAFROC1 is an alternative method that includes these marks. Both methods are lesion-centric in the sense that they assign equal importance to all lesions; an image with many lesions would tend to dominate the performance metric, and clinically less significant lesions are treated identically as more significant ones. In this paper weighted JAFROC and JAFROC1 analyses are described that treat each abnormal image (not each lesion) as a unit of measurement and account for different lesion clinical significances (weights). Lesion-centric and weighted methods were tested using a simulator that includes multiple-reader multiple-case multiple-modality location level correlations. For comparison, ROC analysis was also tested where the rating of the highest rated mark on an image was assumed to be its "ROC" rating. The testing involved random numbers of lesions per image, random weights, case-mixes (ratio of normal to abnormal images) and different correlation structures. We found that for either JAFROC or JAFROC1, both lesion-centric and weighted analyses had correct NH behavior and comparable statistical powers. For either lesion-centric or weighted analyses JAFROC1 yielded the highest power, followed by JAFROC and ROC yielded the least power, confirming a recent study using a less flexible single-reader dual-modality simulator. Provided the number of normal cases is not too small, JAFROC1 is the preferred method for analyzing human observer free-response data. For either JAFROC or JAFROC1 weighted analysis is preferable.

Citation Download Citation

D. P. Chakraborty and Hong-Jun Yoon "JAFROC analysis revisited: figure-of-merit considerations for human observer studies", Proc. SPIE 7263, Medical Imaging 2009: Image Perception, Observer Performance, and Technology Assessment, 72630T (12 March 2009); https://doi.org/10.1117/12.810859

ACCESS THE FULL ARTICLE