Measuring modality ordering consistency of observer performance paradigms

D. P. Chakraborty; Federica Zanca

doi:10.1117/12.844997

23 February 2010 Measuring modality ordering consistency of observer performance paradigms

D. P. Chakraborty, Federica Zanca

Proceedings Volume 7627, Medical Imaging 2010: Image Perception, Observer Performance, and Technology Assessment; 762711 (2010) https://doi.org/10.1117/12.844997
Event: SPIE Medical Imaging, 2010, San Diego, California, United States

Abstract

Two observer performance paradigms applied to the same modalities, readers and cases are said to order the modalities consistently if both confirm the same sign (positive or negative) of the figure of merit difference. The aim of this work was to develop a modality ordering consistency measure. The paradigms considered were receiver operating characteristic (ROC) and jackknife alternative free-response ROC (JAFROC). Clinical FROC data from a previous study was used. Using the highest rating method ROC ratings were inferred from FROC ratings. JAFROC analyses of the FROC data and Dorfman-Berbaum-Metz multiple-reader multiple-case (DBM-MRMC) analysis of the inferred ROC data showed significant and consistent differences in the two figures of merit. Additionally 2000 bootstrap data sets were sampled and analyzed by JAFROC and DBM-MRMC. It was found that a positive JAFROC figure of merit difference was 101 times more likely when the ROC difference was positive than when the ROC difference was negative (odds ratio = 101). Valid modality ordering consistency (or inconsistency) claims are possible only when both figures of merit differences are statistically significant. For those bootstraps where both JAFROC and ROC yielded significant differences there were no inconsistent orderings. The effect of artificially degrading JAFROC performance was investigated. It was found that the odds ratio was more sensitive to the degradation. The results in this work are likely to be optimistic. A more realistic test of modality ordering consistency would require two separate studies (FROC and ROC) using the same readers and cases.

Citation Download Citation

D. P. Chakraborty and Federica Zanca "Measuring modality ordering consistency of observer performance paradigms", Proc. SPIE 7627, Medical Imaging 2010: Image Perception, Observer Performance, and Technology Assessment, 762711 (23 February 2010); https://doi.org/10.1117/12.844997

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE

Includes PDF, HTML & Video, when available

Members: $17.00

Non-members: $21.00 ADD TO CART

PROCEEDINGS
9 PAGES

DOWNLOAD PAPER SAVE TO MY LIBRARY

GET CITATION

RIGHTS & PERMISSIONS

Get copyright permission Get copyright permission on Copyright Marketplace

KEYWORDS

Statistical analysis

Contamination

Data acquisition

Electronic filtering

Diagnostics

Medical imaging

Radiology

Show All Keywords

Keywords/Phrases

Search In:

Publication Years