Multi-reader multi-case analysis of variance software for diagnostic performance comparison of imaging modalities

Brian J. Smith; Stephen L. Hillis

doi:10.1117/12.2549075

16 March 2020 Multi-reader multi-case analysis of variance software for diagnostic performance comparison of imaging modalities

Brian J. Smith, Stephen L. Hillis

Author Affiliations +

Proceedings Volume 11316, Medical Imaging 2020: Image Perception, Observer Performance, and Technology Assessment; 113160K (2020) https://doi.org/10.1117/12.2549075
Event: SPIE Medical Imaging, 2020, Houston, Texas, United States

Abstract

A common study design for comparing the diagnostic performance of imaging modalities is to obtain modalityspecific ratings from multiple readers of multiple cases whose true statuses are known. Typically, there is overlap between the modalities, readers, and/or cases for which special analytical methods are needed to perform statistical comparisons. We describe our new R software package MRMCaov, which is designed for multi-reader multi-case comparisons of two or more imaging modalities. The software allows for the comparison of reader performance metrics, such as area under the receiver operating characteristic curve (ROC AUC), with analysis of variance methods originally proposed by Obuchowski and Rockette (1995) and later unified and improved by Hillis and colleagues (2005, 2007, 2008, 2018). MRMCaov is an open-source package with an integrated command-line interface for performing multi-reader multi-case statistical analysis, plotting, and presenting results. Features of the package include (1) ROC curves estimated parametrically or non-parametrically; (2) reader-specific ROC curves and performance metrics; (3) user-definable performance metrics; (4) modality-specific estimates of mean performance along with confidence intervals and p-values for statistical comparisons; (5) support for factorial, nested, or partially paired study designs; (6) inference for random readers and cases, random readers and fixed cases, or fixed readers and random cases; (7) DeLong, jackknife, or unbiased covariance estimation; and (8) compatibility with Microsoft Windows, Mac OS, and Linux.

Conference Presentation

Citation Download Citation

Brian J. Smith and Stephen L. Hillis "Multi-reader multi-case analysis of variance software for diagnostic performance comparison of imaging modalities", Proc. SPIE 11316, Medical Imaging 2020: Image Perception, Observer Performance, and Technology Assessment, 113160K (16 March 2020); https://doi.org/10.1117/12.2549075

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE

Includes PDF, HTML & Video, when available