Comparing agreement measures

Wei-min Liu; Brandon D. Gallas

doi:10.1117/12.773037

24 March 2008 Comparing agreement measures

Wei-min Liu, Brandon D. Gallas

Proceedings Volume 6917, Medical Imaging 2008: Image Perception, Observer Performance, and Technology Assessment; 69170D (2008) https://doi.org/10.1117/12.773037
Event: Medical Imaging, 2008, San Diego, California, United States

Abstract

Agreement is estimated by comparing correlated/paired scores (e.g. the scores from two doctors reading the same set of images), such as the correlation coefficient and measures of concordance. Some variance estimation techniques for these measures are also available in the literature. In this work, we compared four agreement measures: the widely used Pearson's product moment correlation coefficient, Kendall's tau, and two measures that are generalizations of AUC, the area under the receiver operating characteristics (ROC) curve. The generalization allows for ordinal truth that is polytomous (multi-state) or even continuous instead of just binary, and thus AUC is a special case. We investigate how these measures behave in a multi-reader multi-case (MRMC) simulation experiment as we change the intrinsic correlation and number of rating levels. We also investigate a few variance estimation techniques for these measures that are available in the literature. These agreement measures will help investigators developing model observers to compare their models against a human on a case-by-case basis instead of with a summary figure of merit that requires and is limited by binary truth, like AUC. The model observer AUC can equal the human observer AUC, while making very different decisions on a case-by-case basis.

Citation Download Citation

Wei-min Liu and Brandon D. Gallas "Comparing agreement measures", Proc. SPIE 6917, Medical Imaging 2008: Image Perception, Observer Performance, and Technology Assessment, 69170D (24 March 2008); https://doi.org/10.1117/12.773037

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE

Includes PDF, HTML & Video, when available

Members: $17.00

Non-members: $21.00 ADD TO CART

PROCEEDINGS
11 PAGES

DOWNLOAD PAPER SAVE TO MY LIBRARY

GET CITATION

RIGHTS & PERMISSIONS

Get copyright permission Get copyright permission on Copyright Marketplace

KEYWORDS

Binary data

Monte Carlo methods

Computer simulations

Receivers

Statistical analysis

Gold

Medical imaging

Show All Keywords

Keywords/Phrases

Search In:

Publication Years