Translator Disclaimer
Paper
24 March 2008 Comparing agreement measures
Author Affiliations +
Abstract
Agreement is estimated by comparing correlated/paired scores (e.g. the scores from two doctors reading the same set of images), such as the correlation coefficient and measures of concordance. Some variance estimation techniques for these measures are also available in the literature. In this work, we compared four agreement measures: the widely used Pearson's product moment correlation coefficient, Kendall's tau, and two measures that are generalizations of AUC, the area under the receiver operating characteristics (ROC) curve. The generalization allows for ordinal truth that is polytomous (multi-state) or even continuous instead of just binary, and thus AUC is a special case. We investigate how these measures behave in a multi-reader multi-case (MRMC) simulation experiment as we change the intrinsic correlation and number of rating levels. We also investigate a few variance estimation techniques for these measures that are available in the literature. These agreement measures will help investigators developing model observers to compare their models against a human on a case-by-case basis instead of with a summary figure of merit that requires and is limited by binary truth, like AUC. The model observer AUC can equal the human observer AUC, while making very different decisions on a case-by-case basis.
© (2008) COPYRIGHT Society of Photo-Optical Instrumentation Engineers (SPIE). Downloading of the abstract is permitted for personal use only.
Wei-min Liu and Brandon D. Gallas "Comparing agreement measures", Proc. SPIE 6917, Medical Imaging 2008: Image Perception, Observer Performance, and Technology Assessment, 69170D (24 March 2008); https://doi.org/10.1117/12.773037
PROCEEDINGS
11 PAGES


SHARE
Advertisement
Advertisement
RELATED CONTENT


Back to Top