Investigation of methods for calibration of classifier scores to probability of disease

Weijie Chen; Berkman Sahiner; Frank Samuelson; Aria Pezeshk; Nicholas Petrick

doi:10.1117/12.2082142

17 March 2015 Investigation of methods for calibration of classifier scores to probability of disease

Weijie Chen, Berkman Sahiner, Frank Samuelson, Aria Pezeshk, Nicholas Petrick

Proceedings Volume 9416, Medical Imaging 2015: Image Perception, Observer Performance, and Technology Assessment; 94161E (2015) https://doi.org/10.1117/12.2082142
Event: SPIE Medical Imaging, 2015, Orlando, Florida, United States

Abstract

Classifier scores in many diagnostic devices, such as computer-aided diagnosis systems, are usually on an arbitrary scale, the meaning of which is unclear. Calibration of classifier scores to a meaningful scale such as the probability of disease is potentially useful when such scores are used by a physician or another algorithm. In this work, we investigated the properties of two methods for calibrating classifier scores to probability of disease. The first is a semiparametric method in which the likelihood ratio for each score is estimated based on a semiparametric proper receiver operating characteristic model, and then an estimate of the probability of disease is obtained using the Bayes theorem assuming a known prevalence of disease. The second method is nonparametric in which isotonic regression via the pool-adjacent-violators algorithm is used. We employed the mean square error (MSE) and the Brier score to evaluate the two methods. We evaluate the methods under two paradigms: (a) the dataset used to construct the score-to-probability mapping function is used to calculate the performance metric (MSE or Brier score) (resubstitution); (b) an independent test dataset is used to calculate the performance metric (independent). Under our simulation conditions, the semiparametric method is found to be superior to the nonparametric method at small to medium sample sizes and the two methods appear to converge at large sample sizes. Our simulation results also indicate that the resubstitution bias may depend on the performance metric and, for the semiparametric method, the resubstitution bias is small when a reasonable number of cases (> 100 cases per class) are available.

Citation Download Citation

Weijie Chen, Berkman Sahiner, Frank Samuelson, Aria Pezeshk, and Nicholas Petrick "Investigation of methods for calibration of classifier scores to probability of disease", Proc. SPIE 9416, Medical Imaging 2015: Image Perception, Observer Performance, and Technology Assessment, 94161E (17 March 2015); https://doi.org/10.1117/12.2082142

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE

Includes PDF, HTML & Video, when available