Comparison of classifier performance estimators: a simulation study

Weijie Chen; Robert F. Wagner; Waleed A. Yousef; Brandon D. Gallas

doi:10.1117/12.811584

12 March 2009 Comparison of classifier performance estimators: a simulation study

Weijie Chen, Robert F. Wagner, Waleed A. Yousef, Brandon D. Gallas

Author Affiliations +

Proceedings Volume 7263, Medical Imaging 2009: Image Perception, Observer Performance, and Technology Assessment; 72630X (2009) https://doi.org/10.1117/12.811584
Event: SPIE Medical Imaging, 2009, Lake Buena Vista (Orlando Area), Florida, United States

Abstract

We aim to compare resampling-based estimators of the area under the ROC curve (AUC) of a classifier with a Monte Carlo simulation study. The comparison is in terms of bias, variance, and mean square error. We also examine the corresponding variance estimators of these AUC estimators. We compared three AUC estimators: the hold-out (HO) estimator, the leave-one-out cross validation (LOOCV) estimator, and the leave-pair-out bootstrap (LPOB) estimator. Each performance estimator has its own variability estimator. In our simulations, in terms of the mean square error, HO is always the worst and the ranking of the other two estimators depends on the interplay of sample size, dimensionality, and the population separability. In terms of estimator variability, the LPOB is the least variable estimator and the HO is the most variable estimator. The results also show that the estimation of the variance of LPOB using the influence function approach with a finite data set is unbiased or conservatively biased whereas the estimation of the variance of the LOOCV or the HO is downwardly (i.e., anti-conservatively) biased.

Citation Download Citation

Weijie Chen, Robert F. Wagner, Waleed A. Yousef, and Brandon D. Gallas "Comparison of classifier performance estimators: a simulation study", Proc. SPIE 7263, Medical Imaging 2009: Image Perception, Observer Performance, and Technology Assessment, 72630X (12 March 2009); https://doi.org/10.1117/12.811584

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

;

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE