Translator Disclaimer
12 March 2009 Comparison of classifier performance estimators: a simulation study
Author Affiliations +
We aim to compare resampling-based estimators of the area under the ROC curve (AUC) of a classifier with a Monte Carlo simulation study. The comparison is in terms of bias, variance, and mean square error. We also examine the corresponding variance estimators of these AUC estimators. We compared three AUC estimators: the hold-out (HO) estimator, the leave-one-out cross validation (LOOCV) estimator, and the leave-pair-out bootstrap (LPOB) estimator. Each performance estimator has its own variability estimator. In our simulations, in terms of the mean square error, HO is always the worst and the ranking of the other two estimators depends on the interplay of sample size, dimensionality, and the population separability. In terms of estimator variability, the LPOB is the least variable estimator and the HO is the most variable estimator. The results also show that the estimation of the variance of LPOB using the influence function approach with a finite data set is unbiased or conservatively biased whereas the estimation of the variance of the LOOCV or the HO is downwardly (i.e., anti-conservatively) biased.
© (2009) COPYRIGHT Society of Photo-Optical Instrumentation Engineers (SPIE). Downloading of the abstract is permitted for personal use only.
Weijie Chen, Robert F. Wagner, Waleed A. Yousef, and Brandon D. Gallas "Comparison of classifier performance estimators: a simulation study", Proc. SPIE 7263, Medical Imaging 2009: Image Perception, Observer Performance, and Technology Assessment, 72630X (12 March 2009);

Back to Top