Comparison of classifier performance estimators: a simulation study

Weijie Chen; Robert F. Wagner; Waleed A. Yousef; Brandon D. Gallas

doi:10.1117/12.811584

Translator Disclaimer

12 March 2009 Comparison of classifier performance estimators: a simulation study

Weijie Chen, Robert F. Wagner, Waleed A. Yousef, Brandon D. Gallas

Author Affiliations +

Proceedings Volume 7263, Medical Imaging 2009: Image Perception, Observer Performance, and Technology Assessment; 72630X (2009) https://doi.org/10.1117/12.811584
Event: SPIE Medical Imaging, 2009, Lake Buena Vista (Orlando Area), Florida, United States

Abstract

We aim to compare resampling-based estimators of the area under the ROC curve (AUC) of a classifier with a Monte Carlo simulation study. The comparison is in terms of bias, variance, and mean square error. We also examine the corresponding variance estimators of these AUC estimators. We compared three AUC estimators: the hold-out (HO) estimator, the leave-one-out cross validation (LOOCV) estimator, and the leave-pair-out bootstrap (LPOB) estimator. Each performance estimator has its own variability estimator. In our simulations, in terms of the mean square error, HO is always the worst and the ranking of the other two estimators depends on the interplay of sample size, dimensionality, and the population separability. In terms of estimator variability, the LPOB is the least variable estimator and the HO is the most variable estimator. The results also show that the estimation of the variance of LPOB using the influence function approach with a finite data set is unbiased or conservatively biased whereas the estimation of the variance of the LOOCV or the HO is downwardly (i.e., anti-conservatively) biased.

Citation Download Citation

Weijie Chen, Robert F. Wagner, Waleed A. Yousef, and Brandon D. Gallas "Comparison of classifier performance estimators: a simulation study", Proc. SPIE 7263, Medical Imaging 2009: Image Perception, Observer Performance, and Technology Assessment, 72630X (12 March 2009); https://doi.org/10.1117/12.811584

ACCESS THE FULL ARTICLE