Translator Disclaimer
Paper
9 March 2010 Training variability in the evaluation of automated classifiers
Author Affiliations +
Abstract
The evaluation of automated classifiers in computer-aided diagnosis of medical images often involves a training dataset for classifier design and a test dataset for performance estimation in terms of, e.g., area under the receiver operating characteristic (ROC) curve, or AUC. The traditional approach to assess the uncertainty of the estimated AUC only considers the finite testing set as the source of variability. However, a finite training set is also a random sample and the AUC varies with varying training sets. We categorize the assessment of classifiers into three levels and provide analytical expressions for the variance of the estimated AUC at each level: (1) training treated as a fixed effect, the estimated performance generalizable only to the population of testing sets; (2) training treated as a random effect, the estimated performance generalizable to both the population of training sets and the population of testing sets; (3) training treated as a random effect, performance averaged over training sets generalizable to both the population of training sets and the population of testing sets. The two sources of variability - training and testing - in automated classifiers are analogous to readers and cases in the multi-reader multi-case (MRMC) ROC paradigm in reader studies. We show the one-to-one analogy between the automated classifiers and human readers at these three levels as well as the practical difference in estimating their performance, especially regarding variance.
© (2010) COPYRIGHT Society of Photo-Optical Instrumentation Engineers (SPIE). Downloading of the abstract is permitted for personal use only.
Weijie Chen and Brandon D. Gallas "Training variability in the evaluation of automated classifiers", Proc. SPIE 7624, Medical Imaging 2010: Computer-Aided Diagnosis, 762404 (9 March 2010); https://doi.org/10.1117/12.844605
PROCEEDINGS
9 PAGES


SHARE
Advertisement
Advertisement
Back to Top