Training variability in the evaluation of automated classifiers

Weijie Chen; Brandon D. Gallas

doi:10.1117/12.844605

9 March 2010 Training variability in the evaluation of automated classifiers

Weijie Chen, Brandon D. Gallas

Proceedings Volume 7624, Medical Imaging 2010: Computer-Aided Diagnosis; 762404 (2010) https://doi.org/10.1117/12.844605
Event: SPIE Medical Imaging, 2010, San Diego, California, United States

Abstract

The evaluation of automated classifiers in computer-aided diagnosis of medical images often involves a training dataset for classifier design and a test dataset for performance estimation in terms of, e.g., area under the receiver operating characteristic (ROC) curve, or AUC. The traditional approach to assess the uncertainty of the estimated AUC only considers the finite testing set as the source of variability. However, a finite training set is also a random sample and the AUC varies with varying training sets. We categorize the assessment of classifiers into three levels and provide analytical expressions for the variance of the estimated AUC at each level: (1) training treated as a fixed effect, the estimated performance generalizable only to the population of testing sets; (2) training treated as a random effect, the estimated performance generalizable to both the population of training sets and the population of testing sets; (3) training treated as a random effect, performance averaged over training sets generalizable to both the population of training sets and the population of testing sets. The two sources of variability - training and testing - in automated classifiers are analogous to readers and cases in the multi-reader multi-case (MRMC) ROC paradigm in reader studies. We show the one-to-one analogy between the automated classifiers and human readers at these three levels as well as the practical difference in estimating their performance, especially regarding variance.

Citation Download Citation

Weijie Chen and Brandon D. Gallas "Training variability in the evaluation of automated classifiers", Proc. SPIE 7624, Medical Imaging 2010: Computer-Aided Diagnosis, 762404 (9 March 2010); https://doi.org/10.1117/12.844605

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE

Includes PDF, HTML & Video, when available