A confidence paradigm for classification systems

Nathan J. Leap; Kenneth W. Bauer Jr.

doi:10.1117/12.776755

17 April 2008 A confidence paradigm for classification systems

Nathan J. Leap, Kenneth W. Bauer Jr.

Proceedings Volume 6968, Signal Processing, Sensor Fusion, and Target Recognition XVII; 69680U (2008) https://doi.org/10.1117/12.776755
Event: SPIE Defense and Security Symposium, 2008, Orlando, Florida, United States

Abstract

There is no universally accepted methodology to determine how much confidence one should have in a classifier output. This research proposes a framework to determine the level of confidence in an indication from a classifier system where the output is a measurement value. There are two types of confidence developed in this paper. The first is confidence in a classification system or classifier and is denoted classifier confidence. The second is the confidence in the output of a classification system or classifier. In this paradigm, we posit that the confidence in the output of a classifier should be, on average, equal to the confidence in the classifier as a whole (i.e., classifier confidence). The amount of confidence in a given classifier is estimated using multiattribute preference theory and forms the foundation for a quadratic confidence function that is applied to posterior probability estimates. Classifier confidence is currently determined based upon individual measurable value functions for classification accuracy, average entropy, and sample size, and the form of the overall measurable value function is multilinear based upon the assumption of weak difference independence. Using classifier confidence, a quadratic function is trained to be the confidence function which inputs a posterior probability and outputs the confidence in a given indication. In this paradigm, confidence is not equal to the posterior probability estimate but is related to it. This confidence measure is a direct link between traditional decision analysis techniques and traditional pattern recognition techniques. This methodology is applied to two real world data sets, and results show the sort of behavior that would be expected from a rational confidence measure.

Citation Download Citation

Nathan J. Leap and Kenneth W. Bauer Jr. "A confidence paradigm for classification systems", Proc. SPIE 6968, Signal Processing, Sensor Fusion, and Target Recognition XVII, 69680U (17 April 2008); https://doi.org/10.1117/12.776755

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE

Includes PDF, HTML & Video, when available