We introduce an interesting interpretation of the ROC Curve that, subsequently, opens a new research paradigm. We define the "Diagnostician Operating Choice" (DOC) Curve to be the set of all (True Positive Probability/True Negative Probability) or ("skill in diseased population"/"skill in non-diseased population" when considered from the diagnostician's perspective) options made available to a particular radiologist when interpreting a particular diagnostic technology. The DOC Curve is, thus, the choice set presented to the diagnostician by their interaction with the technology. This new paradigm calls for tools that can measure the particular choice set of any particular individual radiologist interpreting a particular technology when applied in a particular clinical setting. Fundamental requirements for this paradigm are for the DOC Curve to be unique to individuals and constant across similar experimental conditions. To investigate constancy, we analyzed data from a reading study of 10 radiologists. Each radiologist interpreted the same set of 148 screening mammograms twice using a modified version of BI-RADS. ROC Curves for each radiologist were computed and compared between the two reading occasions with the CORROC2 program. None of the areas were statistically significantly different (p<0.05), providing confirmation (but not proof) of constancy across the two reading conditions. The DOC Curve paradigm suggests new areas of research focusing on the behavior in individuals interacting with technology. A clear need is for more efficient estimation of individual DOC Curves based on limited case sets. Paradoxically, the answer to this last problem might lie in using large population-based ("MRMC") studies to develop highly efficient and externally validated standardized testing tools for assessment of the individual.
An approach to sampling the U.S. population of mammographers was devised by Beam, Layde, Sullivan. They now have accumulated the reports of 108 radiologists, each reading the same random sample of 150 cases. We have applied multiple-reader, multiple case (MRMC) ROC analysis to their data and reconfirmed the earlier conclusion of Beam that the variability in their observations is dominated by the range of reader skill in their sample (private communication, C. Beam 2000). The purpose of the present paper is to demonstrate how the range of reader skill may become amplified when it is propagated into the corresponding expected benefit curves. This paper is a works-in-progress for which the full report has been submitted to the journal of Medical Decision Making.
This paper presents a statistical method to explore and assess variability among diagnosticians in their accuracy and the association between accuracy and characteristics of diagnosticians and patients. The method assumes random sampling from a population of patients. It is assumed the diagnosticians provide ordinal diagnostic ratings to all patients. In stage I, the Binormal Model is used to summarize the data into diagnostician-specific accuracy parameters at each patient covariate level. In stage II, the reduced data is then regressed on characteristics of the diagnosticians. Statistical inference is driven by bootstrapping. An application of the method to a national study of mammogram interpretation variability is presented. Empirical and theoretical evaluations are presented which substantiate the method. It will be shown that the model belongs to the well-known class of General Linear Models. The primary strength of the method is that it facilitates familiar and graphical approaches to the analysis of complex diagnostic ratings data arising from the simultaneous sampling of the population of diagnosticians as well as of the population of patients.
KEYWORDS: Mammography, Principal component analysis, Cancer, Imaging technologies, Medicine, Medical imaging, Statistical analysis, Radiology, Medical research, Analytical research
Each year, approximately 60% of all US women over the age of 40 utilize mammography. Through the matrix of an imaging technology, this Population of Patients (POP) interacts with a population of approximately 20,000 physicians who interpret mammograms in the US. This latter Population of Diagnosticians (POD) operationally serves as the interface between an image-centric healthcare technology system and patient. Methods: using data collected from a large POD and POP based study, I evaluate the distribution of several ROC curve-related parameters in the POD and explore the health policy implications of a population ROC curve for mammography. Results and Conclusions: Principal Components Analysis suggests that two Binormal parameters are sufficient to explain variation in the POD and implies that the Binormal model is foundational to Health Policy Research in Mammography. A population ROC curve based on percentiles of the POD can be used to set targets to achieve national health policy goals. Medical Image Perception science provides the framework. Alternatively, a restrictive policy can be envisioned using performance criteria based on area. However, the data suggests this sort of policy would be too costly in terms of reduced healthcare service capacity in the US in the face of burgeoning demands.
Access to the requested content is limited to institutions that have purchased or subscribe to SPIE eBooks.
You are receiving this notice because your organization may not have SPIE eBooks access.*
*Shibboleth/Open Athens users─please
sign in
to access your institution's subscriptions.
To obtain this item, you may purchase the complete book in print or electronic format on
SPIE.org.
INSTITUTIONAL Select your institution to access the SPIE Digital Library.
PERSONAL Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.