The Lung Image Database Consortium (LIDC): a quality assurance model for the collection of expert-defined truth in lung-nodule-based image analysis studies

Samuel G. Armato III; Rachael Y. Roberts; Geoffrey McLennan; Michael F. McNitt-Gray; David Yankelevitz; Ella A. Kazerooni; Edwin J. R. van Beek; Heber MacMahon; Denise R. Aberle M.D.; Charles R. Meyer; Anthony P. Reeves; Claudia I. Henschke; Eric A. Hoffman; Barbara Y. Croft; Laurence P. Clarke

doi:10.1117/12.713227

30 March 2007 The Lung Image Database Consortium (LIDC): a quality assurance model for the collection of expert-defined truth in lung-nodule-based image analysis studies

Samuel G. Armato III, Rachael Y. Roberts, Geoffrey McLennan, Michael F. McNitt-Gray, David Yankelevitz, Ella A. Kazerooni, Edwin J. R. van Beek, Heber MacMahon, Denise R. Aberle M.D., Charles R. Meyer, Anthony P. Reeves, Claudia I. Henschke, Eric A. Hoffman, Barbara Y. Croft, Laurence P. Clarke

Author Affiliations +

Proceedings Volume 6514, Medical Imaging 2007: Computer-Aided Diagnosis; 651429 (2007) https://doi.org/10.1117/12.713227
Event: Medical Imaging, 2007, San Diego, CA, United States

Abstract

The development of computer-aided diagnostic (CAD) systems requires an initial establishment of "truth" by expert human observers. Potential inconsistencies in the "truth" data must be identified and corrected before investigators can rely on this data. We developed a quality assurance model to supplement the "truth" collection process for lung nodules on CT scans. A two-phase process was established for the interpretation of CT scans by four radiologists. During the initial "blinded read," radiologists independently assigned lesions they identified into one of three categories: "nodule ⩾ 3mm," "nodule < 3mm," or "non-nodule ⩾ 3mm." During the subsequent "unblinded read," the blinded read results of all radiologists were revealed. The radiologists then independently reviewed their marks along with their colleague's marks; a radiologist's own marks could be left unchanged, deleted, switched in terms of lesion category, or additional marks could be added. The final set of marks underwent quality assurance, which consisted of identification of potential errors that occurred during the reading process and error correction. All marks were visually grouped into discrete nodules. Six categories of potential error were defined, and any nodule with a mark that satisfied the criterion for one of these categories was referred to the radiologist who assigned the mark in question. The radiologist either corrected the mark or confirmed that the mark was intentional. A total of 829 nodules were identified by at least one radiologist in 100 CT scans through the two-phase process designed to capture "truth." The quality assurance process yielded 81 nodules with potential errors. The establishment of "truth" must incorporate a quality assurance model to guarantee the integrity of the "truth" that will provide the basis for the training and testing of CAD systems.

Citation Download Citation

Samuel G. Armato III, Rachael Y. Roberts, Geoffrey McLennan, Michael F. McNitt-Gray, David Yankelevitz, Ella A. Kazerooni, Edwin J. R. van Beek, Heber MacMahon, Denise R. Aberle M.D., Charles R. Meyer, Anthony P. Reeves, Claudia I. Henschke, Eric A. Hoffman, Barbara Y. Croft, and Laurence P. Clarke "The Lung Image Database Consortium (LIDC): a quality assurance model for the collection of expert-defined truth in lung-nodule-based image analysis studies", Proc. SPIE 6514, Medical Imaging 2007: Computer-Aided Diagnosis, 651429 (30 March 2007); https://doi.org/10.1117/12.713227

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE

Includes PDF, HTML & Video, when available