Forming a reference standard from LIDC data: impact of reader agreement on reported CAD performance

Robert Ochs; Hyun J. Kim; Erin Angel; Christoph Panknin; Michael McNitt-Gray; Matthew Brown

doi:10.1117/12.707916

30 March 2007 Forming a reference standard from LIDC data: impact of reader agreement on reported CAD performance

Robert Ochs, Hyun J. Kim, Erin Angel, Christoph Panknin, Michael McNitt-Gray, Matthew Brown

Proceedings Volume 6514, Medical Imaging 2007: Computer-Aided Diagnosis; 65142A (2007) https://doi.org/10.1117/12.707916
Event: Medical Imaging, 2007, San Diego, CA, United States

Abstract

The Lung Image Database Consortium (LIDC) has provided a publicly available collection of CT images with nodule markings from four radiologists. The LIDC protocol does not require radiologists to reach a consensus during the reading process, and as a result, there are varying levels of reader agreement for each potential nodule with no explicit reference standard for nodules. The purpose of this work was to investigate the effects of the level of reader agreement on the development of a reference standard and the subsequent impact on CAD performance. Ninety series were downloaded from the LIDC database. Four different reference standards were created based on the markings of the LIDC radiologists, reflecting four different levels of reader agreement. All series were analyzed with a research CAD system and its performance was measured against each of the four standards. Between the standards with the lowest (any 1 of 4 readers) and highest (all 4 readers) required level of reader agreement, the number of nodules ⩾ 3 mm decreased 48% (from 174 to 90) and CAD sensitivity for nodules ⩾ 3 mm increased from 0.70 ± 0.34 to 0.79 ± 0.35. Between the same reference standards, the number of nodules < 3 mm decreased 84% (from 483 to 75) and CAD sensitivity for nodules < 3 mm increased from 0.30 ± 0.29 to 0.51 ± 0.45. This research illustrates the importance of indicating the method used to form the reference standard, since the method influences both the number of nodules and reported CAD performance.

Citation Download Citation

Robert Ochs, Hyun J. Kim, Erin Angel, Christoph Panknin, Michael McNitt-Gray, and Matthew Brown "Forming a reference standard from LIDC data: impact of reader agreement on reported CAD performance", Proc. SPIE 6514, Medical Imaging 2007: Computer-Aided Diagnosis, 65142A (30 March 2007); https://doi.org/10.1117/12.707916

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE

Includes PDF, HTML & Video, when available