Modeling visual search behavior of breast radiologists using a deep convolution neural network

Suneeta Mall; Patrick C. Brennan; Claudia Mello-Thoms

doi:10.1117/1.JMI.5.3.035502

11 August 2018 Modeling visual search behavior of breast radiologists using a deep convolution neural network

Suneeta Mall, Patrick C. Brennan, Claudia Mello-Thoms

Author Affiliations +

Journal of Medical Imaging, Vol. 5, Issue 3, 035502 (August 2018). https://doi.org/10.1117/1.JMI.5.3.035502

Abstract

Visual search, the process of detecting and identifying objects using eye movements (saccades) and foveal vision, has been studied for identification of root causes of errors in the interpretation of mammograms. The aim of this study is to model visual search behavior of radiologists and their interpretation of mammograms using deep machine learning approaches. Our model is based on a deep convolutional neural network, a biologically inspired multilayer perceptron that simulates the visual cortex and is reinforced with transfer learning techniques. Eye-tracking data were obtained from eight radiologists (of varying experience levels in reading mammograms) reviewing 120 two-view digital mammography cases (59 cancers), and it has been used to train the model, which was pretrained with the ImageNet dataset for transfer learning. Areas of the mammogram that received direct (foveally fixated), indirect (peripherally fixated), or no (never fixated) visual attention were extracted from radiologists’ visual search maps (obtained by a head mounted eye-tracking device). These areas along with the radiologists’ assessment (including confidence in the assessment) of the presence of suspected malignancy were used to model: (1) radiologists’ decision, (2) radiologists’ confidence in such decisions, and (3) the attentional level (i.e., foveal, peripheral, or none) in an area of the mammogram. Our results indicate high accuracy and low misclassification in modeling such behaviors.

Citation Download Citation

Suneeta Mall, Patrick C. Brennan, and Claudia Mello-Thoms "Modeling visual search behavior of breast radiologists using a deep convolution neural network," Journal of Medical Imaging 5(3), 035502 (11 August 2018). https://doi.org/10.1117/1.JMI.5.3.035502

Received: 14 April 2018; Accepted: 24 July 2018; Published: 11 August 2018

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE

Includes PDF, HTML & Video, when available