Dr. Frank W. Samuelson Profile

Dr. Frank W. Samuelson

Physicist at US Food and Drug Administration

SPIE Involvement:

Conference Chair | Editor | Author | Instructor

Publications (27)

Proceedings Article | 16 March 2020 Presentation + Paper

Efficiently calculating ROC curves, AUC, and uncertainty from 2AFC studies with finite samples

Dylan Shekter, Frank Samuelson

Proc. SPIE. 11316, Medical Imaging 2020: Image Perception, Observer Performance, and Technology Assessment

KEYWORDS: Statistical analysis, Imaging systems, Error analysis, Diagnostics, Imaging devices, Medical imaging

Read Abstract +

Proceedings Article | 16 March 2020 Presentation + Paper

Human observer templates for lesion discrimination tasks

Craig Abbey, Frank Samuelson, Rongping Zeng, John Boone, Miguel Eckstein, Kyle Myers

Proc. SPIE. 11316, Medical Imaging 2020: Image Perception, Observer Performance, and Technology Assessment

KEYWORDS: Signal to noise ratio, Apodization, X-ray computed tomography, Imaging systems, Spatial frequencies, Diagnostics, Lung, Image classification

Read Abstract +

SPIE Journal Paper | 26 February 2020

Computational reader design and statistical performance evaluation of an in-silico imaging clinical trial comparing digital breast tomosynthesis with full-field digital mammography

Rongping Zeng, Frank Samuelson, Diksha Sharma, Andreu Badal, Graff Christian, Stephen Glick, Kyle Myers, Aldo Badano

JMI Vol. 7 Issue 04

KEYWORDS: Digital breast tomosynthesis, Breast, Performance modeling, Signal detection, Data modeling, 3D modeling, Imaging systems, Statistical analysis, Statistical modeling, Clinical trials

Read Abstract +

A recent study reported on an in-silico imaging trial that evaluated the performance of digital breast tomosynthesis (DBT) as a replacement for full-field digital mammography (FFDM) for breast cancer screening. In this in-silico trial, the whole imaging chain was simulated, including the breast phantom generation, the x-ray transport process, and computational readers for image interpretation. We focus on the design and performance characteristics of the computational reader in the above-mentioned trial. Location-known lesion (spiculated mass and clustered microcalcifications) detection tasks were used to evaluate the imaging system performance. The computational readers were designed based on the mechanism of a channelized Hotelling observer (CHO), and the reader models were selected to trend human performance. Parameters were tuned to ensure stable lesion detectability. A convolutional CHO that can adapt a round channel function to irregular lesion shapes was compared with the original CHO and was found to be suitable for detecting clustered microcalcifications but was less optimal in detecting spiculated masses. A three-dimensional CHO that operated on the multiple slices was compared with a two-dimensional (2-D) CHO that operated on three versions of 2-D slabs converted from the multiple slices and was found to be optimal in detecting lesions in DBT. Multireader multicase reader output analysis was used to analyze the performance difference between FFDM and DBT for various breast and lesion types. The results showed that DBT was more beneficial in detecting masses than detecting clustered microcalcifications compared with FFDM, consistent with the finding in a clinical imaging trial. Statistical uncertainty smaller than 0.01 standard error for the estimated performance differences was achieved with a dataset containing approximately 3000 breast phantoms. The computational reader design methodology presented provides evidence that model observers can be useful in-silico tools for supporting the performance comparison of breast imaging systems.

Proceedings Article | 4 March 2019 Presentation + Paper

Adaptive sample size re-estimation in MRMC studies

Weijie Chen, Zhipeng Huang, Frank Samuelson, Lucas Tcheuko

Proc. SPIE. 10952, Medical Imaging 2019: Image Perception, Observer Performance, and Technology Assessment

KEYWORDS: Human-machine interfaces, Statistical analysis, Error analysis, Diagnostics, Computer simulations, Imaging devices, Medical imaging, Software development, Clinical trials

Read Abstract +

Proceedings Article | 4 March 2019 Paper

Template models for forced-localization tasks

Craig Abbey, Frank Samuelson, Rongping Zeng, John Boone, Miguel Eckstein, Kyle Myers

Proc. SPIE. 10952, Medical Imaging 2019: Image Perception, Observer Performance, and Technology Assessment

KEYWORDS: Apodization, Statistical analysis, Visual process modeling, Data modeling, Error analysis, Image analysis, Image classification, Statistical modeling, Performance modeling, Eye models

Read Abstract +

SPIE Journal Paper | 21 January 2019

Impact of prevalence and case distribution in lab-based diagnostic imaging studies

Brandon Gallas, Weijie Chen, Elodia Cole, Robert Ochs, Nicholas Petrick, Etta Pisano, Berkman Sahiner, Frank Samuelson, Kyle Myers

JMI Vol. 6 Issue 01

KEYWORDS: Atomic force microscopy, Cancer, Breast, Statistical analysis, Diagnostics, Mammography, Image processing, Digital mammography, Error analysis, Medical imaging

Read Abstract +

Proceedings Article | 7 March 2018 Presentation + Paper

Analyzing ROC curves using the effective set-size model

Frank Samuelson, Craig Abbey, Xin He

Proc. SPIE. 10577, Medical Imaging 2018: Image Perception, Observer Performance, and Technology Assessment

KEYWORDS: Computer aided diagnosis and therapy, Data modeling, Imaging systems, Diagnostics, Medical imaging, Mammography, Signal detection, Systems modeling, Solid modeling, Digital breast tomosynthesis

Read Abstract +

The Effective Set-Size model has been used to describe uncertainty in various signal detection experiments. The model regards images as if they were an effective number (M*) of searchable locations, where the observer treats each location as a location-known-exactly detection task with signals having average detectability d'. The model assumes a rational observer behaves as if he searches an effective number of independent locations and follows signal detection theory at each location. Thus the location-known-exactly detectability (d') and the effective number of independent locations M* fully characterize search performance. In this model the image rating in a single-response task is assumed to be the maximum response that the observer would assign to these many locations. The model has been used by a number of other researchers, and is well corroborated. We examine this model as a way of differentiating imaging tasks that radiologists perform. Tasks involving more searching or location uncertainty may have higher estimated M* values. In this work we applied the Effective Set-Size model to a number of medical imaging data sets. The data sets include radiologists reading screening and diagnostic mammography with and without computer-aided diagnosis (CAD), and breast tomosynthesis. We developed an algorithm to fit the model parameters using two-sample maximum-likelihood ordinal regression, similar to the classic bi-normal model. The resulting model ROC curves are rational and fit the observed data well. We find that the distributions of M* and d' differ significantly among these data sets, and differ between pairs of imaging systems within studies. For example, on average tomosynthesis increased readers’ d' values, while CAD reduced the M* parameters. We demonstrate that the model parameters M* and d' are correlated. We conclude that the Effective Set-Size model may be a useful way of differentiating location uncertainty from the diagnostic uncertainty in medical imaging tasks.

Proceedings Article | 10 March 2017 Presentation + Paper

Re-use of pilot data and interim analysis of pivotal data in MRMC studies: a simulation study

Weijie Chen, Frank Samuelson, Berkman Sahiner, Nicholas Petrick

Proc. SPIE. 10136, Medical Imaging 2017: Image Perception, Observer Performance, and Technology Assessment

KEYWORDS: Gold, Statistical analysis, Cancer, Data modeling, Error analysis, Diagnostics, Computer simulations, Imaging devices, Medical imaging, Monte Carlo methods, Medical devices, Performance modeling, Instrument modeling

Read Abstract +

Proceedings Article | 9 March 2017 Presentation + Paper

In silico imaging clinical trials for regulatory evaluation: initial considerations for VICTRE, a demonstration study

Aldo Badano, Andreu Badal, Stephen Glick, Christian Graff, Frank Samuelson, Diksha Sharma, Rongping Zeng

Proc. SPIE. 10132, Medical Imaging 2017: Physics of Medical Imaging

KEYWORDS: Breast, Safety, Cancer, Imaging systems, Sensors, Imaging technologies, Physics, Computer simulations, Mammography, Algorithm development, Instrument modeling, New and emerging technologies, Digital breast tomosynthesis, Clinical trials

Read Abstract +

Proceedings Article | 24 March 2016 Paper

Ranking inconsistencies in the assessment of digital breast tomosynthesis (DBT) reconstruction algorithms using a location-known task and a search task

Xin He, Rongping Zeng, Frank Samuelson, Berkman Sahiner

Proc. SPIE. 9787, Medical Imaging 2016: Image Perception, Observer Performance, and Technology Assessment

KEYWORDS: Breast, Statistical analysis, Computer aided diagnosis and therapy, Computer simulations, Image analysis, Medical imaging, Image quality, Reconstruction algorithms, Detection theory, Digital breast tomosynthesis, Algorithms

Read Abstract +

Proceedings Article | 24 March 2016 Paper

Three scenarios of ranking inconsistencies involving search tasks

Xin He, Frank Samuelson, Rongping Zeng, Berkman Sahiner

Proc. SPIE. 9787, Medical Imaging 2016: Image Perception, Observer Performance, and Technology Assessment

KEYWORDS: Imaging systems, Image processing, Medical imaging, Image quality, Solids, Image classification, Reconstruction algorithms, Signal detection, Digital breast tomosynthesis

Read Abstract +

Proceedings Article | 24 March 2016 Paper

A utility/cost analysis of breast cancer risk prediction algorithms

Craig Abbey, Yirong Wu, Elizabeth Burnside, Adam Wunderlich, Frank Samuelson, John Boone

Proc. SPIE. 9787, Medical Imaging 2016: Image Perception, Observer Performance, and Technology Assessment

KEYWORDS: Breast, Cancer, Breast cancer, Ultrasonography, Magnetic resonance imaging, Diagnostics, Mammography, Electronic filtering, Radiology, Algorithm development, Digital mammography, Lawrencium, Algorithms

Read Abstract +

Proceedings Article | 20 March 2015 Paper

Potential reasons for differences in CAD effectiveness evaluated using laboratory and clinical studies

Xin He, Frank Samuelson, Rongping Zeng, Berkman Sahiner

Proc. SPIE. 9414, Medical Imaging 2015: Computer-Aided Diagnosis

KEYWORDS: Computer aided diagnosis and therapy, Cancer, Breast cancer, Mammography, Computer aided design, Performance modeling, Binary data, CAD systems, Solid modeling, Lawrencium

Read Abstract +

Proceedings Article | 17 March 2015 Paper

Investigation of methods for calibration of classifier scores to probability of disease

Weijie Chen, Berkman Sahiner, Frank Samuelson, Aria Pezeshk, Nicholas Petrick

Proc. SPIE. 9416, Medical Imaging 2015: Image Perception, Observer Performance, and Technology Assessment

KEYWORDS: Statistical analysis, Computer aided diagnosis and therapy, Imaging systems, Calibration, Error analysis, Diagnostics, Receivers, Computer simulations, Monte Carlo methods, Binary data

Read Abstract +

Proceedings Article | 17 March 2015 Paper

Approximate maximum likelihood estimation of scanning observer templates

Craig Abbey, Frank Samuelson, Adam Wunderlich, Lucretiu Popescu, Miguel Eckstein, John Boone

Proc. SPIE. 9416, Medical Imaging 2015: Image Perception, Observer Performance, and Technology Assessment

KEYWORDS: Mathematical modeling, Data modeling, Error analysis, Image analysis, Medical imaging, Monte Carlo methods, Lab on a chip, Convolution, Performance modeling, Systems modeling

Read Abstract +

Proceedings Article | 17 March 2015 Paper

Prevalence learning and decision making in a visual search task: an equivalent ideal observer approach

Xin He, Frank Samuelson, Rongping Zeng, Berkman Sahiner

Proc. SPIE. 9416, Medical Imaging 2015: Image Perception, Observer Performance, and Technology Assessment

KEYWORDS: Mathematical modeling, Visual process modeling, Data modeling, Visualization, Image processing, Mathematics, Electronic filtering, Binary data, Process modeling, Lawrencium

Read Abstract +

SPIE Journal Paper | 28 August 2014

Comparison of semiparametric receiver operating characteristic models on observer data

Frank Samuelson, Xin He

JMI Vol. 1 Issue 03

KEYWORDS: Data modeling, Solid modeling, Statistical modeling, Receivers, Computer aided diagnosis and therapy, Medical imaging, Computer aided design, Mammography, Electronic filtering, Monte Carlo methods

Read Abstract +

Proceedings Article | 11 March 2014 Paper

Discover common properties of human observers' visual search and mathematical observers' scanning PART II: emperical studies using human and model observers

Xin He, Frank Samuelson, Berkman Sahiner

Proc. SPIE. 9037, Medical Imaging 2014: Image Perception, Observer Performance, and Technology Assessment

KEYWORDS: Mathematical modeling, Visual process modeling, Visualization, Optical inspection, Gaussian filters, Signal generators, Electronic filtering, Algorithm development, Performance modeling, Systems modeling

Read Abstract +

Proceedings Article | 11 March 2014 Paper

Discover common properties of human observers' visual search and mathematical observers' scanning PART I: theory and conjecture

Xin He, Frank Samuelson, Rongping Zeng, Berkman Sahiner

Proc. SPIE. 9037, Medical Imaging 2014: Image Perception, Observer Performance, and Technology Assessment

KEYWORDS: Mathematical modeling, Visual process modeling, Data modeling, Visualization, Image processing, Medical imaging, Electronic filtering, Statistical modeling, Performance modeling, Lawrencium

Read Abstract +

Proceedings Article | 28 March 2013 Paper

Statistical properties of a utility measure of observer performance compared to area under the ROC curve

Craig Abbey, Frank Samuelson, Brandon Gallas, John Boone, Loren Niklason

Proc. SPIE. 8673, Medical Imaging 2013: Image Perception, Observer Performance, and Technology Assessment

KEYWORDS: Signal to noise ratio, Statistical analysis, Cancer, Breast cancer, Data modeling, Diagnostics, Receivers, Mammography, Electronic filtering, Digital breast tomosynthesis

Read Abstract +

Proceedings Article | 28 March 2013 Paper

The equivalence of a human observer and an ideal observer in binary diagnostic tasks

Xin He, Frank Samuelson, Brandon Gallas, Berkman Sahiner, Kyle Myers

Proc. SPIE. 8673, Medical Imaging 2013: Image Perception, Observer Performance, and Technology Assessment

KEYWORDS: Data modeling, Image processing, Medical imaging, Image quality, Data processing, Psychology, Electronic filtering, Detection theory, Information operations, Lawrencium

Read Abstract +

Proceedings Article | 19 March 2013 Paper

Uncertainty of Monte Carlo variance estimates: application to the simulation of x-ray imaging detectors

Aldo Badano, Frank Samuelson

Proc. SPIE. 8668, Medical Imaging 2013: Physics of Medical Imaging

KEYWORDS: Statistical analysis, Sensors, Particles, Photons, Computer simulations, Medical imaging, Monte Carlo methods, Image sensors, X-ray imaging, Tolerancing

Read Abstract +

Proceedings Article | 13 March 2009 Paper

Non-localization and localization ROC analyses using clinically based scoring

Sophie Paquerault, Frank Samuelson, Kyle Myers, Robert Smith

Proc. SPIE. 7263, Medical Imaging 2009: Image Perception, Observer Performance, and Technology Assessment

KEYWORDS: Statistical analysis, Computer aided diagnosis and therapy, Imaging systems, Computing systems, Receivers, Medical imaging, Computer aided design, Selenium, Signal detection

Read Abstract +

We are investigating the potential for differences in study conclusions when assessing the estimated impact of a computer-aided detection (CAD) system on readers' performance. The data utilized in this investigation were derived from a multi-reader multi-case observer study involving one hundred mammographic background images to which fixed-size and fixed-intensity Gaussian signals were added, generating a low- and high-intensity signal sets. The study setting allowed CAD assessment in two situations: when CAD sensitivity was 1) superior or 2) lower than the average reader. Seven readers were asked to review each set in the unaided and CAD-aided reading modes, mark and rate their findings. Using this data, we studied the effect on study conclusion of three clinically-based receiver operating characteristic (ROC) scoring definitions. These scoring definitions included both location-specific and non-location-specific rules. The results showed agreement in the estimated impact of CAD on the overall reader performance. In the study setting where CAD sensitivity is superior to the average reader, the mean difference in AUC between the CAD-aided read and unaided read was 0.049 (95%CIs: -0.027; 0.130) for the image scoring definition that is based on non-location-specific rules, and 0.104 (95%CIs: 0.036; 0.174) and 0.090 (95%CIs: 0.031; 0.155) for image scoring definitions that are based on location-specific rules. The increases in AUC were statistically significant for the location-specific scoring definitions. It was further observed that the variance on these estimates was reduced when using the location-specific scoring definitions compared to that using a non-location-specific scoring definition. In the study setting where CAD sensitivity is equivalent or lower than the average reader, the mean differences in AUC are slightly above 0.01 for all image scoring definitions. These increases in AUC were not statistical significant for any of the image scoring definitions. The results on the variance analysis differed from those observed in the other study setting. This investigation furthers our understanding of the relationships between non-localization-specific and localization-specific ROC assessment methodologies and their relevance to clinical practice.

Proceedings Article | 24 March 2008 Paper

Comparing signal-based and case-based methodologies for CAD assessment in a detection task

Sophie Paquerault, Frank Samuelson, Nicholas Petrick, Kyle Myers

Proc. SPIE. 6917, Medical Imaging 2008: Image Perception, Observer Performance, and Technology Assessment

KEYWORDS: Statistical analysis, Computer aided diagnosis and therapy, Computing systems, Receivers, Medical imaging, Computer aided design, Signal analyzers, Signal detection, CAD systems

Read Abstract +

Proceedings Article | 8 March 2007 Paper

Observer evaluation of computer-aided detection: second reader versus concurrent reader scenario

Sophie Paquerault, Darin Wade, Nicholas Petrick, Kyle Myers, Frank Samuelson

Proc. SPIE. 6515, Medical Imaging 2007: Image Perception, Observer Performance, and Technology Assessment

KEYWORDS: Human-machine interfaces, Computer aided diagnosis and therapy, Detection and tracking algorithms, Receivers, Mammography, Computer aided design, Algorithm development, Signal detection, CAD systems

Read Abstract +

Proceedings Article | 6 April 2005 Paper

Bootstrapped MRMC confidence intervals

Frank Samuelson, Robert Wagner

Proc. SPIE. 5749, Medical Imaging 2005: Image Perception, Observer Performance, and Technology Assessment

KEYWORDS: Statistical analysis, Data modeling, Error analysis, Receivers, Computer simulations, Medical imaging, Monte Carlo methods, Physiology, Performance modeling, Statistical inference

Read Abstract +

Proceedings Article | 6 April 2005 Paper

Influence of panel size and expert skill on truth panel performance when combining expert ratings

Nicholas Petrick, Brandon Gallas, Frank Samuelson, Robert Wagner, Kyle Myers

Proc. SPIE. 5749, Medical Imaging 2005: Image Perception, Observer Performance, and Technology Assessment

KEYWORDS: Gold, Computer aided diagnosis and therapy, Imaging systems, Image processing, Lung, Medical imaging, Monte Carlo methods, Statistical modeling, Performance modeling, Binary data

Read Abstract +

Showing 5 of 27 publications

Proceedings Volume Editor (3)

SPIE Conference Volume | 1 May 2020

Medical Imaging 2020: Image Perception, Observer Performance, and Technology Assessment

SPIE Conference Volume | 17 June 2019

Medical Imaging 2019: Image Perception, Observer Performance, and Technology Assessment

SPIE Conference Volume | 11 May 2018

Medical Imaging 2018: Image Perception, Observer Performance, and Technology Assessment

Conference Committee Involvement (7)

Image Perception, Observer Performance, and Technology Assessment

15 February 2021 | Online Only, California, United States

Image Perception, Observer Performance, and Technology Assessment

19 February 2020 | Houston, Texas, United States

Image Perception, Observer Performance, and Technology Assessment

20 February 2019 | San Diego, California, United States

Image Perception, Observer Performance, and Technology Assessment

11 February 2018 | Houston, Texas, United States

Image Perception, Observer Performance, and Technology Assessment Posters

12 February 2017 | Orlando, FL, United States

Showing 5 of 7 Conference Committees

Course Instructor

SC1127: ROC Analysis and Observer Studies to Evaluate Imaging Technology

Short of a clinical trial, observer studies are used to measure the performance of radiologists and to compare different imaging technologies with the aim of understanding how the technology might be used clinically. The goal of this course is to provide guidance on how to design, conduct, and analyze an observer study. We will examine how to plan an observer study, step by step, and then how to implement the plan in detail. The focus will be on pitfalls and common mistakes. We will review basic ROC analysis and commonly used methods for analyzing multireader, multicase (MRMC) ROC observer data. Related software and sample-size estimation will also be discussed and demonstrated using real datasets. The participants will take part in some simulated reader studies using a computer interface, compete for prizes, and depart with a better understanding of reader studies of diagnostic imaging devices.

Expand Description +

View contact details

UPDATE YOUR PROFILE

Is this your profile? Update it now.

Sign into your SPIE.org account

Don’t have a profile and want one?

Create an account on SPIE.org

Keywords/Phrases

Search In:

Publication Years