Dr. Manish Sharma Profile

Dr. Manish Sharma

Chief Scientific Officer at Imaging Endpoints

SPIE Involvement:

Author

Area of Expertise:

Medical Imaging , Radiology , Clinical Research , AI / ML , Data Science , Predictive Modeling

Passionate & highly motivated researcher with more than 20 years of rich experience in the field of Clinical Research, Medical Imaging and Data Science, and proven track record for innovation, transformation, and intrapreneurship. Instrumental in launching new services, geographies, or products across each role utilizing diverse experience and expertise for an immediate positive impact on patient health and company growth.

MD with extensive expertise in implementation of existing trial methodologies and adept at collaboration with regulatory agencies discussing implementation of ‘new’ clinical trial designs. Pioneering work in imaging data analytics, metrics, and utilization of big data for reader performance and data driven expansion. Use of AI/ML techniques including deep learning in radiology and imaging through COVID (and other lesion) detection models, multi-variate confusion matrix for reader performance and predictive models using industry first uses cases. Development and deployment of PACS based Central Review System in record time more than a decade back for an immediate impact on company strategy leading to successful NDA submission.

Responsible for designing medical and regulatory strategy including IND, NDA, ANDA, 505(b)(2) submissions to various regulatory agencies including USFDA, EMA, MHRA, DCGI etc. Implemented global Biosimilar development & registration pathway including study design, regulatory advice, endpoint selection & protocol development resulting in some of the earliest biosimilar approvals.

Proceedings Article | 2 April 2024 Presentation + Paper

Leveraging LLMs like ChatGPT for robust quality checks and medical text agreement rationale: enhancing adjudication quality and alignment in BICR for oncology clinical trials

Manish Sharma, Samira Farough, Andre Burkett, Jerome Prasanth, Nabil El-Shafeey, Dominic Zygadlo, Chera Dunn, Ron Korn

Proceedings Volume 12931, 1293103 (2024) https://doi.org/10.1117/12.3009153

KEYWORDS: Error analysis, Clinical trials, Tumors, Oncology, Analytical research, Radiology, Education and training

Read Abstract +

Blinded Independent Central Review (BICR) is recommended by the US FDA for registration of oncology trials as image assessment bias is avoided and no chance of unblinding of patient data. Double read with adjudication is the method used to reduce endpoint assessment variability. In cases of disagreement between the readers, a third reader called an adjudicator, reviews the assessment by the two radiologists and decides which assessment is most accurate. Adjudication Rate (AR) and Adjudicator Agreement Rate (AAR) are the two indicators used to evaluate reviewer performance and overall trial variability and quality. Sentiment Analysis (SA) is based on natural language processing and can tag the data as ‘positive’, ‘negative’ or ‘neutral’ although current technologies can provide a more complex analysis of emotions in the written text. Medical SA can analyze patients’ and doctors’ opinions, sentiments, attitudes, and emotions in the clinical background. Python, the most frequently used programming language for deep learning worldwide and ChatGPT, an AI-based chatbot can be used for assessing adjudicator comment quality based on sentiment analysis. If successful, this analysis can open another novel implementation for Large Language Models (LLMs) or ChatGPT in clinical research and medical imaging. This prospective study involved the review of cases for 100 subjects by board-certified radiologists using the Response Evaluation Criteria in Solid Tumors (RECIST) 1.1 criteria. The study employed a double read with adjudication paradigm in a central imaging review setup. The agreement of adjudication was assessed and compared with the overall response, agreed reader, and medical text. The medical text entered by the adjudicator is usually a free text field that typically lacks standardization and control over its content, which may affect its correlation with reviewer selection for agreement. Although uncommon, errors by the adjudicator can occur due to ambiguous text, mis-clicks, or application delay errors. To analyze the adjudicator’s comments, sentiment analysis was conducted using a Python plug-in with ChatGPT as a large language model. Based on this analysis, the subjects were categorized as either having “Potential Error” or “No Error”. The algorithm supported by ChatGPT was evaluated against a Gold Standard, determined by a board-certified radiologist with over 20 years of experience in the BICR process. A comparison was made to assess accuracy and reproducibility, revealing that only four out of 100 subjects had different outcomes. The sensitivity was calculated as 0.857, specificity as 1.0, and accuracy as 0.96. The remarkable Natural Language Processing (NLP) capabilities of ChatGPT are evident in its ability to classify the sentiment as positive, negative, or neutral based on the free-text adjudicator comments provided during the review process. This classification enables a comparison with the actual assessment, adjudicator agreement, and overall patient outcome, highlighting the impressive performance of ChatGPT in this regard.

Proceedings Article | 2 April 2024 Poster + Paper

Enhancing interpretation assistance by real-time query resolution in BICR of oncology clinical trials by leveraging ChatGPT bots

Manish Sharma, Samira Farough, Andre Burkett, Jerome Prasanth, Nabil El-Shafeey, Dominic Zygadlo, Chera Dunn, Ron Korn

Proceedings Volume 12931, 1293117 (2024) https://doi.org/10.1117/12.3011602

KEYWORDS: Lung cancer, Clinical trials, Education and training, Design, Tumors, Oncology, Pulmonary disorders, Medicine, Brain diseases, Radiology

Read Abstract +

Blinded Independent Central Review (BICR) is pivotal in maintaining unbiased assessment in oncology clinical trials employing various assessment criteria like Response Evaluation Criteria In Solid Tumors (RECIST) in clinical trials framework. This paper emphasizes the potential of Large Language Models (LLMs) such as OpenAI’s GPT-4 and ChatGPT trained on clinical trials’ documents and other reader training materials, to significantly improve interpretation assistance and real-time query resolution. During central review process these documents are easily accessible to readers but sometimes, given that most readers read on multiple ongoing clinical trials on regular basis, it is a daunting task to search details like trial design, endpoints and specific reader rules. Through various pre-trained frameworks using ChatGPT Application Programming Interface (API), an AI-based chatbot can be used for helping readers saving time by providing accurate study design related questions based on uploaded training documents. If successful, this analysis can open another novel implementation for LLMs (or ChatGPT) in clinical research and medical imaging. This prospective study involved the review of study design and protocol available from clinicaltrials.gov database maintained by The National Library of Medicine (NLM) at the National Institutes of Health (NIH). ClinicalTrials.gov is a registry of clinical trials that contains information on clinical studies funded by the NIH, other federal agencies, and private industry. The database includes over 444,000 trials from 221 countries. The NLM works with the US Food and Drug Administration (FDA) to develop and maintain the database. The ChatGPT based Chatbot was trained on clinical trial design data from respective studies to grasp the intricacies of assessment criteria, patient population, inclusion / exclusion criteria and other assessment nuances, as applicable. Fine-tuning with prompt engineering ensured Chatbot to understand the language and context specific to BICR. The resulting models serve as intelligent assistants to provide a user-friendly interface for reviewers. Reviewers can engage with the chatbot in natural language, obtaining clarifications on assessment guidelines, terminology, and complex cases. To train the Chatbot, we searched for lung cancer studies with following specifications: “Completed Studies | Studies with Results | Interventional Studies | Lung Cancer | Phase 3 | Study Protocols | Statistical Analysis Plans (SAPs)” from the clinicaltrials.gov website. Without any bias, we selected the first three studies in search results for our prospective study and seven questions to ask, representative of commonly encountered queries for readers. The algorithm supported by ChatGPT was evaluated against a Gold Standard medical opinion, determined by a board-certified radiologist with over 20 years of experience in the BICR process. The Chatbot provided immediate, contextually accurate insights for all the questions across three studies. The trick questions which did not have the answers, data or references in the provided text were rightly called out by the Chatbot, suggesting the user to check with document or study team. Though sometimes, in addition to referencing the study team or documents, it did provide a clinical practice or general criteria related feedback. Real-time query resolution reduces response time, preventing delays in assessment and decision-making. LLMs can streamline training by offering on-the-spot explanations and references, enhancing reviewer proficiency and efficiency. By acting as interactive chatbots, LLMs have immense potential to improve quality and efficiency by offering contextual guidance and expedited responses, ultimately enhancing decision-making and study efficiency.

Proceedings Article | 3 April 2023 Poster + Paper

Predictive adjudication rate modeling for prioritizing case distribution in BICR for oncology clinical trials

Manish Sharma, Madhuri Madasu, Surabhi Bajpai, Kimberly Fernandes-Thomas, Sree Sudha Kota, Yibin Shao, Rajesh Selvaraj, Rajesh Kaja, Kira Cheng, Joy Luo

Proceedings Volume 12467, 124671E (2023) https://doi.org/10.1117/12.2654344

KEYWORDS: Autoregressive models, Oncology, Modeling, Tumors, Data modeling, Linear regression, Clinical trials, Tumor growth modeling, Medical imaging, Algorithm development

Read Abstract +

Purpose: Blinded independent central review is recommended by the US FDA for registration of oncology trials as it provides bias-free image assessment and avoidance of potential unblinding of patient data. Double read with adjudication is a highly advocated review model used in such trials. Disagreement between readers is natural and inevitable. Radiological disagreement rates or Adjudication Rate (AR) of 30–65% are reported by several papers since 1959 for different oncologic indications. The aim of the study is to develop and use an algorithm to identify reader pair with predicted high AR and investigate if overall study AR can be kept constant or improved by assigning less cases to a reader pair with high AR.
Methods: A retrospective analysis was performed of 285 subjects with 3351 post-baseline timepoints reviewed by board-certified radiologist reviewers using Response Evaluation Criteria in Solid Tumors (RECIST) 1.1 criteria in a BICR set up. The reader adjudication rate was calculated and analyzed throughout the duration of review. The distribution of cases per reader and distribution of cases per reader pair was calculated and overall study AR, and each reader pair AR were calculated.
Data Analysis Methods: Data was prepared and analyzed using linear regression with MS Excel and R programming script (R version 4.1.2 (2021-11-01) -- "Bird Hippie" augmented by RStudio 2022.07.0+548 "Spotted Wakerobin".
Results: Using the data at completion of 50% reads, predicted AR per reader pair was found to correlate well for five out of six reader pairs on the study. This predicted AR can then be used to assign more cases to reader pair with low AR and less cases to reader pair with high AR to keep study level variability low.
Conclusions: Predictive modeling of AR using linear regression can provide an insight into variability of each reader pair, which in turn determines the AR of that reader pair and collectively determines study AR. However, it is still not clear to what level of prioritization in case assignment to specific pairs can be considered acceptable and not artificial.

Proceedings Article | 3 April 2023 Poster + Paper

Utilizing network analysis in blinded independent central review for clinical trials as adjudication agreement dashboard

Manish Sharma, Sree Sudha Kota, Surabhi Bajpai, Kemberly Fernandes-Thomas, Madhuri Madasu, Yibin Shao, Rajesh Kaja, Rajesh Selvaraj, Kira Cheng, Joy Luo

Proceedings Volume 12467, 124671C (2023) https://doi.org/10.1117/12.2653991

KEYWORDS: Network security, Visualization, Statistical analysis, Biological research, Multiple sclerosis, Data analysis, Clinical trials, Visual analytics, Telecommunication networks, Open source software

Read Abstract +

Purpose: Variability in observer performance BICR is common but not well understood and various measures like AR, AAR, RDI help quantify it which leads to multiple complex data points. Network analysis uses mathematically based algorithms to characterize the components of a network of entities and identifying, visualizing, and analysing their relationships. In a network, variables are represented by nodes, the relationships represented by edges between these nodes. The visualization technique involves mapping relationships among entities based on the symmetry or asymmetry of data. Maps from data points generated during double read adjudication study can provide the performance of each reader pair primarily based on AAR.
Methods: Adjudication data from four oncology clinical trials with 2163 subjects, 16937 post-baseline responses was analyzed. Performance metrics included number of cases, adjudication rate, adjudication agreement rate for each read and reader pair. The data were aggregated and prepared for network analysis in Python-a high-level, cross-platform, and open-sourced programming language released under a GPL-compatible license. Python Software Foundation (PSF), a non-profit organization, holds the copyright. Url-https://www.python.org Version 3.9.0
Results: This graphic visualization provides simplistic organization of a complicated data analysis and supports the quality monitoring process of independent reviews. The tool provides a snapshot of the review performance of all the readers in the trial allowing the study team to investigate and intervene in a timely manner with the intent of supporting robust and accurate data analysis.
Conclusions: Network analysis plots for reader performance metrics in BICR provide excellent visual mapping to interpret multiple critical metrics in a single plot which would otherwise require multiple plots and tables. Timely review of these plots during the trial can help demonstrate the effectiveness of interventions as well.

Proceedings Article | 4 April 2022 Paper

Using reader disagreement index as a tool for monitoring impact on read quality due to reader fatigue in central reviewers

Manish Sharma, Madhuri Madasu, Sree Sudha Kota, Surabhi Bajpai, Yibin Shao, Srinivas Pasupuleti, Michael O’Connor

Proceedings Volume 12035, 120350J (2022) https://doi.org/10.1117/12.2613082

KEYWORDS: Clinical trials, Oncology, Visualization, Medical imaging, Medicine, Tumor growth modeling, Statistical analysis, Tumors, Solids, Radiology

Read Abstract +

Purpose: Fatigue may lead to high medical errors by the radiologists. Fatigue is often described as feelings of weakness, lack of energy, and a desire to rest, and is associated with impairments in the ability to function. Visual fatigue has importance in medical imaging as errors (false-negatives) are relatively common. Blinded independent central review (BICR) is a well-used method employed in many oncology registration trials. Ongoing monitoring of radiologist “reviewer” performance is both good clinical trial practice and a requirement by regulatory authorities. We use reader disagreement index (RDI) as a potential tool to identify reader fatigue and compare reader fatigue in reviewers performing single versus multiple types of study.

Methods: A retrospective analysis of reviewers’ RDI in four different clinical trials were performed. Fourteen reviewers’ performance was analyzed with data for 3750 subjects having a total of 15105 timepoints across all clinical trials. These individual trial reviews were conducted by 14 board-certified radiologist reviewers using several established imaging assessment trial criteria. The objective of the study was to establish RDI as an effective tool to analyze if it could be a good surrogate marker for quality impacted by reader fatigue.

Results: The results indicate the RDI can be used as a tool to track reader quality which in turn may be able to predict reader fatigue. In the random pool of readers and studies analyzed, we did not notice any major trend or impact on read quality given that these trials were anyway actively monitored for read volume distribution and quality.

Conclusions: Fatigue may lead to high medical errors by the radiologists. RDI can be used as a good surrogate for read quality to monitor reader fatigue. Based on the results, it can be said that it is better to undertake more cases in a single study than undertake less number of cases in different types of studies to prevent reader fatigue.

Proceedings Article | 4 April 2022 Paper

Satisfaction of search (SOS) error and new lesions identification on imaging in central review for clinical trials

Manish Sharma, Sree Sudha Kota, Madhuri Madasu, Surabhi Bajpai, Yibin Shao, Srinivas Pasupuleti, Michael O’Connor

Proceedings Volume 12035, 1203509 (2022) https://doi.org/10.1117/12.2612995

KEYWORDS: Tumors, Gold, Diagnostics, Error analysis, Clinical trials, Solids, Chest imaging, Therapeutics, Statistical analysis, Radiology

Read Abstract +

Proceedings Article | 16 March 2020 Presentation + Paper

Using “Reader Disagreement Index” as a predictive reviewer performance monitoring tool for timely intervention

Manish Sharma, J. Michael O'Connor, Anitha Singareddy, Madhuri Madasu, Nicholas Enus, Kassel Fotinos-Hoyer, Yibin Shao

Proceedings Volume 11316, 113160A (2020) https://doi.org/10.1117/12.2549980

KEYWORDS: Oncology, Clinical trials, Statistical analysis, Analytical research

Read Abstract +

Purpose: Blinded Independent Central Review (BICR) is a well-accepted method employed in many oncology registration trials. Ongoing monitoring radiologist “reader” performance is both good clinical trial practice and a requirement by regulatory authorities. We continue to use Reader Disagreement Index (RDI) as an important measure in BICR. In this work we studied RDI as an early indicator to identify an outlier reader during the monitoring of reader performance in BICR. Early indication would enable early intervention and thus possibly improve trial outcomes.
Methods: We performed a retrospective analysis of readers’ RDIs in nineteen different clinical trials. Ninety-two reader performances were examined at five intervals in each trial. These individual trial reviews were conducted by forty-three board-certified radiologist readers using several established imaging assessment trial criteria. The objective was to see how well RDI performance above a threshold at progressive monitoring intervals would “flag” a potential overall end-point performance “issue” for that specific reader.
Results: We present results for the prediction of exceeding threshold (one standard deviation above a study mean RDI). Sensitivity, Specificity, Positive Predictive Value (PPV) and Negative Predictive Value (NPV) were determined for the predicted performance outcomes. We explored interpreting multiple “flags” for each trial to improve the aforementioned metrics.
Conclusions: One would expect that a “flag” of RDI exceeding threshold at an early-stage would likely give a useful prediction of end-point reader performance. We refined our methods to use multiple flags which enable statistically improved Specificity and PPV. Improved predictive capability at early stage intervals coupled with persistent monitoring across subsequent intervals will enable trial managers to focus on specific readers. An earlier indication of possible reader performance issues can permit proactive intervention and enhance good trial practices.

Proceedings Article | 4 March 2019 Presentation + Paper

Reader disagreement index: a better measure of overall review quality monitoring in an oncology trial compared to adjudication rate

Manish Sharma, J. Michael O'Connor, Anitha Singareddy

Proceedings Volume 10952, 109520Q (2019) https://doi.org/10.1117/12.2512611

KEYWORDS: Oncology, Clinical trials, Tumors, Analytical research, Image registration, Tumor growth modeling, Data acquisition, Image acquisition, Performance modeling

Read Abstract +

Proceedings Article | 4 March 2019 Presentation + Paper

Development of methods to evaluate probability of reviewer’s assessment bias in Blinded Independent Central Review (BICR) imaging studies

J. Michael O'Connor, Manish Sharma, Anitha Singareddy

Proceedings Volume 10952, 109520P (2019) https://doi.org/10.1117/12.2512603

KEYWORDS: Clinical trials, Statistical analysis, Analytical research, Solids, Tumors, Image processing, Matrices, Data analysis, Clinical research, Computer programming

Read Abstract +

Proceedings Article | 10 March 2017 Presentation + Paper

Development of discordant likelihood metrics for use in evaluation of 'double reviewer' performance in blinded independent central review (BICR) RECIST imaging studies

J. Michael O'Connor, Manish Sharma, Anitha Singareddy, Annette Schmid

Proceedings Volume 10136, 1013609 (2017) https://doi.org/10.1117/12.2255753

KEYWORDS: Diagnostics, Data analysis, Analytical research, Clinical research, Performance modeling, Image analysis, Clinical trials, Tumors, Matrices, Chromium, Statistical analysis, Oncology

Read Abstract +

Showing 5 of 10 publications

Keywords/Phrases

Search In:

Publication Years