Annotating image ROIs with text descriptions for multimodal biomedical document retrieval

Daekeun You; Matthew Simpson; Sameer Antani; Dina Demner-Fushman; George R. Thoma

doi:10.1117/12.2005937

4 February 2013 Annotating image ROIs with text descriptions for multimodal biomedical document retrieval

Daekeun You, Matthew Simpson, Sameer Antani, Dina Demner-Fushman, George R. Thoma

Proceedings Volume 8658, Document Recognition and Retrieval XX; 86580D (2013) https://doi.org/10.1117/12.2005937
Event: IS&T/SPIE Electronic Imaging, 2013, Burlingame, California, United States

Abstract

Regions of interest (ROIs) that are pointed to by overlaid markers (arrows, asterisks, etc.) in biomedical images are expected to contain more important and relevant information than other regions for biomedical article indexing and retrieval. We have developed several algorithms that localize and extract the ROIs by recognizing markers on images. Cropped ROIs then need to be annotated with contents describing them best. In most cases accurate textual descriptions of the ROIs can be found from figure captions, and these need to be combined with image ROIs for annotation. The annotated ROIs can then be used to, for example, train classifiers that separate ROIs into known categories (medical concepts), or to build visual ontologies, for indexing and retrieval of biomedical articles. We propose an algorithm that pairs visual and textual ROIs that are extracted from images and figure captions, respectively. This algorithm based on dynamic time warping (DTW) clusters recognized pointers into groups, each of which contains pointers with identical visual properties (shape, size, color, etc.). Then a rule-based matching algorithm finds the best matching group for each textual ROI mention. Our method yields a precision and recall of 96% and 79%, respectively, when ground truth textual ROI data is used.

Citation Download Citation

Daekeun You, Matthew Simpson, Sameer Antani, Dina Demner-Fushman, and George R. Thoma "Annotating image ROIs with text descriptions for multimodal biomedical document retrieval", Proc. SPIE 8658, Document Recognition and Retrieval XX, 86580D (4 February 2013); https://doi.org/10.1117/12.2005937

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE

Includes PDF, HTML & Video, when available