Video-TransUNet: temporally blended vision transformer for CT VFSS instance segmentation

Chengxi Zeng; Xinyu Yang; Majid Mirmehdi; Alberto M. Gambaruto; Tilo Burghardt

doi:10.1117/12.2679409

7 June 2023 Video-TransUNet: temporally blended vision transformer for CT VFSS instance segmentation

Chengxi Zeng, Xinyu Yang, Majid Mirmehdi, Alberto M. Gambaruto, Tilo Burghardt

Proceedings Volume 12701, Fifteenth International Conference on Machine Vision (ICMV 2022); 127010D (2023) https://doi.org/10.1117/12.2679409
Event: Fifteenth International Conference on Machine Vision (ICMV 2022), 2022, Rome, Italy

Abstract

We propose Video-TransUNet, a deep architecture for instance segmentation in medical CT videos constructed by integrating temporal feature blending into the TransUNet deep learning framework. In particular, our approach amalgamates strong frame representation via a ResNet CNN backbone, multi-frame feature blending via a Temporal Context Module (TCM), non-local attention via a Vision Transformer, and reconstructive capabilities for multiple targets via a UNet-based convolutional-deconvolutional architecture with multiple heads. We show that this new network design can significantly outperform other state-of-the-art systems when tested on the segmentation of bolus and pharynx/larynx in Videofluoroscopic Swallowing Study (VFSS) CT sequences. On our VFSS2022 dataset it achieves a dice coefficient of 0.8796 and an average surface distance of 1.0379 pixels. Note that tracking the pharyngeal bolus accurately is a particularly important application in clinical practice since it constitutes the primary method for diagnostics of swallowing impairment. Our findings suggest that the proposed model can indeed enhance the TransUNet architecture via exploiting temporal information and improving segmentation performance by a significant margin. We publish key source code, network weights, and ground truth annotations for simplified performance reproduction.

Citation Download Citation

Chengxi Zeng, Xinyu Yang, Majid Mirmehdi, Alberto M. Gambaruto, and Tilo Burghardt "Video-TransUNet: temporally blended vision transformer for CT VFSS instance segmentation", Proc. SPIE 12701, Fifteenth International Conference on Machine Vision (ICMV 2022), 127010D (7 June 2023); https://doi.org/10.1117/12.2679409

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE

Includes PDF, HTML & Video, when available

Members: $17.00

Non-members: $21.00 ADD TO CART

PROCEEDINGS
8 PAGES

DOWNLOAD PAPER SAVE TO MY LIBRARY

GET CITATION

RIGHTS & PERMISSIONS

Get copyright permission Get copyright permission on Copyright Marketplace

KEYWORDS

Transformers

Video

Image segmentation

Computed tomography

Ablation

Deep learning

Feature extraction

Keywords/Phrases

Search In:

Publication Years