Unveiling the power of unpaired multi-modal data for RGBT tracking

Shen Qing; Wang Yifan; Guo Yu; Mengmeng Yang

doi:10.1117/12.2685082

19 October 2023 Unveiling the power of unpaired multi-modal data for RGBT tracking

Shen Qing, Wang Yifan, Guo Yu, Mengmeng Yang

Proceedings Volume 12709, Fourth International Conference on Artificial Intelligence and Electromechanical Automation (AIEA 2023); 127092N (2023) https://doi.org/10.1117/12.2685082
Event: Fourth International Conference on Artificial Intelligence and Electromechanical Automation (AIEA 2023), 2023, Nanjing, China

Abstract

RGBT tracking receives increasing interests due to its flexible application in all-day and all-weather environments. However, the training of deep RGBT trackers usually relies on large-scale aligned RGBT pairs, which usually require high human labor and time cost. Considering the strong commonality and specificity of multi-modal data, we propose a novel two-stage learning framework to capture modality-shared and modality-specific features using large-scale unpaired RGBT data, and thus achieve state-of-the-art performance in RGBT tracking. In specific, in the first stage, we aim to learn the modality-shared representations and thus design a generic transformer network which only requires mixed modal data for training. In the second stage, we aim to learn the modality specific representations and achieve adaptive back-propagation using unpaired data. To achieve these goals, we design a modality transformer network, in which two modality encoders are used to capture modality-specific features and a modality-adaptive attention module is designed to enforce the interchange of information between different modalities in a separate-gather way. Since the two training stages do not rely on paired or aligned multi-modal data, the power of unpaired multi-modal data is unveiled in the training of deep RGBT tracker. Extensive experiments on three benchmark datasets demonstrate the effectiveness of our method against state-ofthe-art RGBT trackers.

Citation Download Citation

Shen Qing, Wang Yifan, Guo Yu, and Mengmeng Yang "Unveiling the power of unpaired multi-modal data for RGBT tracking", Proc. SPIE 12709, Fourth International Conference on Artificial Intelligence and Electromechanical Automation (AIEA 2023), 127092N (19 October 2023); https://doi.org/10.1117/12.2685082

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE

Includes PDF, HTML & Video, when available

Members: $17.00

Non-members: $21.00 ADD TO CART

PROCEEDINGS
6 PAGES

DOWNLOAD PAPER SAVE TO MY LIBRARY

GET CITATION

RIGHTS & PERMISSIONS

Get copyright permission Get copyright permission on Copyright Marketplace

KEYWORDS

Transformers

Education and training

Design and modelling

Detection and tracking algorithms

Feature extraction

Matrices

Visualization

Show All Keywords

Keywords/Phrases

Search In:

Publication Years