Multimodal aerial view object classification with disjoint unimodal feature extraction and fully connected-layer fusion

Saurav Singh; Manish Sharma; Jamison Heard; Jesse D. Lew; Eli Saber; Panos P. Markopoulos

doi:10.1117/12.2664041

13 June 2023 Multimodal aerial view object classification with disjoint unimodal feature extraction and fully connected-layer fusion

Saurav Singh, Manish Sharma, Jamison Heard, Jesse D. Lew, Eli Saber, Panos P. Markopoulos

Author Affiliations +

Proceedings Volume 12522, Big Data V: Learning, Analytics, and Applications ; 1252206 (2023) https://doi.org/10.1117/12.2664041
Event: SPIE Defense + Commercial Sensing, 2023, Orlando, Florida, United States

Abstract

Fusion of multimodal data can offer enhanced machine learning. One of the most common fusion approaches in deep learning is end-to-end training of a neural network on all available modalities. However, paired multimodal data from all the modalities is required to train such a network. Collecting paired data from multiple modalities can be challenging and expensive due to the requirement of specialized equipment, atmospheric conditions, limitation of individual modalities to probe a scene, data integration from modalities with different spatial and spectral resolutions, and annotation challenges for obtaining ground truth. A two-phase multi-stream fusion approach is presented in this work to counteract this issue. First, we train the unimodal streams in parallel with their own decision layers, loss, and hyper-parameters. Then, we discard the individual decision layers, concatenate the last feature map of all unimodal streams, and jointly train a common multimodal decision layer. We tested the proposed approach on the NTIRE-21 dataset. Our experiments corroborate that in multiple cases, the proposed method can outperform the alternatives.

Conference Presentation

Citation Download Citation

Saurav Singh, Manish Sharma, Jamison Heard, Jesse D. Lew, Eli Saber, and Panos P. Markopoulos "Multimodal aerial view object classification with disjoint unimodal feature extraction and fully connected-layer fusion", Proc. SPIE 12522, Big Data V: Learning, Analytics, and Applications , 1252206 (13 June 2023); https://doi.org/10.1117/12.2664041

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE

Includes PDF, HTML & Video, when available