Multimodal object detection using unsupervised transfer learning and adaptation techniques

Rachael Abbott; Neil Robertson; Jesus Martinez del Rincon; Barry Connor

doi:10.1117/12.2532794

19 September 2019 Multimodal object detection using unsupervised transfer learning and adaptation techniques

Rachael Abbott, Neil Robertson, Jesus Martinez del Rincon, Barry Connor

Proceedings Volume 11169, Artificial Intelligence and Machine Learning in Defense Applications; 1116907 (2019) https://doi.org/10.1117/12.2532794
Event: SPIE Security + Defence, 2019, Strasbourg, France

Abstract

Deep neural networks achieve state-of-the-art performance on object detection tasks with RGB data. However, there are many advantages of detection using multi-modal imagery for defence and security operations. For example, the IR modality offers persistent surveillance and is essential in poor lighting conditions and 24hr operation. It is, therefore, crucial to create an object detection system which can use IR imagery. Collecting and labelling large volumes of thermal imagery is incredibly expensive and time-consuming. Consequently, we propose to mobilise labelled RGB data to achieve detection in the IR modality. In this paper, we present a method for multi-modal object detection using unsupervised transfer learning and adaptation techniques. We train faster RCNN on RGB imagery and test with a thermal imager. The images contain object classes; people and land vehicles and represent real-life scenes which include clutter and occlusions. We improve the baseline F1-score by up to 20% through training with an additional loss function, which reduces the difference between RGB and IR feature maps. This work shows that unsupervised modality adaptation is possible, and we have the opportunity to maximise the use of labelled RGB imagery for detection in multiple modalities. The novelty of this work includes; the use of the IR imagery, modality adaption from RGB to IR for object detection and the ability to use real-life imagery in uncontrolled environments. The practical impact of this work to the defence and security community is an increase in performance and the saving of time and money in data collection and annotation.

Conference Presentation

Citation Download Citation

Rachael Abbott, Neil Robertson, Jesus Martinez del Rincon, and Barry Connor "Multimodal object detection using unsupervised transfer learning and adaptation techniques", Proc. SPIE 11169, Artificial Intelligence and Machine Learning in Defense Applications, 1116907 (19 September 2019); https://doi.org/10.1117/12.2532794

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE

Includes PDF, HTML & Video, when available