Stabilization and registration of full-motion video data using deep convolutional neural networks

Derek J. Walvoord; Doug W. Couwenhoven; Michael A. Bayer

doi:10.1117/12.2305072

27 April 2018 Stabilization and registration of full-motion video data using deep convolutional neural networks

Derek J. Walvoord, Doug W. Couwenhoven, Michael A. Bayer

Proceedings Volume 10646, Signal Processing, Sensor/Information Fusion, and Target Recognition XXVII; 1064612 (2018) https://doi.org/10.1117/12.2305072
Event: SPIE Defense + Security, 2018, Orlando, FL, United States

Abstract

Stabilization and registration are common techniques applied to overhead imagery and full-motion video (FMV) during production to facilitate further exploitation by the end user. Algorithms designed to accom- plish these tasks must accommodate changes in capture geometry, atmospheric effects, and sensor charac- teristics. Moreover, algorithms that rely on a controlled image base (CIB) reference typically require some degree of robustness with respect to differences in imaging modality. While many factors contributing to gross misalignment can be mitigated using available sensor telemetry and rigorous photogrammetric modeling, the subsequent image-based registration task often relies on loose model assumptions and poor generalizations. This work presents a modality-agnostic deep learning approach to automatically stabilize and register overhead FMV data to a reference image such as a CIB. The field of deep learning has received significant attention in recent years with advances in high-performance computing and the availability of widely adopted open source tools for numerical computation using data flow graphs. We leverage recent developments in the use of fully differentiable spatial transformer networks to simultaneously remove coarse geometric differences and fine local misalignments in the registration process. Most importantly, no model is required. A convolutional neural network (ConvNet), complete with a spatial transformer, is trained using pairs of frames of FMV data as the input and corresponding label. Once the mechanism by which the deformable warp is learned, the trained network ingests new data and returns a version of the input image sequence that has been warped to a user-specified reference. The performance of our approach is evaluated using several real FMV data sets.

Conference Presentation

Citation Download Citation

Derek J. Walvoord, Doug W. Couwenhoven, and Michael A. Bayer "Stabilization and registration of full-motion video data using deep convolutional neural networks", Proc. SPIE 10646, Signal Processing, Sensor/Information Fusion, and Target Recognition XXVII, 1064612 (27 April 2018); https://doi.org/10.1117/12.2305072

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE

Includes PDF, HTML & Video, when available