7 June 2021 Space-time super-resolution with motion-perceptive deformable alignment
Zhuojun Cai, Xiang Tian, Ze Chen, Yaowu Chen
Author Affiliations +
Abstract

We proposed a motion-perceptive deformable alignment network that introduces a pre-computed optical flow to improve the motion perception of the deformable alignment process. The pre-computed flows share the burden of the learned offsets for motion estimation while the flexibility of the deformable convolutional network is maintained. In addition, a motion-adaptive pyramid structure in which the features are aligned in multi-scale levels and then merged based on the motion strength among input frames is proposed. With the above structures, an innovative space-time super-resolution (STSR) network is constructed with an improved motion compensation ability. STSR aims to restore a high-resolution and high-frame-rate sequence from its corresponding low-resolution and low-frame-rate version. The proposed STSR network is trained with a Vimeo-90K dataset, and tests are conducted on the Vimeo-90K, densely-annotated video segmentation (DAVIS), and realistic and diverse scenes (REDS) datasets. Performance is evaluated using the peak signal-to-noise ratio and structural similarity index of the entire restored frame on the Y channel. Extensive experiments demonstrate the superiority of the proposed network among both the one- and two-stage STSR methods, its improved alignment ability, and its significantly improved interpolated frame synthesis.

© 2021 SPIE and IS&T 1017-9909/2021/$28.00© 2021 SPIE and IS&T
Zhuojun Cai, Xiang Tian, Ze Chen, and Yaowu Chen "Space-time super-resolution with motion-perceptive deformable alignment," Journal of Electronic Imaging 30(3), 033020 (7 June 2021). https://doi.org/10.1117/1.JEI.30.3.033020
Received: 27 January 2021; Accepted: 19 May 2021; Published: 7 June 2021
Lens.org Logo
CITATIONS
Cited by 1 scholarly publication.
Advertisement
Advertisement
RIGHTS & PERMISSIONS
Get copyright permission  Get copyright permission on Copyright Marketplace
KEYWORDS
Lawrencium

Super resolution

Optical flow

Video

Motion models

Motion estimation

Visualization

RELATED CONTENT


Back to Top