IterationNet: accelerating model incremental update for small datasets based on knowledge distillation

Bing Hao; Zhiqiang Guo; Junlin Zhang

doi:10.1117/12.2577464

10 November 2020 IterationNet: accelerating model incremental update for small datasets based on knowledge distillation

Bing Hao, Zhiqiang Guo, Junlin Zhang

Proceedings Volume 11584, 2020 International Conference on Image, Video Processing and Artificial Intelligence; 1158403 (2020) https://doi.org/10.1117/12.2577464
Event: Third International Conference on Image, Video Processing and Artificial Intelligence, 2020, Shanghai, China

Abstract

Model iteration with new data is important to improve generalization of the model. In general, there are two methods to deal with model incremental update: (a) retraining the model with merging all data together and (b) training a separate model with the new data based on transfer learning. However, the above methods are either time-consuming or suffering from over-fitting problems when the sample size of new data is small. To address this practical issue, we propose a new iteration model, the IterationNet, which can learn features of new data while maintain the performance on the old data. It is a new model iteration method based on knowledge distillation which adds consistency network and truncate L1 regularization. In classifying fake avatar images of Weibo users, IterationNet extremely decreased training time from 8 hours to 5 minutes while the accuracy rate is only reduced from 96% to 91% comparing to training with merged data. Compared with transfer learning, IterationNet showed increased accuracy rate by 21 percent with similar training time.

Citation Download Citation

Bing Hao, Zhiqiang Guo, and Junlin Zhang "IterationNet: accelerating model incremental update for small datasets based on knowledge distillation", Proc. SPIE 11584, 2020 International Conference on Image, Video Processing and Artificial Intelligence, 1158403 (10 November 2020); https://doi.org/10.1117/12.2577464

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE

Includes PDF, HTML & Video, when available

Members: $17.00

Non-members: $21.00 ADD TO CART

PROCEEDINGS
8 PAGES

DOWNLOAD PAPER SAVE TO MY LIBRARY

GET CITATION

RIGHTS & PERMISSIONS

Get copyright permission Get copyright permission on Copyright Marketplace

KEYWORDS

Data modeling

Statistical modeling

Image filtering

Image processing

Mathematical modeling

Network architectures

Structural design

Show All Keywords

Keywords/Phrases

Search In:

Publication Years