Learning multi-level representations for image emotion recognition in the deep convolutional network

Hao Zhang; Yanan Liu; Dan Xu; Kangjian He; Guoqing Peng; Yingying Yue; Ruhan Liu

doi:10.1117/12.2623414

16 February 2022 Learning multi-level representations for image emotion recognition in the deep convolutional network

Hao Zhang, Yanan Liu, Dan Xu, Kangjian He, Guoqing Peng, Yingying Yue, Ruhan Liu

Proceedings Volume 12083, Thirteenth International Conference on Graphics and Image Processing (ICGIP 2021); 1208321 (2022) https://doi.org/10.1117/12.2623414
Event: Thirteenth International Conference on Graphics and Image Processing (ICGIP 2021), 2021, Kunming, China

Abstract

Images can convey rich semantic information and arouse strong emotions in the viewer. With the growing trend of online images and videos to express opinions, evaluating emotions from visual content has attracted considerable attention. Image emotion recognition aims to classify the emotions conveyed by images automatically. The existing image sentiment classification studies using manual features or deep models mainly focus on low-level visual features or high-level semantic representation without considering all factors. In this paper, we adopt visualization to study the working principle of deep representation in emotion recognition. Research shows that the deep model mainly relies on deep semantic information while ignoring the features of shallow visual details, which are essential to evoke emotions. To form a more discriminative representation for emotion recognition, we propose a multi-level representation model with side branches that learns and integrates different depth representations of the backbone for sentiment analysis. Unlike the hierarchy CNN structure, our model provides a description from the deep semantic representation to shallow visual representation. Additionally, several feature fusion approaches are analyzed and discussed to optimize the deep model. Extensive experiments on several image emotion recognition datasets show that our model outperforms various existing methods.

Citation Download Citation

Hao Zhang, Yanan Liu, Dan Xu, Kangjian He, Guoqing Peng, Yingying Yue, and Ruhan Liu "Learning multi-level representations for image emotion recognition in the deep convolutional network", Proc. SPIE 12083, Thirteenth International Conference on Graphics and Image Processing (ICGIP 2021), 1208321 (16 February 2022); https://doi.org/10.1117/12.2623414

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE

Includes PDF, HTML & Video, when available