Empowering retail service robots with empathy is one of the current research hotspots in the field of artificial intelligence. Identifying consumer emotions, understanding the changes in shopping interests, and developing appropriate sales strategies is a challenging task for retail service robots. We investigate the feasibility of using computer vision methods for empowering robots with empathy by examining the correlation between consumer emotion and levels of shopping interest. To this end, we construct the first video database of consumer sentiment changes in a business context and propose a deep learning method that uses multimodal information to infer consumers’ shopping intentions, and conduct preliminary experimental validation on this database. The experimental results show that the proposed method is 7% and 10% more accurate than manual assessment (n  =  40) in identifying consumer emotions and predicting consumer shopping interest levels, respectively. Thus, the proposed method is valid and effective. We anticipate that the results of this study will have considerable implications for human–computer interaction research in service robots.