Remote sensing image classification has important research significance and application value in image information extraction, ground object detection and identification, and is widely used in military reconnaissance, disaster relief, crop recognition and yield estimation and other military and civil fields. In the past few decades, scholars have done a lot of research on remote sensing image classification, and put forward multiple classification methods, which are mainly divided into supervised classification and unsupervised classification. However, with the increasement of remote sensing image resolution, traditional classification algorithms can not meet the needs for high-precision classification, and also unable to solve “the different objects with same spectrum” and “the same object with different spectrum” problem. In recent years, machine learning has made breakthroughs in image classification research. As a branch of machine learning, deep learning stands out among many machine algorithms for its applicability of learning models and accuracy of classification results. Therefore, more and more scholars apply deep learning to remote sensing image classification. In this paper, the application of deep learning in remote sensing image classification is analyzed and prospected. Firstly, the basic process of classification is summarized, and the common data sets are introduced. Secondly, frequently-used models and open source tools in application has been introduced, with the analysis of the latest application progress in rapidly developing deep learning methods. Finally, the difficulties and challenges existing in the application is discussed and the trend is prospected.