Content-based image retrieval (CBIR) which provide an effective and advanced means to manage and utilize image database, is one of the most active research topic of image comprehension, image database and computer vision. CBIR system is one of the most important offers of services and applications to Spatial Data Infrastructure (SDI), and SDI will rely on CBIR more and more. Nevertheless, at present, the main research result of CBIR concentrates on small-sized simple image's retrieval, such as fingerprints' and trademarks' retrieval. Limited by remote sensing images' property, such as various dimension, huge data size and plenty of information, research about remote sensing image CBIR rarely reported and there isn't a mature remote sensing image CBIR system now. This paper analyzes the challenges and difficulties that remote sensing image CBIR system facing, including the technique of remote sensing image's feature extraction and representation, the organizing and management of remote sensing image data in CBIR system, the utilizing of remote sensing image's topological relationship in CBIR system, high-dimensional vector indexing technique and self-learning method. Then a three-layer architecture is constructed for remote sensing image CBIR system. Finally, we predict the tendency and trend of the remote sensing image CBIR, including the feature extraction and representation method will rely more and more on image's semantic information, the research of compressed image's retrieval will be more extractive, and unified model of content-based remote sensing image retrieval system will be constructed.