Cloud detection is an important prerequisite for remote sensing image application. Any remote sensing image from which the information of ground object could be obtained will inevitably be preprocessed on cloud occlusion. In the traditional method, the segmentation of cloud and its shadow will be affected by the complex background. In the detection process, due to insufficient information extraction, misjudgment often occurs, and the cloud boundary processing is also very rough. In order to improve the accuracy of cloud and cloud shadow segmentation, we propose a multilevel feature context semantic fusion network. The network takes the residual network as the backbone networkand adopts the structure of encoding and decoding as a whole. In the model, we introduce the multibranch residual context semantic module, the multiscale convolution subchannel attention module, and the feature fusion upsampling module to strengthen the feature extraction, refine the cloud and cloud shadow edge information, and enhance the actual segmentation ability of the model. The experimental results show that the proposed method is more accurate than the previous network in the segmentation of cloud and cloud shadow and has good generalization ability on other datasets, which is of great significance for the study of cloud.