With the development of synthetic aperture radar (SAR) technology, more SAR datasets with high resolution and large scale have been obtained. Research using SAR images to detect and monitor marine targets has become one of the most important marine applications. In recent years, deep learning has been widely applied to target detection. However, it was difficult to use deep learning to train an SAR ship detection model in complex scenes. To resolve this problem, an SAR ship detection method combining YOLOv4 and the receptive field block (CY-RFB) was proposed in this paper. Extensive experimental results on the SAR-Ship-Dataset and SSDD datasets demonstrated that the proposed method had achieved supreme detection performance compared to the state-of-the-art ship detection methods in complex scenes, whether they were in offshore or inshore scenes of SAR images.