Infrared and visible image fusion aims to fuse infrared targets and visible details in a composite scene. Recently, many fusion methods have been proposed, but most cannot balance characteristics of infrared and visible images well. We propose an infrared and visible image fusion method. First, to enhance the quality of the infrared and visible images to assist the subsequent prefusion and decomposition, a preprocessing method based on singular value decomposition combined with high dynamic range compression and guided filter (GF) is proposed. Then the preprocessing results are fused to produce the prefusion result by multiscale structural image decomposition-based fusion method. Next, the prefusion result is decomposed by the multilevel image decomposition method based on latent low-rank representation (MDLatLRR). Then the source images are decomposed by MDLatLRR. For the base layer, we combine the visual saliency map with GF to obtain the final fused base layer. For the detail layers, we propose a nonlinear function to highlight the infrared target and use weighted least square to optimize the details. We also adopt a structural similarity-based weight map via L₂-norm to improve the fusion performance. The experimental results demonstrate that the proposed method outperforms some state-of-the-art methods both qualitatively and quantitatively.