Defect detection is critical for product quality assessment. Currently, machine vision technology has gradually replaced inefficient manual inspections. Due to the intricate textures and assorted defects on the product surface, conventional defect detection technology still requires ameliorating. We propose an improved deep learning network for defect detection of magnetic sheets, which contains the major contributions at two aspects. (a) Image preprocessing is utilized to enhance the defect features of the dataset. (b) With the superiorities of the core structures of various efficient convolutional neural networks, several perception modules are formulated for multiscale feature extraction and are stacked to construct our inspection network. The parameters are effectively reduced while pursuing the detection accuracy, which is more in line with industrial computing requirements. Experiments show that our defect detection on magnetic sheets has achieved great results, and the computational resources are saved. Moreover, a case extended to the defect detection of hot rolled steel indicates that the proposed network is scalable and has great application potential.