Solutions for the quality control of metallic surfaces are proposed. Specifically, we study a deflectometric apparatus based on coaxial structured light and the related algorithmic procedure, which is able to detect the faulty surface of a sample captured by a video sequence. First, by considering the metallic surface a dynamic scene illuminated under different light conditions, we develop the descriptor residuals of linear evolution of light (RLEL) that extracts the defectiveness information starting from the movement of the object without explicitly considering the physical characteristics of the light structure. Then, leveraging on RLEL, we present a hybrid learning (HL) technique capable of overcoming the data-driven approach used in classic deep learning (DL). By exploiting a multisteps training process, we combine the model-based descriptor RLEL and a classical data-driven convolutional neural network (CNN) to obtain an unconventional gray-box CNN, which exceeds the performance of popular DL solutions such as 3-D-inception and 3-D-residual DL networks. Remarkably, HL also shows its validity in comparing the performance of the same network structures trained not in a hybrid way, namely without the injection of the model-based information given by RLEL.