When laser interferometry is applied to measure the form deviation of a helical tooth flank, dense interference fringes are almost inevitable because of the distorted shape of the helical tooth flank. The high-density fringe leads to poor processing accuracy of interferograms and is an important indicator of the interferogram quality. An optical path optimization method based on position errors analysis is proposed to modulate the fringe density. First, some factors that may affect fringe density in the optical system are modeled. Consequently, the incident angle of the measured light is proved to be the main factor affecting the fringe density and is the primary parameter that modulates the interference fringe density. Subsequently, an analytical model is established, and a fringe density modulation method based on the optimized incident angle is proposed for designing the optical path. Finally, simulated and actual experimental results are provided to verify the correctness of the analysis of the main factor affecting the fringe density and feasibility of the modulation method.