Doppler asymmetric spatial heterodyne spectroscopy (DASH) has significant advantages in the passive measurement of wind fields. Wind speed measurement accuracy depends heavily on the phase inversion accuracy of DASH interference fringes. However, the errors in the conventional phase inversion algorithm in the DASH technique make it difficult to obtain the phase of DASH interferometric stripes with high accuracy. The error mechanism of the conventional phase inversion algorithm in the DASH technique is dissected from both theoretical derivation and numerical simulation. It is found that the effect of spectrum leakage on the inversion phase error is greater compared with the window function used for isolating spectrum features. Based on the analysis, an optimization measure by improving the recovered spectrum is proposed and verified by numerical simulation. Comparative simulation results with existing phase inversion algorithms show that our optimized processing method has significant advantages, regardless of whether dealing with single or multiple spectral lines. The research results indicate that our proposed optimization measure can effectively improve the phase inversion accuracy, obtain more accurate wind field data, and contribute to the development of the DASH technique in the field of high accuracy passive measurement of global wind fields for the middle and upper atmosphere.