Optical encoding technology is one of the most popular technologies with nano-meter degree accuracy in precise displacement metrology field. Basically, two optical gratings overlap in the encoder, resulting in Moiré fringes. An optical sensor records the Moiré fringes signal. When one of the optical gratings moves, the Moiré fringes will vary. The phase of the Moiré fringes is extracted from the signal, and the displacement is obtained from the phase. The fabrication always becomes more difficult and the cost is higher when the measurement accuracy becomes higher for such optical encoder. We have developed a simple, novel encoder with only one optical grating and an optical imaging system. The Moiré fringe curve is obtained when the optical grating overlaps with two complementary digital gratings which are virtually constructed with a CCD or CMOS sensor in a camera. In this technique, the Moiré fringe curve is not a strict sine wave, leading to some difficulty to accurately extract the phase value. This paper compares the performance of four phase extraction algorithms, i.e., Fourier transform, polynomial fitting, Hilbert transform and wavelet transform. The experimental results show that both the measurement accuracy and repeatability of the four algorithms are within 30 nm after calibration. The overall accuracy of the wavelet transform is the best with minimum error of only 5 nm. The processing speed of FFT is the fastest, reaching sub millisecond level.