The wide angle lens, like fish eye lens, suffers great optical distortion that causes severe deformation of the real world. A method to correct the strong distortion was presented in this work. Due to the nonlinear distribution of the distortion, linear algorithms are generally not under consideration to establish the math model of distorted-to-ideal images. However, this method employed the calibration pattern that comprised of regular array of dots to divides the full field of view (FOV) to subsections, each subsection is a small FOV, the mapping parameters between the distorted image and ideal image in each small FOV can be calculated by employing the very simple linear polynomial. Thus, applying the determined parameters to their corresponding sub-FOVs, respectively, all the ideal pixel coordinates of the distorted image can be obtained. The method employed linear polynomial characterizes the geometric deformation between the distorted and ideal images directly. Therefore, it contains both of radial distortion and tangential distortion and there is no need of concerning any intrinsic or extrinsic parameters of the optical systems. So, this algorithm reliefs the computational work that employed by conventional radial models and other mathematical models. Experiments performed on off-axis optical systems which exist complicated distortion, such as the head mounted displays (HMDs), had already yielded accurate correcting results. Likewise, in this paper, the experiments refer to the fish-eye lens also verify the effectiveness and flexibility of this method, as well, high correcting accuracy is achieved.