The simulation can evaluate the performance of the optical system and provide guidance for its improvement. At present, most of the research on imaging modeling of hyperspectral system is based on modulation transfer function method. However, this method has its limitations. Therefore, this paper proposes a reliable method for imaging modeling of hyperspectral optical system based on ray tracing: (1) In the spatial dimension, the vignetting coefficients corresponding to the image sampling points are obtained by ZEMAX, and the diffuse spots model of optical system is obtained by tracing the position of the light on the image plane which emitted by the object points after passing through the mesh points on the pupil surface. (2) In the spectral dimension, the spectral lines on the detector are obtained by non-sequential ray tracing, and separate the spectral lines of different wavelengths. The spectral bending coefficient and spectral distortion coefficient can be obtained by least square fitting. In this paper, high spatial and high spectral resolution images served as input sources for imaging simulation. By changing the grating constants, band range, defocusing quantity of the optical system and length of the slit, the changing trends of the output images in radiation intensity, spatial correlation, spectral characteristic curves and vignetting curves are analyzed, and the reliability of the model is verified.