Visual simultaneous localization and mapping (VSLAM) is one of the foremost principal technologies for intelligent robots to implement environment perception. Many research works have focused on proposing comprehensive and integrated systems based on the static environment assumption. However, the elements whose motion status changes frequently, namely short-term dynamic elements, can significantly affect the system performance. Therefore, it is extremely momentous to cope with short-term dynamic elements to make the VSLAM system more adaptable to dynamic scenes. This paper proposes a coarse-to-fine elimination strategy for short-term dynamic elements based on motion status check (MSC) and feature points update (FPU). First, an object detection module is designed to obtain semantic information and screen out the potential short-term dynamic elements. And then an MSC module is proposed to judge the true status of these elements and thus ultimately determine whether to eliminate them. In addition, an FPU module is introduced to update the extracted feature points according to calculating the dynamic region factor to improve the robustness of VSLAM system. Quantitative and qualitative experiments on two challenging public datasets are performed. The results demonstrate that our method effectively eliminates the influence of short-term dynamic elements and outperforms other state-of-the-art methods.