In order to enhance the robustness of IR fast small target tracking, a novel mean shift tracking algorithm using improved similarity measure of is proposed. Firstly, problems of local background interfering in original mean shift algorithm for tracking fast motion small target is analyzed, and the reasons is located in the Bhattacharyya coefficient similarity measure expression for all gray weights of components are same, which cannot reflect the advantage contribution of the small target’s gray component in the process of measuring similarity, causing serious interference of the background in the tracking process, leaving the algorithm converging easily. Therefore, to solve this problem, the improvements Bhattacharyya coefficient similarity measure with the local background information fused is proposed. Then, shift vector is deduced in the framework of mean shift by regarding Bhattacharyya coefficients as the similarity measure.In shifting process, the robustness of the small target tracking is improved effectively according to target gray level of large membership degree with high shift weight, and vice versa with low shift weight, which the background interference is suppressed to some extent. In sake of verifying the performance of the proposed algorithm, the classical mean shift algorithm and the algorithm of this paper is used in the target tracking simulation experiment, as well as the infrared image sequences containing the small fast targets of uncooled infrared camera is used. Finally the experimental result indicates that the performance of tracking the small fast target in IR images is robust.