Visual object tracking in rain is crucial for various civilian and military applications, such as rainy traffic, flood rescue, boundary surveillance, and all-weather combat. However, most existing trackers are designed for favorable weather conditions and experience a remarkable decline in performance when used in rainy environments. Resulting reasons include image degradation caused by rain streak pollution and contrast reduction, as well as frequent changes in target appearance in rainy video sequences. To solve this problem, we propose a visual object tracking algorithm called SiamDRS based on adaptive rain streak removal and fused dynamic template matching. The proposed algorithm first measures the rain streak interference level in the input image by using a specially designed calculation mechanism. An adaptive rain streak removal module then generates a high-contrast, rain-free image to ensure that the tracker can extract sufficient target features. Subsequently, the fused dynamic template matching mechanism continually updates the tracking template fused with historical target information for matching and locating the target during tracking, thereby reducing the tracking failure rate caused by target appearance changes. In addition, to break through the limit of scarce rainy scene datasets for object tracking testing, we present nine rainy testing datasets created through artificial synthesis and manual collection and annotation of real-world rainy data. Furthermore, we perform a comparative evaluation of the proposed algorithm against existing algorithms to demonstrate its effectiveness and robustness in rainy scenes.