For an optical tracking system, due to the limited sampling rate and time delays in the image-based tracking loop, control bandwidth is usually restricted. An optimized controller based on Youla–Kucera parameterization is proposed to improve both vibration rejection and target tracking performance. This optimized controller combined the position error signal and the control output to compensate for the original feedback loop. The performance in low frequency was enhanced using an optimized Q₃₁-filter, although the close-loop bandwidth of the tip-tilt mirror was not improved. The stability and robustness of this control configuration were analyzed by gain margin and phase margin of the open-loop transfer function. Because of relying on a simple and low-frequency model of the tip-tilt mirror control system, this new controller did not lead to a compromise between vibration rejection and noise propagation in the loop. Simulations and experiments were used to testify to the effectiveness of the new controller.