Due to the limited computing power of unmanned aerial vehicles (UAVs) and the problems of missed detection and wrong detection of small objects, the current object detection algorithm cannot achieve real-time and high-precision detection. To solve these problems, we propose a vehicle detection network Shuffle CarNet for UAVs aerial images, which is composed of a feature extraction network, a feature fusion network, and a three-scale prediction network. First, according to the limited hardware resources of embedded devices, a lightweight feature extraction network Light CarNet is proposed by fusing the attention mechanism. Second, a four-scale feature bidirectional weighted fusion module is designed. According to the characteristics of the object scale, multilevel feature map bidirectional weighted fusion is selected for target classification and bounding box regression on three scales. Finally, Car-non-maximum suppression is used to reduce false detection and missed detection. Experiments show that compared with other algorithms on the VisDrone-2019 dataset, the proposed method improves the mean average precision by 1.14%, achieves a precision of 82.96%, and can meet the needs of real-time vehicle detection. The superiority of this method is proved by many comparative experiments.