In order to improve the accuracy of brain magnetic resonance imaging (MRI) classification and to reduce classification time, this paper proposes a brain MRI image classification algorithm combined with transfer learning and support vector machine (SVM). Firstly, the three deep convolutional neural networks pre-trained on the ImageNet database, AlexNet, VGG16 and GoogleNet, are used as feature extractors to extract the features of brain MRI images in the Harvard Medical School Website database. The feature extraction process does not require to fine-tune the transferred networks. Then, the features extracted in each convolutional neural network are combined to form a feature vector of each brain MRI image and it is input to the SVM for classification. Finally the SVM classified the brain MRI images into healthy, Alzheimer's disease, and stroke. The experimental results show that the classification accuracy can achieve 100%, and the classification time is only 26 seconds. Compared with the brain MRI image classification algorithm proposed by the literature, the accuracy of the proposed method is increased by 8.67%, 1.09%, 0.55%, respectively. The proposed method can provide effective help for the diagnosis of brain diseases.