The deep convolution neural network (CNN) has been of great interest in hyperspectral image classification recently. Current CNN-based approaches adopt a two- or three-dimensional convolution network to extract spectral–spatial features from local pixel neighborhoods. High computation cost and overfitting problems should be handled carefully for these large-scale networks. A compact one-dimensional (1-D) CNN-based method is proposed to explore the joint spectral–spatial features by employing the pixel coordinates as the spatial information. Furthermore, two kinds of CNN architecture for spectral–spatial feature fusion are compared. To be specific, the shallow fusion model (SF-CNN) concatenates the spatial information to the spectral features before they are fed into the final fully connected layer, whereas the deep fusion model (DF-CNN) combines the spectral and the spatial information in an early stage and extracts the high-level spectral-spatial features by the 1-D convolution layers. The experimental results demonstrate that the DF-CNN method provides competitive classification results to state-of-the-art methods. Moreover, attributed to the concise spatial information and the effective feature fusion structure, the proposed method is economical in terms of computation cost when compared with the current CNN-based spectral-spatial methods.