The draft is a measurement of the vertical distance between the waterline and the bottom of the ship hull. The displacement tonnage of a ship then can be calculated by the observed draft. The current draft survey is done by surveyors, which is subject to human errors. We propose to use computer vision with deep learning for draft reading from images. First, mask R-CNN is used to segment the region of interest—draft marks and water—from images. Then UNet is used to refine the waterline detection. The detection of marks is based on the computer vision methods and the content of marks is recognized by ResNet. Finally, we can infer the draft of a ship based on the extracted visual information. Experimental results on a realistic dataset have shown that the proposed method can perform the task of draft reading on a par with humans.