A machine learning (ML) model with physical constraints is introduced to perform diffuse optical tomography (DOT) reconstruction. Here, for the first time, we combine ultrasound-guided DOT with ML to facilitate DOT reconstruction. Our method has two key components: (i) An unsupervised auto-encoder with transfer learning is adopted for clinical data without a ground truth, and (ii) physical constraints are implemented to achieve accurate reconstruction. Both qualitative and quantitative results demonstrate that the accuracy of the proposed method surpasses that of the existing model. In a phantom study, compared with the Born conjugate gradient descent (CGD) reconstruction method, the ML method improves the reconstructed maximum absorption coefficient by 18.3% on high contrast phantom and by 61.3% on low contrast phantom, with improved depth distribution of absorption maps. In a clinical study, better contrast was obtained from a treated breast cancer pre- and post- treatment.