Ultrasound (US)-guided diffuse optical tomography (DOT) has demonstrated potential value for screening and treatment monitoring of breast cancers. However, in clinical cases, the chest wall, bad probe-tissue contact, and tissue heterogeneity can create image artifacts, causing misinterpretation of lesion images. In the current work, realistic and flexible threedimensional numerical breast phantoms were generated using the Virtual Imaging Clinical Trials for Regulatory Evaluation (VICTRE) tools developed by U.S. Food and Drug Administration (FDA). By selecting physical attributes and tissue optical properties, the VICTRE breast phantoms were, for the first time, adopted in DOT for in silico studies. Monte Carlo simulations were conducted to generate the forward data. Edge artifacts (hot spots on the edge of the regions of interests) were found on the reconstructed images when there was a mismatch between the lesion-side breast and the contralateral reference-side breast. We propose a fully automated, connected components analysis-based algorithm that can remove these edge artifacts and improve lesion reconstruction.