The standard practice in the histopathology of breast cancers is to examine a hematoxylin and eosin (H&E) stained tissue biopsy under a microscope. The pathologist looks at certain morphological features, visible under the stain, to diagnose whether a tumor is benign or malignant. This determination is made based on qualitative inspection making it subject to investigator bias. Furthermore, since this method requires a microscopic examination by the pathologist it suffers from low throughput. A quantitative, label-free and high throughput method for detection of these morphological features from images of tissue biopsies is, hence, highly desirable as it would assist the pathologist in making a quicker and more accurate diagnosis of cancers. We present here preliminary results showing the potential of using quantitative phase imaging for breast cancer screening and help with differential diagnosis. We generated optical path length maps of unstained breast tissue biopsies using Spatial Light Interference Microscopy (SLIM). As a first step towards diagnosis based on quantitative phase imaging, we carried out a qualitative evaluation of the imaging resolution and contrast of our label-free phase images. These images were shown to two pathologists who marked the tumors present in tissue as either benign or malignant. This diagnosis was then compared against the diagnosis of the two pathologists on H&E stained tissue images and the number of agreements were counted. In our experiment, the agreement between SLIM and H&E based diagnosis was measured to be 88%. Our preliminary results demonstrate the potential and promise of SLIM for a push in the future towards quantitative, label-free and high throughput diagnosis.