We recently completed a reader study to compare optical and digital pathology (DP) for the assessment of two tissue-based biomarkers with immunohistochemistry. Eight pathologists reviewed 50 breast cancer whole slides (25 stained with HER2 and 25 with Ki-67) and 2 TMAs (1 stained with HER2, 1 with Ki-67, 97 cores each), using digital and optical microscopy. All reviews took place in a single office, using the same microscope, same computer/color calibrated monitor combination, and the same ambient light, in order to eliminate sources of variability due to these parameters. Agreement analysis was performed using the Kendall’s tau-b metric and percent correct agreement. Results showed relatively high overall inter-observer and inter-modality agreement. However, significant uncertainty was observed for the whole slide evaluation with 95% confidence intervals (CI) in the order of 0.30 for the Kendall’s tau-b metric, despite taking care to reduce sources of uncertainty. For the better-sampled TMAs, CIs were in the order of 0.15. It can be deduced that the sample size of 25 slides for each biomarker was not adequate even though it is in line with recent guidelines for the validation of DP from the College of American Pathologists (20 slides for immunohistochemistry without specifying task). Significant uncertainty was observed in our study, despite controlling for several variables. Further work is needed to identify sources of uncertainty for observer tasks in DP, and to account for it in study designs to assess DP.