Most metrics of medical image quality typically treat all variability components of the background as a Gaussian noise process. This includes task based model observers (non-prewhitening matched filter without and with an eye filter, NPW and NPWE; Hotelling and Channelized Hotelling) as well as Fourier metrics of medical image quality based on the noise power spectra. However, many investigators have observed that unlike many of the models/metrics, physicians often can discount signal-looking structures that are part of the normal anatomic background. This process has been referred to as reading around the background or noise. The purpose of this paper is to develop an experimental framework to systematically study the ability of human observers to read around learned backgrounds and compare their ability to that of an optimal ideal observer which has knowledge of the background. We measured human localization performance of one of twelve targets in the presence of a fixed background consisting of randomly placed Gaussians with random contrasts and sizes, and white noise. Performance was compared to a condition in which the test images contained only white noise but with higher contrast. Human performance was compared to standard model observers that treat the background as a Gaussian noise process (NPW, NPWE and Hotelling), a Fourier-based prewhitening matched filter, and an ideal observer. The Hotelling, NPW, NPWE models as well as the Fourier-based prewhitening matched filter predicted higher performance for the white noise test images than the background plus white noise. In contrast, ideal and human performance was higher for the background plus white noise condition. Furthermore, human performance exceeded that of the NPW, NPWE and Hotelling models and reached an efficiency of 19% relative to the ideal observer. Our results demonstrate that for some types of images human signal localization performance is consistent with use of knowledge about the high order moments of the backgrounds to discount signal-looking structures that belong to the background. In such scenarios model observers and metrics that either ignore the background or treat the background as a Gaussian process (Hotelling, Channelized Hotelling, Task-based SNR) under predict human performance.