Accurate retrieval of top of the atmosphere (TOA) radiance is a sustained research work in ocean color remote sensing. Spectral shapes of sensor input and sensor spectral response (SRF) function may significantly affect the achievable accuracy. A model is presented here to quantify the effects of spectral shapes. We also proposed an innovative method to retrieve the TOA spectral radiance that accounts for spectral shapes. Ten-fold improvement is demonstrated in retrieval accuracy using the proposed method on multiple data sets. Input spectral shape variations are simulated using spectra of TOA radiance, path radiance, exoatmospheric solar irradiance, and integrating sphere. SRF changes are simulated using SRFs of Ocean Color Monitor-2 and Sea-Viewing Wide Field-of-View Sensor. System spectral shape factor, a new parameter, is found to capture the spectral changes and helps quantitative assessment of the inaccuracy. The proposed method will be highly beneficial in deriving accurate geophysical products from the existing and upcoming ocean color remote sensing instruments.