In mode-division multiplexing (MDM) systems, transmission quality is restricted by differential mode delay (DMD). We have proposed and developed a spatial-mode exchange technique using volume holograms (VHET) as a leading technology to reduce DMD. VHET can equilibrate the transmission time of each spatial mode which has different transmission speeds in a few mode fiber using a volume hologram. This technology enables low signal distortion and high spectral efficiency, which are indispensable to the long-haul transmission for the MDM system. However, in this technology, modal cross-talks (MXTs), which are caused by inter-page cross-talks in the volume hologram, seriously degrade the performance of VHET. In this study, we proposed a method of combining a volume hologram and a random optical diffuser to reduce the MXTs. In our method, the intensity distribution of the input spatial mode is diffused uniformly by a random optical diffuser. The high exchange performance will be attained because the non-targeted holograms included in the multiplexed holograms do not affect most of the spatial mode. Moreover, our method can be applied to the communication wavelength bands using the dual wavelength method. We confirmed the basic operation of the proposed scheme using a linearly polarized mode group comprising LP0,1, LP1,1, and LP2,1. Compared with the conventional VHET, the simulation results show that the maximum MXTs are significantly suppressed from 0.4 to -15.8 dB.