The application of X-ray reflectivity (XRR) and X-ray scattering (XRS) technique for studying thin film surfaces is discussed. A simple method to evaluate thin film mass density accurately from the XRR distribution has been given, named XRR-DE method. According to the measured mass density, optical constant of thin film could be calculated, which is very important for evaluating the surface roughness of thin film from XRS distribution. The first-order perturbation theory (FOPT), one of the XRS methods, is extensively validated for smooth surfaces and can extract power spectral density (PSD) function of the surface roughness directly and uniquely from measured scattering distribution. The conditions of applicability of FOPT have also been discussed. While the three Al2O3 thin films were deposited by Atomic layer deposition (ALD), Au thin film was evaporated by electron beam coating. The density of the Al2O3 thin films evaluated by XRR-DE method is 3.25g/cm3, 3.227 g/cm3, and 3.224 g/cm3. The density difference is less than 1%, and the results are in good agreement with the reported data. However, as an amorphous film, its density is much smaller than that of the sapphire single crystal (3.965g/cm3). The density of Au film evaluated by XRR-DE method is 17.05g/cm3. The scattering indicatrix for different thin films were measured, and one-dimensional PSD functions and effective RMS roughness were calculated by FOPT, which coincide well with the measurements of atomic force microscopy (AFM). The results demonstrated that FOPT is valid for measuring the surface roughness of thin films. Specifically, calculating optical constants from the experimentally measured mass densities by XRR-DE of the studied thin films could improve the measurement accuracy.