To study new damping augmentation methods for helicopter rotor systems, accurate and reliable nonlinear damping identification techniques are needed. For example, current studies on applications of magnetorheological (MR) dampers for rotor stability augmentation suggest that a strong Coulomb damping characteristic will be manifested as the field applied to the MR fluid is maximized. Therefore, in this work, a single degree of freedom (SDOF) system having either nonlinear Coulomb or quadratic damping is considered. This paper evaluates three analyses for identifying damping from transient test data; an FFT-based moving block analysis, an analysis based on a periodic Fourier series decomposition, and a Hilbert transform based technique. Analytical studies are used to determine the effects of block length, noise, and error in identified modal frequency on the accuracy of the identified damping level. The FFT-based moving block has unacceptable performance for systems with nonlinear damping. These problems were remedied in the Fourier series based analysis and acceptable performance is obtained for nonlinear damping identification from both this technique and the Hilbert transform based method. To more closely simulate a helicopter rotor system test, these techniques were then applied to a signal composed of two closely spaced modes. This data was developed to simulate a response containing the first lag and 1/rev modes. The primary mode of interest (simulated lag mode) had either Coulomb or quadratic damping, and the close mode (1/rev) was either undamped or had a specified viscous damping level. A comprehensive evaluation of the effects of close mode amplitude, frequency, and damping level was performed. A classifier was also developed to identify the dominant damping mechanism in a signal of 'unknown' composition. This classifier is based on the LMS error of a fit of the analytical envelope expression to the experimentally identified envelope signal. In most instances, the classifier identifies the damping mechanism correctly, erring only when the close mode significantly affects the envelope signal.