Multitrait-Multi Method Analysis of Three Rating Formats

Abstract

A multitrait-multimethod analysis was performed on instructors' ratings obtained from three formats, behaviorally anchored rating scales, graphic rating scale, and mixed standard scale, for two samples of 100 undergraduate students each. The two samples were distinguished on the basis of whether the statements on the mixed-standard scale were behaviorally specific or more generic descriptions of the dimensions. The more specific mixed-standard scale yielded a greater proportion of inconsistent ratings than the less specific one. Also, convergent and discriminant validity were smaller and method variance and unexplained error were greater for the more specific mixed-standard scale. However, a more detailed examination of these effects in terms of selected average correlations indicated that some of these results were not necessarily due to the format. Relative levels of convergent validity were higher and relative levels of discriminant validity were lower than found by Dickinson and Zellinger in 1980 for faculty ratings in a professional school. Over-all, the mixed-standard scale engendered as much convergent and discriminant validity as did the other two rating formats.