Random Responding as a Threat to the Validity of Effect Size Estimates in Correlational Research

Abstract

Random responding to psychological inventories is a long-standing concern among clinical practitioners and researchers interested in interpreting idiographic data, but it is typically viewed as having only a minor impact on the statistical inferences drawn from nomothetic data. This article explores the impact of random responding on the size and direction of correlations observed between multi-item inventory scores. Random responses to individual items result in nonrandomly distributed inventory-level scores. Therefore, even low base rates of random responding can significantly affect the statistical inferences made from inventory-level data. Study 1 uses simulations to show that even low base rates of random responding can significantly affect observed correlations, especially when the inventories in question assess low or high base rate phenomena. Study 2 uses archival data to illustrate the moderating effect of random responding on observed correlations in two samples.