Is Measurement Noninvariance a Threat to Inferences Drawn from Randomized Control Trials? Evidence From Empirical and Simulation Studies

Abstract

Randomized control trials (RCTs) are considered the gold standard when evaluating the impact of psychological interventions, educational programs, and other treatments on outcomes of interest. However, few studies consider whether forms of measurement bias like noninvariance might impact estimated treatment effects from RCTs. Such bias may be more likely to occur when survey scales are utilized in studies and evaluations in ways not supported by validation evidence, which occurs in practice. This study consists of simulation and empirical studies examining whether measurement noninvariance impacts treatment effects from RCTs. Simulation study results demonstrate that bias in treatment effect estimates is mild when the noninvariance occurs between subgroups (e.g., male and female participants), but can be quite substantial when being assigned to control or treatment induces the noninvariance. Results from the empirical study show that surveys used in two federally funded evaluations of educational programs were noninvariant across student age groups.