Abstract
A Monte Carlo study was conducted to evaluate six models commonly used to evaluate change. The results revealed specific problems with each. Analysis of covariance and analysis of variance of residualized gain scores appeared to substantially and consistently overestimate the change effects. Multiple factor analysis of variance models utilizing pretest and post-test scores yield invalidly low F ratios. The analysis of variance of difference scores and the multiple factor analysis of variance using repeated measures were the only models which adequately controlled for pre-treatment differences, however, they appeared to be robust only when the error level was 50% or more. These results indicated that one of the persistent flaws of models traditionally used to evaluate change is that the generated statistics are confounded with the amount of error variance in the data.
With this insight, a modification of the analysis of variance model was developed. A second Monte Carlo study was conducted to further evaluate the traditionally used models and the modified models. The results confirmed the findings of the first study and suggested that the modified model may be an adequate model with which change phenomena can be evaluated.
Get full access to this article
View all access options for this article.