Sage Journals: Discover world-class research

Abstract

This paper investigated consequences of measurement error in the pretest on the estimate of the treatment effect in a pretest–posttest design with the analysis of covariance (ANCOVA) model, focusing on both the direction and magnitude of its bias. Some prior studies have examined the magnitude of the bias due to measurement error and suggested ways to correct it. However, none of them clarified how the direction of bias is affected by measurement error. This study analytically derived a formula for the asymptotic bias for the treatment effect. The derived formula is a function of the reliability of the pretest, the standardized population group mean difference for the pretest, and the correlation between pretest and posttest true scores. It revealed a concerning consequence of ignoring measurement errors in pretest scores: treatment effects could be overestimated or underestimated, and positive treatment effects can be estimated as negative effects in certain conditions. A simulation study was also conducted to verify the derived bias formula.

Keywords

analysis of covariance structural equation model measurement error in pretest bias in treatment effect

Get full access to this article

View all access options for this article.

References

Battauz

Bellio

Gori

(2011). Covariate measurement error adjustment for multilevel models with application to educational data. Journal of Educational and Behavioral Statistics, 36(3), 283–306.

Battistin

Chesher

(2014). Treatment effect estimation with covariate measurement error. Journal of Econometrics, 178(2), 707–715.

Campbell

D. T.

Boruch

R. F.

(1975). Making the case for randomized assignment to treatments by considering the alternatives: Six ways in which quasi-experimental evaluations in compensatory education tend to underestimate effects. In Bennett

C. A.

Lumsdaine

A. A.

(Eds.), Evaluation and experiment: Some critical issues in assessing social programs (pp. 195–296). Academic Press.

Campbell

D. T.

Erlebacher

(1970). How regression artifacts in quasi-experimental evaluations can mistakenly make compensatory education look harmful. In Hellumuth

(Ed.), The disadvantaged child: Vol. 3, compensatory education: A national debate (pp. 185–210). Brunner/Mazel

New York

Carroll

R. J.

Ruppert

Stefanski

L. A.

Crainiceanu

C. M.

(2006). Measurement error in nonlinear models: A modern perspective (2nd ed.). Chapman and Hall/CRC.

Cicirelli

V. G.

Evans

J. W.

Schiller

J. S.

(1969). The impact of Head Start: An evaluation of the effects of Head Start on children’s cognitive and affective development (Vols. 1–2). Ohio University and Westinghouse Learning Corporation.

Cohen

(1988). Statistical power analysis for the behavioral sciences. Lawrence Erlbaum.

Cohen

West

S. G.

Aiken

L. S.

(2003). Applied multiple regression/correlation analysis for the behavioral sciences (3rd ed.). Lawrence Erlbaum.

Cook

J.R.

Stefanski

L.A.

(1994). Simulation-Extrapolation Estimation in Parametric Measurement Error Models. Journal of the American Statistical Association, 89, 1314–1328.

10.

Crocker

Algina

(1986). Introduction to classical and modern test theory. Harcourt Brace Jovanovich.

11.

Culpepper

S. A.

Aguinis

(2011). Using analysis of covariance (ANCOVA) with fallible covariates. Psychological Methods, 16(2), 166–178.

12.

Darlington

R. B.

(1990). Regression and linear models. McGraw-Hill.

13.

Devanarayan

Stefanski

L. A.

(2002). Empirical simulation extrapolation for measurement error models with replicate measurements. Statistics & Probability Letters, 59(3), 219–225.

14.

Egbewale

B. E.

Lewis

Sim

(2014). Bias, precision and statistical power of analysis of covariance in the analysis of randomized trials with baseline imbalance: A simulation study. BMC Medical Research Methodology, 14(1), 1–12.

15.

Fuller

W. A.

(1987). Measurement error models. Wiley.

16.

Greene

W. H.

(2008). Econometric analysis (6th ed.). Pearson.

17.

Hong

Aaby

D. A.

Siddique

Stuart

E. A.

(2019). Propensity score–based estimators with multiple error-prone covariates. American Journal of Epidemiology, 188(1), 222–230.

18.

Kumenta

(1986). Elements of econometrics (2nd ed.). University of Michigan Press.

19.

Little

T. D.

(2013). Longitudinal structural equation modeling. Guilford press.

20.

Lockwood

J. R.

McCaffrey

D. F.

(2014). Correcting for test score measurement error in ANCOVA models for estimating treatment effects. Journal of Educational and Behavioral Statistics, 39(1), 22–52.

21.

Marsh

H. W.

(1998). Simulation study of nonequivalent group-matching and regression-discontinuity designs: Evaluations of gifted and talented programs. The Journal of Experimental Education, 66(2), 163–192.

22.

Nunnaly

J. C.

Bernstein

I. H.

(1994). Psychometric theory (3rd ed.). McGraw-Hill.

23.

Office of Head Start. (2020). About the office of head start. U.S. Department of Health & Human Services. https://www.acf.hhs.gov/ohs/about

24.

Rabe-Hesketh

Pickles

Skrondal

(2003). Correcting for covariate measurement error in logistic regression using nonparametric maximum likelihood estimation. Statistical Modelling, 3(3), 215–232.

25.

Rabe-Hesketh

Skrondal

Pickles

(2003). Maximum likelihood estimation of generalized linear models with covariate measurement error. The Stata Journal, 3(4), 386–411.

26.

Schafer

D. W.

Purdy

K. G.

(1996). Likelihood analysis for errors-in-variables regression with replicate measurements. Biometrika, 83(4), 813–824.

27.

Sengewald

Steiner

P. M.

Pohl

(2019). When does measurement error in covariates impact causal effect estimates? Analytic derivations of different scenarios and an empirical illustration. British Journal of Mathematical and Statistical Psychology, 72(2), 244–270.

28.

What Works Clearinghouse. (2020). What works clearinghouse standards handbook, Version 4.1. The Institute of Education Sciences. https://ies.ed.gov/ncee/wwc/Docs/referenceresources/WWC-Standards-Handbook-v4-1-508.pdf

29.

Wooldridge

J. M.

(2020). Introductory econometrics: A modern approach (7th ed.). Cengage Learning.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.06 MB

Bias for Treatment Effect by Measurement Error in Pretest in ANCOVA Analysis

Abstract

Keywords

Get full access to this article

References

Supplementary Material