When the covariate of an evaluation study using regression analysis is falhbly measured, the
statistical test of program effectiveness is biased. In programs where the target population is
"below average, " the bias tends to suppress the beneficial effects of the program, while raising
them in programs designed for those "above average. " The magnitude of this bias is calculated,
and a correction method is derived and illustrated. In many cases, these biases will make
practical differences in program assessment, lending further support to the general distrust of
nonexperimental methods. Nonetheless, our approach improves the reliability of nonexperimen
tal methods by correcting one potential source of their bias.
Get full access to this article
View all access options for this article.
References
1.
Ashley, R., and D. Vaughan.1986. Measuring measurement error in economic time series. Journal of Business and Economic Statistics4:95-103.
2.
Cain, G.G.1975. Regression and selection models improve nonexperimental comparisons. In Evaluation and experiment: Some critical issues in assessing social programs, ed. C. A. Bennet and A. A. Lumdaine.New York: Academic Press.
3.
Campbell, D.T.1969. Reforms as experiments. American Psychologist24:409-429.
4.
Campbell, D.T., and J.C. Stanley.1966. Experimental and quasi-experimental designs for research. Chicago. Rand-McNally.
5.
Cohen, J.1969. Statistical power analysis in the behavioral sciences . New York: Academic Press.
6.
Friedman, M.1957. A theory of the consumption function. Princeton, NJ: Princeton University Press .
7.
Fuller, W.1987. Measurement error models. New York : John Wiley.
8.
Fuller, W., and M.A. Hidiroglou .1978. Regression estimation after correcting for attenuation. Journal of the American Statistical Association73:99-104.
Griliches, Z.1986. Data problems in econometrics. In Handbook of econometrics, vol. 3, ed. Z. Griliches and M. Intriligator.Amsterdam: North Holland.
11.
Heckman, J.J., V.J. Holtz, and M. Dabos.1987. Do we need experimental data to evaluate the impact of manpower training on earnings? Evaluation Review11:395-427.
12.
Herbert, M.1984. Comprehensive school mathematics program: Final evaluation report. St. Louis, MO: McREL.
13.
Judd, C.M., and D.A. Kenny.1981. Estimating the effects of social interventions . Cambridge: Cambridge University Press.
14.
Kahneman, D., and A. Tversky.1973. On the psychology of prediction. Psychological Review80:237-51.
15.
Klepper, S. and E.E. Leamer.1984. The consistent sets of estimates for regression with errors in all variables. Econometrics52:163-83.
16.
Kmenta, J.1971. Elements of econometrics. New York : Macmillan.
17.
Lichtenberg, F.R., and Z. Griliches.1989. Errors of measurement in output deflatorsJournal of Business and Economic Statistics7:1-10.
18.
Lord, F.M.1960. Large-sample covariance analysis when the control variable is fallible.Journal of the American Statistical Association55:307-21.
19.
—. 1967. A paradox in the interpretation of group comparisons . Psychological Bulletin68:304-05.
20.
Marquis, K.M., M.S. Marquis, and J.M. Polich.1986. Response bias and reliability in sensitive topic surveys. Journal of the American Statistical Association81:381-89.
21.
Moran, G.E.1985. Regulating strategies for workplace injury reduction: A program review. Evaluation Review9:21-23.
22.
Newcomb, T.M.1986. The Seattle automobile inspection and maintenance program: Multiple analysis of program impact. Evaluation Review10:217-29.
23.
Raven, J.C., J.H. Court, and J. Raven.1983. Manual for Raven's progressive matrices and vocabulary scales. London: Lewis.
24.
Robinson, A., and T.D. Stanley.1989. Teaching to talent: Evaluating an enriched and accelerated mathematics program. Journal of the Education of the Gifted12:253-67.
25.
Robinson, A., R.H. Bradley, and T.D. Stanley.1990. Opportunity to achieve: Identifying and serving mathematically talented black students. Contemporary Educational Psychology15, no. 1:1-12.
26.
Stanley, T.D.1986. Stein-rule least squares estimation: A heuristic for fallible data. Economics Letters20:147-50.
27.
—. 1988a. Forecasting from fallible data: Correcting prediction bias with Stein-rule least squares. Journal of Forecasting7:103-13.
28.
—. 1988b. At the statistical crossroads, Stein-rule least squares . In Decision sciences: Theory and applications, Proceedings of the nineteenth annual Southwest Decision Sciences Institute, ed. R. B. Martin and V. P. Shah , 68-70.
29.
Stanley, T.D., and A. Robinson.1986. Regression discontinuity: Integrating research and program design in programs for the gifted. Journal for the Education of the Gifted9:181-91.
30.
Step Iii.1979. Manual and technical report. Menlo Park, CA: Addison-Wesley.
31.
Stock, J.H.1988. A reexamination of Friedman's consumption puzzle . Journal of Business and Economics Statistics6:401-07.
32.
Stromsdorfer, E.W.1987. Economic evaluation of the Comprehensive Employment and Training Act: An overview of recent findings and advances in evaluation methods . Evaluation Review11:387-94.
33.
Trochim, T.1984. Research design for program evaluation. Beverly Hills, CA: Sage.
