This paper addresses the question of internal validity in quasi-experimental and nonexperimental social policy research. We focus primary attention on problems of selection in research designs where assignment to experimental conditions occurs on a nonrandom basis. Five different solutions to the problems of selection are discussed-randomization, covariance adjustment, gain scores, matching, and explicit selection-and we examine the conditions under which these solutions are useful. The problem of random measurement error is discussed within the context of our elaboration of these techniques.
Get full access to this article
View all access options for this article.
References
1.
Althauser, R.P. and D. Rubin (1971) "Measurement error and regression to the mean in matched samples." Social Forces50 (December): 206-214.
2.
Alwin, D.F. and R.C. Tessler (1974) "Causal models, unobserved variables, and experimental data." Amer. J. of Sociology80 (July): 58-86.
3.
Armor, D. (1973) "The double standard: a reply." Public Interest30 (Winter): 119-131.
4.
Barnow, B.S. (1972) "Conditions for the presence or absence of a bias in treatment effect: some statistical models for Head Start evaluation." Discussion Paper 122-72. Institute for Research on Poverty. University of Wisconsin, Madison.
5.
Bereiter, C. (1962) "Some persisting dilemmas in the measurement of change," in C. W. Harris (ed.) Problems in the Measurement of Change. Madison : Univ. of Wisconsin Press.
6.
Bohrnstedt, G.W. (1969) "Observations on the measurement of change," pp. 113-133 in E. F. Borgatta and G. W. Bohrnstedt (eds.) Sociological Methodology 1969. San Francisco: Jossey-Bass.
7.
Bowles, S. and H.M. Levin (1968a) "More on multicollinearity and the effectiveness of schools." J. of Human Resources3 (Spring): 393-400.
8.
--- ( 1968b) "The determinants of scholastic achievement-an appraisal of some recent evidence." J. of Human Resources3 (Winter): 3-24.
9.
Cain, G.C. and H.W. Watts (1970) "Problems in making policy inferences from the Coleman Report." Amer. Soc. Rev.35 (April): 228-242.
10.
Campbell, D.T. (1969) "Reforms as experiments." Amer. Psychologist24 (April): 409-429.
11.
--- and K.N. Clayton (1961) "Avoiding regression effects in panel studies of communication impact." Studies in Public Communication3 (Summer): 99-118.
12.
Campbell, D.T. and A. Erlebacher (1970a) "How regression artifacts in quasi-experimental evaluations can mistakenly make compensatory education look harmful," pp. 185-210 in J. Hellmuth (ed.) Disadvantaged Child. VoL3.New York: Brunner/Mazel .
13.
--- ( 1970b) "Reply to the replies," pp. 221-225 in J. Hellmuth (ed.) Disadvantaged Child. VoL3.New York: Brunner/Mazel .
14.
Campbell, D.T. and J. Stanley (1963) "Experimental and quasi-experimental designs for research on teaching," pp. 171-247 in N. Gage (ed.) Handbook for Research on Teaching. Chicago: Rand McNally.
15.
Caro, F.G. (1971) Readings in Evaluation Research. New York: Russell Sage.
16.
Cicirelli, V.G. (1970) "The relevance of the regression to the mean artifact problem to the Westinghouse-Ohio evaluation of Head Start: a reply to Campbell and Erlebacher," pp. 211-215 in J. Hellmuth (ed.) Disadvantaged Child. Vol. 3. New York: Brunner/MazeL
17.
Cohen, J. (1968) "Multiple regression as a general data-analytic system." Psych. Bull.70 (December): 426-443.
18.
Coleman, J.S. (1970) "Reply to Cain and Watts." Amer. Soc. Rev.35 (April): 242-249.
19.
--- ( 1968) "Equality of educational opportunity: reply to Bowles and Levin." J. of Human Resources3 (Spring): 237-246.
20.
Cronbach, L.J. and L. Furby (1970) "How should we measure 'change'-or should we?" Psych. Bull.74 (July): 68-80.
21.
Evans, J.W. and J. Schiller (1970) "How preoccupation with possible regression artifacts can lead to a faulty strategy for the evaluation of social action programs: a reply to Campbell and Erlebacher," pp. 216-220 in J. Hellmuth (ed.) Disadvantaged Child. Vol. 3. New York: Brunner/Mazel .
22.
Goldberger, A.S. (1972) "Selection bias in evaluating treatment effects: some formal illustrations." Discussion Paper 123-72. Institute for Research on Poverty. University of Wisconsin, Madison.
23.
Graybill, F.A. (1961) An Introduction to Linear Statistical Models. Vol. 1. New York: McGraw-Hill.
24.
Kenny, D. (1975) "A quasi-experimental approach to assessing treatment effects in the nonequivalent control group design." Psych. Bull.82 (May): 345-362.
25.
Kish, L. (1959) "Some statistical problems in research design." Amer. Soc. Rev.24 (June): 328-338.
26.
Lord, F.M. and M.R. Novick (1968) Statistical Theories of Mental Test Scores. Reading, Mass.: Addison-Wesley.
27.
Mosteller, F. and D.P. Moynihan (1972) On Equality of Educational Opportunity. New York: Random House.
28.
Namboodiri, N.K., L.F. Carter, and H.M. Blalock, Jr. (1975) Applied Multivariate Analysis and Experimental Designs. New York: McGraw-Hill.
29.
Pettigrew, T.F., E.L. Useem, C. Normand, and M.S. Smith (1973) "Busing: a review of 'the evidence'." Public Interest30 (Winter): 88-118.
30.
Porter, A.C. and T.R. Chibucos (1975) "Common problems of design and analysis in evaluative research." Soc. Methods and Research3 (February): 235-257.
31.
Riecken, H.W. and R.F. Boruch (1974) Social Experimentation: A Method for Planning and Evaluating Social Intervention. New York: Academic Press.
32.
Ross, J. and P. Smith (1968) "Orthodox experimental designs," pp. 333-389 in H. M. Blalock, Jr. and A. B. Blalock (eds.) Methodology in Social Research. New York : McGraw-Hill.
33.
Rossi, P.H. and W. Williams (1972) Evaluating Social Programs. New York: Seminar Press.
34.
Suchman, E.A. (1967) Evaluative Research. New York : Russell Sage.
35.
Weiss, C.H. (1972a) Evaluation Research: Methods of Assessing Program Effectiveness. Englewood Cliffs, N.J.: Prentice-Hall.
36.
——— (1972b) Evaluating Action Programs: Readings in Social Action and Education. Boston: Allyn & Bacon.
