This paper delimits and explicates threats to external validity particularly problematic in evaluation research. Five categories of factors are discussed: selection effects, measurement effects, confounded treatment effects, situational effects, and effects due to differential mortality. The paper focuses on pointing up specific ways in which each of the factors threaten generalizability and possible solutions to the methodological problems presented.
Get full access to this article
View all access options for this article.
References
1.
Anderson, A. (1975) Principal Investigator: Evaluation of Gary Income Maintenance Experiment Personal conversations with author.
2.
Bernstein, I. and H.E. Freeman (1975) Academic and Entrepreneurial Research: The Consequences of Diversity in Federal Evaluation Studies. New York: Russell Sage.
3.
Bernstein, I. and E.B. Sheldon (1975) "Method of evaluative research," in R. Smith (ed.) Social Science Methods. New York: Free Press.
4.
Borgatta, E.F. and R.R. Evans (1968) Smoking and Health. Chicago : Aldine.
5.
Bracht, G.H. and G.V. Glass (1968) "The external validity of experiments." Amer. Educ. Research J.5: 437-474.
6.
Cain, G.C. and R.G. Hollister (1972) "The methodology of evaluating social action programs ," pp. 109-137 in P. Rossi and W. Williams (eds.) Evaluating Social Programs . New York: Seminar Press.
7.
Campbell, D.T. (1969) "Perspective: artifact and control," pp. 351-382 in R. Rosenthal and R. L. Rosnow (eds.) Artifact in Behavioral Research. New York: Academic Press .
8.
——— (1957) "Factors relevant to the validity of experiments in social settings." Psych. Bull.54: 297-312.
9.
——— and H.L. Ross (1968) "The Connecticut crackdown in speeding: time-series data in quasi experimental analysis." Law and Society Rev.3, 1: 33-53.
10.
Campbell, D. and J.C. Stanley (1963) Experimental and Quasi-Experimental Designs for Research. Chicago: Rand McNally .
11.
Cohen, J. (1968) "Multiple regression as a general data analytic system." Psych. Bull.70: 426-443.
12.
Cornfield, J. and J.W. Tukey (1956) "Average values of mean squares in factorials." Annals of Mathematical Statistics27: 907-949.
13.
Freeman, H.E. and C.C. Sherwood (1965) "Research in large-scale intervention programs ," pp. 262-276 in F. G. Caro (ed.) Readings in Evaluation Research. New York: Russell Sage.
14.
Glass, G.V. (1968) "Analysis of data on the Connecticut speeding crackdown as a time series quasi-experiment." Law and Society Rev.3, 1: 55-76.
15.
Hovland, C.I., A.A. Lumsdaine, and F.D. Sheffield (1949) Experiments on Mass Communication. Princeton: Princeton Univ. Press.
16.
Hunt, D.E. and R.H. Hardt (1969) "The effect of Upward Bound programs on the attitudes, motivation, and academic achievement of Negro students." J. of SocialTimes25: 117-129.
17.
Hyman, H. and C.R. Wright (1967) "Evaluating social action programs," pp. 185-220 in F. G. Caro (ed.) Readings in Evaluation Research. New York: Russell Sage.
18.
——— and T. Hopkins (1962) Applications of Methods of Evaluation: Four Studies of the Encampment for Citizenship. Berkeley: Univ. of California Press.
19.
Kerlinger, F.N. and E.J. Pedhazur (1973) Multiple Regression in Behavioral Research. New York: Holt, Rinehart & Winston .
20.
Kershaw, D. (1972) "Issues in income maintenance experimentation," in P. H. Rossi and W. Williams (eds.) Evaluating Social Programs. New York: Seminar Press.
21.
Lana, R.E. (1969) "Pretest sensitization," pp. 119-141 in R. Rosenthal and R. L. Rosnow (eds.) Artifact in Behavioral Research. New York : Academic Press.
22.
Lord, F.N. and M.R. Novick (1968) Statistical Theories of Mental Test Scores. Reading, Mass.: Addison-Wesley.
23.
Lubin, A. (1961) "The interpretation of significant interaction ." Educ. and Psych. Measurement21: 807-817.
24.
McDill, E.L., M.S. McDill, and J.T. Sprehe (1972) "Evaluation in practice: contemporary education ," pp. 141-185 in P. Rossi and W. Williams (eds.) Evaluating Social Programs . New York: Seminar Press.
25.
McGuire, W.J. (1969) "Suspiciousness of experimenter 'intent', " pp. 13-57 in R. Rosenthal and R L Rosnow (eds.) Artifact in Behavioral Research . New York: Academic Press.
26.
Orne, M.T. (1969) "Demand characteristics and the concept of quasi controls," pp. 143-179 in R Rosenthal and R. L. Rosnow (eds.) Artifact in Behavioral Research. New York: Academic Press.
27.
Pelz, D.C. and R.A. Lew (1970) "Heise's causal model applied," pp. 28-37 in E. F. Borgatta and G. W. Bohrnstedt (eds.) Sociological Methodology. San Francisco: Jossey-Bass.
28.
Roethlisberger, K. and W. Dickson (1939) Management and the Worker. Cambridge: Harvard Univ. Press.
29.
Rosenberg, M.J. (1969) "The conditions and consequences of evaluation apprehension," pp. 279-349 in R Rosenthal and R. L. Rosnow (eds.) Artifact in Behavioral Research. New York: Academic Press.
30.
Rosenthal, R. (1969) "Interpersonal expectations: effects of the experimenter'o hypothesis," pp. 181-277 in R Rosenthal and R L. Rosnow (eds.) Artifact in Behavioral Research. New York: Academic Press .
31.
Rossi, P. (1972) "Testing for success and failure in social action ," pp. 11-49 in P. Rossi and W. Williams (eds.) Evaluating Social Programs. New York: Seminar Press.
32.
——— and W. Williams [eds.] (1972) Evaluating Social Programs. New York: Seminar Press.
33.
Suchman, E. (1967) Evaluative Research. New York : Russell Sage.
34.
Webb, E.J. et al. (1966) Unobtrusive Measures: Nonreactive Research in the Social Sciences. Chicago: Rand McNally.
35.
Weiss, C. (1970) "The politicization of evaluation research." J. of Social Issues26, 4: 57-67.
36.
Williams, W. and J.W. Evans (1972) "The politics of evaluation: the case of Head Start ," pp. 247-264 in P. Rossi and W. Williams (eds.) Evaluating Social Programs . New York: Seminar Press.
