Donald Campbell and Lee Cronbach had a long history of mutual respect for and fundamental disagreement with each other's ideas about experimental validity. Issues that Campbell labeled as external validity, Cronbach labeled internal validity. Issues that Campbell labeled internal validity, Cronbach suggested are trivial. Nevertheless, these methodological pioneers share much common ground, in part because of their alliance with Egon Brunswik. As science moved from a deterministic to a probabilistic paradigm, all 3 endeavored to protect behavioral science from validity-threatening practices that could result from naive use of the Fisherian approach to scientific investigation. This review shows that issues concerning the prioritization of types of validity still need to be resolved and that most social scientists do not understand internal validity. Several empirical practices for enhancing validity are suggested.
Get full access to this article
View all access options for this article.
References
1.
AiryB. B. (1861). On the algebraical and numerical theory of errors of observations and the combinations of observations. London: Macmillan.
2.
AlbrightL., & MalloyT. E. (1999). Self-observation of social behavior and metaperception. Journal of Personality and Social Psychology, 77, 726–734.
3.
BanduraA. (1986). Social foundations of thought and action: A social–cognitive theory. Englewood Cliffs, NJ: Prentice Hall.
4.
BollenK. A., & LongJ. S. (1993). Testing structural equation models. Newbury Park, CA: Sage.
BrowneM. W. (1984). The decomposition of multitrait-multimethod matrices. British Journal of Mathematical and Statistical Psychology, 37, 1–21.
7.
BrunswikE. (1934). Wahrnehmung und Gegenstandswelt. [Perception and the world of objects]. Leipzig, Germany: Deuticke.
8.
BrunswikE. (1947). Systematic and representative design of psychological experiments. Berkeley: University of California Press.
9.
BrunswikE. (1952). The conceptual framework of psychology. In CarnapR. & MorrisC., International encyclopedia of unified science (Vol. 1, No. 10, iv–102).Chicago: University of Chicago Press.
10.
BrunswikE. (1956). Perception and the representative design of psychological experiments. Berkeley: University of California Press.
11.
BrunswikE. (1957). Scope and aspects of the cognitive problem. In GruberH., JessorR., & HammondK., Cognition: The Colorado Symposium (pp. 5–31). Cambridge, MA: Harvard University Press.
12.
BrykA. S., & RaudenbushS. W. (1988). On heterogeneity of variance in experimental studies: A challenge to conventional interpretations. Psychological Bulletin, 104, 396–404.
13.
BrykA. S., & RaudenbushS. W. (1992). Hierarchical linear models. Newbury Park, CA: Sage.
14.
CampbellD. T. (1957). Factors relevant to the validity of experiments in social settings. Psychological Bulletin, 54, 297–312.
15.
CampbellD. T. (1960). Blind variation and selective retention in creative thought as in other knowledge processes. Psychological Review, 67, 380–400.
16.
CampbellD. T. (1969). Definitional versus multiple operationism. et al, 2, 14–17.
17.
CampbellD. T. (1974). Evolutionary epistemology. In SchilppP. A., The philosophy of Karl Popper (pp. 413–463). LaSalle, IL: Open Court.
18.
CampbellD. T. (1979). A tribal model of the social system vehicle carrying scientific knowledge. Knowledge, 2, 181–201.
19.
CampbellD. T. (1986). Relabelling internal and external validity for applied social scientists. In TrochimW. M. K., Advances in quasi-experimental design and analysis: New directions for program evaluation (Vol. 31, pp. 66–77). San Francisco: Jossey-Bass.
20.
CampbellD. T. (1993). Plausible coselection of belief by referent: All the “objectivity” that is possible. Perspectives on Science, 1, 88–108.
21.
CampbellD. T., & FiskeD. (1959). Convergent and discriminant validity by the multitrait-multimethod matrix. Psychological Bulletin, 65, 81–105.
22.
CampbellD. T., & OvermanE. S.. (1988). Methodology and epistemology for social science. Chicago: University of Chicago Press.
23.
CampbellD. T., & StanleyJ. (1966). Experimental and quasi-experimental designs for research. Chicago: Rand McNally.
24.
CookT. D. (1985). Post-positivist critical multiplism. In ShotlandL. & MarkM. M., Social science and social policy (pp. 21–62). Beverly Hills, CA: Sage.
25.
CookT. D., & CampbellD. T. (1976). The design and conduct of quasi-experiments and true experiments in field settings. In DunnetteM. D., Handbook of industrial and organizational psychologyChicago: Rand McNally.
26.
CookT. D., & CampbellD. T. (1979). Quasi-experimentation: Design and analysis for field settings. Chicago: Rand McNally.
27.
CookT. D., CampbellD. T., & PeracchioL. (1990). Quasi-experimentation. In DunnetteM. D. & HoughL. M., Handbook of industrial and organizational psychology (Vol. 1, 2nd ed., (pp. 491–576). Palo Alto, CA: Consulting Psychologists Press.
28.
CronbachL. J. (1955). Processes affecting scores on “understanding of others” and “assumed similarity.”Psychological Bulletin, 52, 177–193.
29.
CronbachL. J. (1975). Beyond the two disciplines of scientific psychology. American Psychologist, 30, 116–127.
CronbachL. J. (1982). Designing evaluations of educational and social programs. San Francisco: Jossey-Bass.
32.
CronbachL. J. (1995). Giving method variance its due. In ShroutP. E. & FiskeS. T., Personality research, methods, and theory (pp. 145–157). Hillsdale, NJ: Erlbaum.
33.
CronbachL. J., GleserG. C., NandaH., & RajaratnamN. (1972). The dependability of behavioral measurements: Theory of generalizability for scores and profiles. New York: Wiley.
34.
CronbachL. J., & MeehlP. E. (1955). Construct validity in psychological tests. Psychological Bulletin, 52, 281–302.
35.
DempsterA. P., LairdN. M., & RubinD. B. (1977). Maximum likelihood for incomplete data via the EM algorithm. Journal of the Royal Statistical Society, Series B, 39, 1–8.
36.
EinsteinA. (1961). Relativity: The special and the general theory. New York: Bonanza Books.
37.
EisenhartC. (1947). The assumptions underlying the analysis of variance. Biometrics, 3, 1–21.
38.
ErlebacherA. (1977). Design and analysis of experiments contrasting the within and between subjects manipulation of the independent variable. Psychological Bulletin, 84, 212–219.
39.
FisherR. A. (1925). Statistical methods for research workers. London: Oliver & Boyd.
40.
GigerenzerG. (1987). Survival of the fittest probabilist: Brunswik, Thurstone, and the two disciplines of psychology. In KrugerL., GigerenzerG., & MorganM. S., The probabilistic revolution: Vol. 2. Ideas in the sciences (pp. 49–72). Cambridge, MA: MIT Press.
41.
HammondK. R. (1955). Probabilistic functioning and the clinical method. Psychological Review, 62, 255–262.
42.
HammondK. R.. (1966). The psychology of Egon Brunswik. New York: Holt, Rinehart & Winston.
43.
HammondK. R. (1990). Functionalism and illusionism: Can integration be usefully achieved? In R. M. Hogarth (Ed.), Insights in decision making: A tribute to Hillel J. Einhorn (pp. 227–261). Chicago: University of Chicago Press.
44.
HammondK. R. (1995). Expansion of Egon Brunswik's psychology, 1955–1995. (Vienna Circle Institute Yearbook 3). Dordrecht, the Netherlands: Kluwer.
45.
HeisenbergW. (1930). The quantum theory. Chicago: University of Chicago Press.
46.
HendersonC. R. (1953). Estimation of variance and covariance components. Biometrics, 9, 226–252.
47.
HoutsA. C., CookT. D., & ShadishW. R. (1986). The person-situation debate: A critical multiplist perspective. Journal of Personality, 54, 52–105.
48.
HurschC., HammondK., & HurschJ. (1964). Some methodological considerations in multiple-cue probability studies. Psychological Review, 71, 42–60.
49.
KennyD. A. (1979). Correlation and causality. New York: Wiley.
50.
KennyD. A. (1995). The multitrait-multimethod matrix: Design, analysis, and conceptual issues. In ShroutP. E. & FiskeS. T., Personality research, methods, and theory (pp. 111–124). Hillsdale, NJ: Erlbaum.
51.
KrugerL., GigerenzerG., & MorganM. S.. (1987). The probabilistic revolution: Vol. 2. Ideas in the sciences. Cambridge, MA: MIT Press.
52.
KuhnT. S. (1970). The structure of scientific revolutions (2nd ed.). Chicago: University of Chicago Press.
53.
KuhnT. S. (1977). The essential tension: Selected studies in scientific tradition and change. Chicago: University of Chicago Press.
54.
LandauH. G. (1951a). On dominance relations and the structure of animal societies: I. Effect of inherent characteristics. Bulletin of Mathematical Biophysics, 13, 1–19.
55.
LandauH. G. (1951b). On dominance relations and the structure of animal societies: II. Some effects of possible social factors. Bulletin of Mathematical Biophysics, 13, 245–262.
56.
LearyD. (1987). From act psychology to probabilistic functionalism: The place of Egon Brunswik in the history of psychology. In AshM. G. & WoodwardW. R., Psychology in twentieth-century thought and society (pp. 115–142). Cambridge, England: Cambridge University Press.
57.
LindleyD. V., & SmithA. F. M. (1972). Bayes estimates for the linear model. Journal of the Royal Statistical Society, Series B, 34, 1–41.
58.
LordF. M., & NovickM. R. (1968). Statistical theories of mental test scores. Reading, MA: Addison-Wesley.
59.
MookD. G. (1983). In defense of external invalidity. American Psychologist, 38, 379–387.
60.
MookD. G. (1989). The myth of external validity. In PoonL. W., RubinD. C., & WilsonB. A., Everyday cognition in adulthood and late life (pp. 25–43). New York: Cambridge University Press.
61.
MulaikS. A., & MillsapR. E. (2000). Doing the four-step right. Structural Equation Modeling, 7, 36–73.
62.
RaudenbushS. W., & WillmsJ. D. (1991). Schools, classrooms, and pupils: International studies of schooling from a multilevel perspective. Orlando, FL: Academic Press.
63.
SearleS. R. (1971). Linear models. New York: Wiley.
64.
SearleS. R., CasellaG., & McCullochC. E. (1992). Variance components. New York: Wiley.
65.
SnowR. E., & WileyD. E.. (1991). Improving inquiry in social science: A volume in honor of Lee J. Cronbach. Hillsdale, NJ: Erlbaum.
66.
TolmanE. C., & BrunswikE. (1935). The organism and the causal texture of the environment. Psychological Review, 42, 43–77.
67.
TryonR. C. (1930). Studies in individual differences in maze ability I: The measurement of the reliability of individual differences. Journal of Comparative Psychology, 11, 145–170.
68.
WothkeW. (1995). Covariance components analysis of the multitrait-multimethod matrix. In ShroutP. E. & FiskeS. T., Personality research, methods, and theory (pp. 125–144). Hillsdale, NJ: Erlbaum.
69.
WothkeW., & BrowneM. W. (1990). The direct product model for the MTMM matrix parameterized as a second order factor analysis model. Psychometrika, 55, 255–262.
