Only true experiments offer definitive evidence for causal inferences, but not all educational interventions are readily amenable to experiments. Correlational evidence can at least tentatively inform evidence-based practice when sophisticated causal modeling or exclusion methods are employed. Correlational evidence is most informative when exemplary practices are followed as regards (a) measurement, (b) quantifying effects, (c) avoiding common analysis errors, and (d) using confidence intervals to portray the range of possible effects and the precisions of the effect estimates.
Get full access to this article
View all access options for this article.
References
1.
AlginaJ.KeselmanH. J. (2003). Approximate confidence intervals for effect sizes. Educational and Psychological Measurement, 63, 537–553.
2.
American Psychological Association. (2001). Publication manual of the American Psychological Association (5th ed.). Washington, DC: Author.
3.
AndersonD. R.BurnhamK. P.ThompsonW. (2000). Null hypothesis testing: Problems, prevalence, and an alternative. Journal of Wildlife Management, 64, 912–923.
4.
AtkinsonR. C.JacksonG. B. (Eds.). (1992). Research and education reform: Roles for the Office of Educational Research and Improvement. Washington, DC: National Academy of Sciences. (ERIC Document Reproduction Service No. ED 343 961).
5.
BagozziR. P. (1980). Performance and satisfaction in an industrial sales force: An examination of their antecedents and simultaneity. Journal of Marketing, 44, 65–77.
6.
BagozziR. P.FornellC.LarckerD. F. (1981). Canonical correlation analysis as a special case of a structural relations model. Multivariate Behavioral Research, 16, 437–454.
7.
BerlinerD. C. (2002). Educational research: The hardest science of all. Educational Researcher, 31 (8), 18–20.
8.
BorgenF. H.SelingM. J. (1978). Uses of discriminant analysis following MANOVA: Multivariate statistics for multivariate purposes. Journal of Applied Psychology, 63, 689–697.
9.
BrennanR. L. (2001). Some problems, pitfalls, and paradoxes in educational measurement. Educational Measurement: Issues and Practices, 20 (4), 6–18.
10.
CampbellD. T.ErlebacherA. (1975). How regression artifacts in quasiexperimental evaluations can mistakenly make compensatory education look harmful. In GuttentagM.StrueningE. L. (Eds.), Handbook of evaluation research (Vol. 1, pp. 597–617). Beverly Hills, CA: Sage.
11.
ChamblessD. (1998). Defining empirically supported therapies. Journal of Consulting and Clinical Psychology, 66, 7–18.
12.
ChandlerR. (1957). The statistical concepts of confidence and significance. Psychological Bulletin, 54, 429–430.
CohenJ. (1968). Multiple regression as a general data-analytic system. Psychological Bulletin, 70, 426–443.
15.
CohenJ. (1994). The earth is round (p < .05). American Psychologist, 49, 997–1003.
16.
CourvilleT.ThompsonB. (2001). Use of structure coefficients in published multiple regression articles: β is not enough. Educational and Psychological Measurement, 61, 229–248.
17.
CrockerL.AlginaJ. (1986). Introduction to classical and modern test theory. New York: Holt, Rinehart & Winston.
18.
CummingG.FinchS. (2001). A primer on the understanding, use and calculation of confidence intervals that are based on central and noncentral distributions. Educational and Psychological Measurement, 61, 532–575.
19.
DuncanO. D. (1975). Introduction to structural equation models. New York: Academic Press.
20.
DunlapW. P.LandisR. S. (1998). Interpretations of multiple regression borrowed from factor analysis and canonical correlation. The Journal of General Psychology, 125, 397–407.
21.
FanX.ThompsonB. (2001). Confidence intervals about score reliability coefficients, please: An EPM guidelines editorial. Educational and Psychological Measurement, 61, 517–531.
22.
FeuerM. J.TowneL.ShavelsonR. J. (2002). Scientific culture and educational research. Educational Researcher, 31 (8), 4–14.
23.
FidlerF. (2002). The fifth edition of the APA Publication Manual: Why its statistics recommendations are so controversial. Educational and Psychological Measurement, 62, 749–770.
24.
FidlerF.ThomasonN.CummingG.FinchS.LeemanJ. (2004). Editors can lead researchers to confidence intervals, but they can't make them think: Statistical reform lessons from medicine. Psychological Science, 15, 119–127.
25.
FishL. J. (1988). Why multivariate methods are usually vital. Measurement and Evaluation in Counseling and Development, 21, 130–137.
26.
GallM. D.BorgW. R.GallJ. P. (1996). Educational research: An introduction (6th ed.). White Plains, NY: Longman.
27.
GlassG. V.McGawB.SmithM. L. (1981). Meta-analysis in social research. Beverly Hills, CA: Sage.
28.
GorsuchR. L. (1983). Factor analysis (2nd ed.). Hillsdale, NJ: Erlbaum.
29.
GrahamJ. M.GuthrieA. C.ThompsonB. (2003). Consequences of not interpreting structure coefficients in published CFA research: A reminder. Structural Equation Modeling, 10, 142–153.
30.
HarlowL. L.MulaikS. A.SteigerJ. H. (Eds.). (1997). What if there were no significance tests?Mahwah, NJ: Erlbaum.
31.
HesterY. C. (2000). An analysis of the use and misuse of ANOVA. (Doctoral dissertation, Texas A&M University, 2000). Dissertation Abstracts International, 61, 4332A. (UMI No. 9994257).
32.
HubertyC. J. (1994). Applied discriminant analysis. New York: Wiley & Sons.
33.
HubertyC. J. (2002). A history of effect size indices. Educational and Psychological Measurement, 62, 227–240.
34.
JacobsonN. S.RobertsL. J.BernsS. B.McGlincheyJ. B. (1999). Methods for defining and determining the clinical significance of treatment effects: Description, application, and alternatives. Journal of Consulting and Clinical Psychology, 67, 300–307.
35.
JöreskogK. G. (1969). A general approach to confirmatory maximum likelihood factor analysis. Psychometrika, 34, 183–220.
36.
JöreskogK. G. (1970). A general method for analysis of covariance structures. Biometrika, 57, 239–251.
37.
JöreskogK. G. (1971). Simultaneous factor analysis in several populations. Psychometrika, 36, 409–426.
38.
JöreskogK. G. (1978). Structural analysis of covariance and correlation matrices. Psychometrika, 43, 443–477.
39.
JöreskogK. G.SörbomD. (1989). LISREL 7: A guide to the program and applications (2nd ed.). Chicago: SPSS.
40.
KendallP. C. (1999). Clinical significance. Journal of Consulting and Clinical Psychology, 67, 283–284.
41.
KerlingerF. N. (1986). Foundations of behavioral research (3rd ed.). New York: Holt, Rinehart & Winston.
42.
KeselmanH. J.HubertyC. J.LixL. M.OlejnikS.CribbieR.DonahueB. (1998). Statistical practices of educational researchers: An analysis of their ANOVA, MANOVA and ANCOVA analyses. Review of Educational Research, 68, 350–386.
43.
KiefferK. M.ReeseR. J.ThompsonB. (2001). Statistical techniques employed in AERJ and JCP articles from 1988 to 1997: A methodological review. Journal of Experimental Education, 69, 280–309.
44.
KirkR. (1996). Practical significance: A concept whose time has come. Educational and Psychological Measurement, 56, 746–759.
45.
KlineR. (2004). Beyond significance testing: Reforming data analysis methods in behavioral research. Washington, DC: American Psychological Association.
46.
KnappT. R. (1978). Canonical correlation analysis: A general parametric significance testing system. Psychological Bulletin, 85, 410–416.
47.
LudbrookJ.DudleyH. (1998). Why permutation tests are superior to t and F tests in medical research. The American Statistician, 52, 127–132.
48.
McLeanJ. E.KaufmanA. S. (2000). Editorial: Statistical significance testing and other changes to Research in the Schools, 7 (2), 1–2.
NickersonR. S. (2000). Null hypothesis significance testing: A review of an old and continuing controversy. Psychological Methods, 5, 241–301.
51.
OlejnikS.AlginaJ. (2000). Measures of effect size for comparative studies: Applications, interpretations, and limitations. Contemporary Educational Psychology, 25, 241–286.
52.
PedhazurE. J. (1982). Multiple regression in behavioral research: Explanation and prediction (2nd ed.). New York: Holt, Rinehart & Winston.
53.
SackettD. L.StrausS. E.RichardsonW. S.RosenbergW.HaynesR. B. (2000). Evidence-based medicine: How to practice and teach EBM (2nd ed.). New York: Churchill Livingstone.
54.
SchmidtF. L. (1996). Statistical significance testing and cumulative knowledge in psychology: Implications for the training of researchers. Psychological Methods, 1, 115–129.
55.
SchmidtF. L.HunterJ. E. (1977). Development of a general solution to the problem of validity generalization. Journal of Applied Psychology, 62, 529–540.
56.
ShavelsonR. J.TowneL. (Eds.). (2002). Scientific research in education. Washington, DC: National Academy Press.
57.
SmithsonM. (2001). Correct confidence intervals for various regression effect sizes and parameters: The importance of noncentral distributions in computing intervals. Educational and Psychological Measurement, 61, 605–632.
58.
SnyderP. (1991). Three reasons why stepwise regression methods should not be used by researchers. In ThompsonB. (Ed.), Advances in educational research: Substantive findings, methodological developments (Vol. 1, pp. 99–105). Greenwich, CT: JAI Press.
59.
SnyderP. (2000). Guidelines for reporting results of group quantitative investigations. Journal of Early Intervention, 23, 145–150.
60.
SnyderP.LawsonS. (1993). Evaluating results using corrected and uncorrected effect size estimates. Journal of Experimental Education, 61, 334–349.
61.
SpearmanC. (1904). The proof and measurement of association between two things. Journal of Psychology, 15, 72–101.
62.
SteigerJ. H.FouladiR. T. (1992). R2: A computer program for interval estimation, power calculation, and hypothesis testing for the squared multiple correlation. Behavior Research Methods, Instruments, and Computers, 4, 581–582.
63.
ThompsonB. (1984). Canonical correlation analysis: Uses and interpretation. Newbury Park, CA: Sage.
64.
ThompsonB. (1986). ANOVA versus regression analysis of ATI designs: An empirical investigation. Educational and Psychological Measurement, 46, 917–928.
65.
ThompsonB. (1992). Misuse of ANCOVA and related “statistical control” procedures. Reading Psychology, 13, iii–xviii.
66.
ThompsonB. (1995). Stepwise regression and stepwise discriminant analysis need not apply here: A guidelines editorial. Educational and Psychological Measurement, 55, 525–534.
ThompsonB. (1998a). In praise of brilliance: Where that praise really belongs. American Psychologist, 53, 799–800.
69.
ThompsonB. (1998b, July). The ten commandments of good Structural Equation Modeling behavior: A user-friendly, introductory primer on SEM. Paper presented at the annual meeting of the U.S. Department of Education, Office of Special Education Programs Project Directors' Conference, Washington, DC. (ERIC Document Reproduction Service No. ED 420 154).
70.
ThompsonB. (1999a, April). Common methodology mistakes in educational research, revisited, along with a primer on both effect sizes and the bootstrap. Paper presented at the annual meeting of the American Educational Research Association, Montreal, Canada. (ERIC Document Reproduction Service No. ED 429 110).
71.
ThompsonB. (1999b). Improving research clarity and usefulness with effect size indices as supplements to statistical significance tests. Exceptional Children, 65, 329–337.
72.
ThompsonB. (2000a). Canonical correlation analysis. In GrimmL.YarnoldP. (Eds.), Reading and understanding more multivariate statistics (pp. 285–316). Washington, DC: American Psychological Association.
73.
ThompsonB. (2000b). Ten commandments of structural equation modeling. In GrimmL.YarnoldP. (Eds.), Reading and understanding more multivariate statistics (pp. 261–284). Washington, DC: American Psychological Association.
74.
ThompsonB. (2001). Significance, effect sizes, stepwise methods, and other issues: Strong arguments move the field. Journal of Experimental Education, 70, 80–93.
75.
ThompsonB. (2002a). “Statistical,” “practical,” and “clinical”: How many kinds of significance do counselors need to consider?Journal of Counseling and Development, 80, 64–71.
76.
ThompsonB. (2002b). What future quantitative social science research could look like: Confidence intervals for effect sizes. Educational Researcher, 31 (3), 24–31.
77.
ThompsonB. (Ed.). (2003). Score reliability: Contemporary thinking on reliability issues. Newbury Park, CA: Sage.
78.
ThompsonB. (in press-a). Research synthesis: Effect sizes. In CamilliG.ElmoreP. B.GreenJ. (Eds.), Complementary methods for research in education. Washington, DC: American Educational Research Association.
79.
ThompsonB. (in press-b). The “significance” crisis in psychology and education. Journal of Socio-Economics.
80.
ThompsonB.BorrelloG. M. (1985). The importance of structure coefficients in regression research. Educational and Psychological Measurement, 45, 203–209.
81.
Vacha-HaaseT. (1998). Reliability generalization: Exploring variance in measurement error affecting score reliability across studies. Educational and Psychological Measurement, 58, 6–20.
82.
Vacha-HaaseT.HensonR. K.CarusoJ. (2002). Reliability generalization: Moving toward improved understanding and use of score reliability. Educational and Psychological Measurement, 62, 562–569.
83.
Vacha-HaaseT.KoganL. R.ThompsonB. (2000). Sample compositions and variabilities in published studies versus those in test manuals: Validity of score reliability inductions. Educational and Psychological Measurement, 60, 509–522.
84.
Vacha-HaaseT.NilssonJ. E.ReetzD. R.LanceT. S.ThompsonB. (2000). Reporting practices and APA editorial policies regarding statistical significance and effect size. Theory & Psychology, 10, 413–425.
85.
WhittingtonD. (1998). How well do researchers report their measures? An evaluation of measurement in published educational research. Educational and Psychological Measurement, 58, 21–37.
86.
WilcoxR. R. (1998). How many discoveries have been lost by ignoring modern statistical methods?American Psychologist, 53, 300–314.
87.
WilkinsonL., & APA Task Force on Statistical Inference. (1999). Statistical methods in psychology journals: Guidelines and explanations. American Psychologist, 54, 594–604. [reprint available through the APA Home Page: http://www.apa.org/journals/amp/amp548594.html].
88.
WrightS. (1921). Correlation and causality. Journal of Agricultural Research, 20, 557–585.
89.
WrightS. (1934). The method of path coefficients. Annals of Mathematical Statistics, 5, 161–215.
