How doctoral programs train future researchers in quantitative methods has important implications for the quality of scientifically based research in education. The purpose of this article, therefore, is to examine how quantitative methods are used in the literature and taught in doctoral programs. Evidence points to deficiencies in quantitative training and application in several areas: (a) methodological reporting problems, (b) researcher misconceptions and inaccuracies, (c) overreliance on traditional methods, and (d) a lack of coverage of modern advances. An argument is made that a culture supportive of quantitative methods is not consistently available to many applied education researchers. Collective quantitative proficiency is defined as a vision for a culture representative of broader support for quantitative methodology (statistics, measurement, and research design).
AikenL. S.WestS. G.MillsapR. E. (2008). Doctoral training in statistics, measurement, and methodology in psychology: Replication and extension of Aiken, West, Sechrest, and Reno’s (1990) survey of PhD programs in North America. American Psychologist, 63, 32–50.
2.
AikenL. S.WestS. G.MillsapR. E. (2009). Improving training in methodology enriches the science of psychology. American Psychologist, 64, 51–52.
3.
AikenL. S.WestS. G.SechrestL.RenoR. R. (1990). Graduate training in statistics, methodology, and measurement in psychology: A survey of PhD programs in North America. American Psychologist, 45, 721–734.
4.
American Association for the Advancement of Science. (1993). Benchmarks for science literacy. New York: Oxford University Press.
5.
American Educational Research Association. (2006). Standards for reporting on empirical social science research in AERA publications. Educational Researcher, 35(6), 33–40.
6.
American Psychological Association. (2010). Publication manual of the American Psychological Association (6th ed.). Washington, DC: Author.
7.
BeauchampK. L.MayR. B. (1964). Replication report: Interpretation of levels of significance by psychological researchers. Psychological Reports, 14, 272.
8.
BeliaS.FidlerF.WilliamsJ.CummingG. (2005). Researchers misunderstand confidence intervals and standard error bars. Psychological Methods, 10, 389–396.
9.
BeretvasS. N.RobinsonD. H. (2004). How are effect sizes and p-values interpreted by professors of education?Research in the Schools, 11(2), 41–50.
10.
BurkhardtH.SchoenfeldA. H. (2003). Improving educational research: Toward a more useful, more influential, and better-funded enterprise. Educational Researcher, 32(9), 3–14.
11.
CapraroR. M.ThompsonB. (2008). The educational researcher defined: What will future researchers be trained to do?Journal of Educational Research, 101, 247–253.
12.
ChanceB. L. (2002). Components of statistical thinking and implications for instruction and assessment. Journal of Statistics Education, 10(3). Retrieved October 28, 2009, from www.amstat.org/publications/jse/v10n3/chance.html
13.
Cochran-SmithM.ZeichnerK. M. (Eds.). (2005). Studying teacher education: The report of the AERA Panel on Research and Teacher Education. Mahwah, NJ: Lawrence Erlbaum.
14.
CohenJ. (1994). The earth is round (p < .05). American Psychologist, 49, 997–1003.
15.
ColeM. (1990). Cognitive development and formal schooling: The evidence from cross-cultural research. In MollL. C. (Ed.), Vygotsky and education: Instructional implications and applications of sociohistorical psychology (pp. 89–110). Cambridge, UK: Cambridge University Press.
16.
CronbachL. J.FurbyL. (1970). How should we measure “change”—Or should we?Psychological Bulletin, 74, 68–80.
17.
CummingG.FinchS. (2001). A primer on the understanding, use, and calculation of confidence intervals that are based on central and noncentral distributions. Educational and Psychological Measurement, 61, 532–574.
18.
CurtisD. A.HarwellM. (1998). Training doctoral students in educational statistics in the United States: A national survey. Journal of Statistics Education, 6, 1–22.
19.
EdgingtonE. S. (1964). A tabulation of inferential statistics used in psychology journals. American Psychologist, 19, 202–203.
20.
EisenhartM.DeHaanR. L. (2005). Doctoral preparation of scientifically based education researchers. Educational Researcher, 34(4), 3–13.
21.
ElliotJ. (1990). Educational research in crisis: Performance indicators and the decline in excellence. British Educational Research Journal, 16, 1–17.
22.
ElmoreP. B.WoehlkeP. L. (1988). Statistical methods employed in American Educational Research Journal, Educational Researcher, and Review of Educational Research from 1978 to 1987. Educational Researcher, 17(9), 19–20.
23.
EmbretsonS. E. (2007). Construct validity: A universal validity system or just another test evaluation procedure?Educational Researcher, 36, 449–455.
24.
ErcikanK.RothW. M. (2006). What good is polarizing research into qualitative and quantitative?Educational Researcher, 35(5), 14–23.
25.
FabrigarL. R.WegenerD. T.MacCallumR. C.StrahanE. J. (1999). Evaluating the use of exploratory factor analysis in psychological research. Psychological Methods, 4, 272–299.
26.
FeuerM. J.TowneL.ShavelsonR. J. (2002). Scientific culture and educational research. Educational Researcher, 31(8), 4–14.
27.
FriedrichJ.BudayE.KerrD., (2000). Statistical training in psychology: A national survey and commentary on undergraduate programs. Teaching of Psychology, 27, 248–257.
28.
GigerenzerG. (1992). Discovery in cognitive psychology: New tools inspire new theories. Science in Context, 5, 329–350.
29.
GoldeC.WalkerG. (Eds.). (2006). Envisioning the future of doctoral education: Preparing stewards of the discipline—Carnegie essays on the doctorate. San Francisco: Jossey-Bass.
30.
GoodwinL. D.GoodwinW. L. (1985a). An analysis of statistical techniques used in the Journal of Educational Psychology, 1979–1983. Educational Psychologist, 20, 13–21.
31.
GoodwinL. D.GoodwinW. L. (1985b). Statistical techniques in AERJ articles, 1979–1983: The preparation of graduate students to read the educational research literature. Educational Researcher, 14(2), 5–11.
32.
GuoF.NitkoA. J. (1996). Graduate programs that prepare educational measurement specialists. Educational Measurement: Issues and Practice, 15(4), 28–31.
33.
HensonR. K. (2006). Effect size measures and meta-analytic thinking in counseling psychology research. The Counseling Psychologist, 34, 601–629.
34.
HensonR. K.CapraroR. M.CapraroM. M. (2004). Reporting practices and use of exploratory factor analyses in educational research journals: Errors and explanation. Research in the Schools, 11(2), 61–72.
35.
HensonR. K.RobertsJ. K. (2006). Use of exploratory factor analysis in published research: Common errors and some comment on improved practice. Educational and Psychological Measurement, 66, 393–416.
36.
HensonR. K.WilliamsC. (2006, April). Doctoral training in research methodology: A national survey of education and related disciplines. Paper presented at the annual meeting of the American Educational Research Association, San Francisco, CA.
37.
HollandP. W. (1986). Statistics and causal inference. Journal of the American Statistics Association, 81, 945–970.
38.
HollandP. W. (1988). Causal inference, path analysis, and recursive structural equations models. In CloggC. (Ed.), Sociological methodology (pp. 449–484). Washington, DC: American Sociological Association.
39.
HollandP. W.RubinD. B. (1983). On Lord’s paradox. In WainerH.MessickS. (Eds.), Principals of modern psychological measurement (pp. 3–25). Hillsdale, NJ: Lawrence Erlbaum.
40.
HsiehP.AceeT.ChungW-H.HsiehY-P.KimH.ThomasG. D.. (2005). Is educational intervention research on the decline?Journal of Educational Psychology, 97, 523–529.
41.
HullD. M.SaxonT. S. (2009). Negotiation of meaning and co-construction of knowledge: An experimental analysis of asynchronous online instruction. Computers and Education, 52, 624–639.
42.
ImbensG.RubinD. B. (1997). Bayesian inference for causal effects in randomized experiments with noncompliance. Annals of Statistics, 25, 305–327.
43.
JohnsonR. B.OnwuegbuzieA. J. (2004). Mixed methods research: A research paradigm whose time has come. Educational Researcher, 33(7), 14–26.
44.
KeselmanH. J.HubertyC. JLixL. M.OlejnikS.CribbieR. A.DonahueB.. (1998). Statistical practices of educational researchers: An analysis of their ANOVA, MANOVA, and ANCOVA analyses. Review of Educational Research, 68, 350–386.
45.
KirkR. E. (1996). Practical significance: A concept whose time has come. Educational and Psychological Measurement, 56, 746–759.
46.
KirkR. E. (2001). Promoting good statistical practices: Some suggestions. Educational and Psychological Measurement, 61, 213–218.
47.
KozulinA. (1996). The concept of activity in Soviet psychology: Vygotsky, his disciples and critics. In DanielsH. (Ed.), An introduction to Vygotsky (pp. 99–122). New York: Sharpe.
48.
KozulinA. (1998). Psychological tools: A sociocultural approach to education. Cambridge, MA: Harvard University Press.
49.
LabareeD. F. (2003). The peculiar problems of preparing educational researchers. Educational Researcher, 32(4), 13–22.
50.
LaveJ. (1991). Situating learning in communities of practice. In ResnickL. B.LevineJ. M.TeasleyS. D. (Eds.), Perspectives on socially shared cognition (pp. 63–83). Washington, DC: American Psychological Association.
51.
LeechN. L.GoodwinL. D. (2008). Building a methodological foundation: Doctoral-level methods courses in colleges of education. Research in the Schools, 15(1), 1–8.
52.
LeontievA. N. (1930). Studies in the cultural development of the child: 3. The development of voluntary attention in the child. Journal of Genetic Psychology, 37, 52–81.
LissitzR. W.SamuelsenK. (2007). A suggested change in terminology and emphasis regarding validity and education. Educational Researcher, 36, 437–448.
55.
LuriaA. R. (1928). The problem of the cultural development of the child. Journal of Genetic Psychology, 37, 52–81.
56.
LuriaA. R. (1932). The nature of human conflicts. London: Liveright.
57.
MaxwellJ. A. (2004). Causal explanation, qualitative research, and scientific inquiry in education. Educational Researcher, 33(2), 3–11.
58.
MessickS. (1989). Validity. In LinnR. L. (Ed.), Educational measurement (3rd ed., pp. 13–104). New York: Macmillan.
59.
MetzM. H. (2001). Intellectual border crossing in graduate education: A report from the field. Educational Researcher, 30(5), 12–18.
60.
MislevyR. J. (2007). Validity by design. Educational Researcher, 36, 463–469.
61.
MittagK. C.ThompsonB. (2000). A national survey of AERA members’ perceptions of statistical significance tests and other statistical issues. Educational Researcher, 29(4), 14–20.
62.
MurtonenM.LehtinenE. (2003). Difficulties experienced by education and sociology students in quantitative methods courses. Studies in Higher Education, 28, 171–185.
63.
National Research Council. (2002). Scientific research in education. Washington, DC: National Academy Press.
64.
National Research Council. (2004). Advancing scientific research in education. Washington, DC: National Academy Press.
65.
NelsonN.RosenthalR.RosnowR. L. (1986). Interpretation of significance levels and effect sizes by psychological researchers. American Psychologist, 41, 1299–1301.
66.
O’ConnorB. P. (2000). SPSS and SAS programs for determining the number of components using parallel analysis and Velicer’s MAP test. Behavior Research, Instruments, and Computers, 32, 396–402.
67.
OlejnikS.AlginaJ. (2000). Measures of effect size for comparative studies: Applications, interpretations, and limitations. Contemporary Educational Psychology, 25, 241–286.
68.
OlsonD. R. (2004). The triumph of hope over experience in the search for “what works”: A response to Slavin. Educational Researcher, 33(1), 24–26.
69.
OnwuegbuzieA. J. (2000). Statistics anxiety and the role of self-perceptions. Journal of Educational Research, 93, 323–335.
70.
OnwuegbuzieA. J. (2004). Academic procrastination and statistics anxiety. Assessment and Evaluation in Higher Education, 29, 3–19.
71.
OnwuegbuzieA. J.DaleyC. E. (1999). Perfectionism and statistics anxiety. Personality and Individual Differences, 26, 287–303.
72.
OnwuegbuzieA. J.DanielL. G. (2005). Editorial: Evidence-based guidelines for publishing articles in Research in the Schools and beyond. Research in the Schools, 12(2), 1–11.
73.
PageR. N. (2001). Reshaping graduate preparation in educational research methods: One school’s experience. Educational Researcher, 30(5), 19–25.
74.
PalinscarA.BrownA. L. (1984). Reciprocal teaching of comprehension—Fostering and monitoring activities. Cognition and Instruction, 1, 117–175.
75.
PallasA. M. (2001). Preparing education doctoral students for epistemological diversity. Educational Researcher, 30(5), 6–11.
76.
PanofskyC. P. (2003). The relations of learning and student social class: Toward re-“socializing” sociocultural learning theory. In KozulinA.GindisB.AgeyevV. S.MillerS. M. (Eds.), Vygotsky’s educational theory in cultural context (pp. 411–431). New York: Cambridge University Press.
77.
PearlJ. (1996). Structural and probabilistic causality. In ShanksD. R.HolyoakK. J.MedinD. L. (Eds.), The psychology of learning and motivation (Vol. 34, pp. 393–435). San Diego, CA: Academic Press.
Peterson’s Graduate Schools in the U.S. (2006). Lawrenceville, NJ: Thomson Peterson’s.
80.
RatnerC. (2000). Outline of a coherent, comprehensive concept of culture. Cross-Cultural Psychology Bulletin, 34(1–2), 5–11.
81.
RaudenbushS. W. (2005). Learning from attempts to improve schooling: The contribution of methodological diversity. Educational Researcher, 34(5), 25–31.
82.
RaudenbushS. W. (2006). How shall we study the cause and effects of classroom teaching?Distinguished Contributions to Education Research Award (2006) Lecture, presented April 12, 2007. Retrieved on November 2, 2009, from http://www.softconference.com/Media/WMP/270409/s41.htm
83.
RobinsonD. H.LevinJ. R.ThomasG. D.PituchK. A.VaughnS. (2007). The incidence of “causal” statements in teaching-and-learning research journals. American Educational Research Journal, 44, 400–413.
84.
RogoffB. (1995). Observing sociocultural activity on three planes. In WertschJ.Del RioP.AlvarezA. (Eds.), Sociocultural studies of the mind (pp. 139–164). New York: Cambridge University Press.
85.
RosenbaumP. R. (2002). Observational studies (2nd ed.). New York: Springer.
86.
RosenthalR.GaitoJ. (1963). The interpretation of level of significance by psychological researchers. Journal of Psychology, 55, 33–38.
87.
RubinD. B. (1974). Estimating causal effects of treatments in randomized and nonrandomized studies. Journal of Educational Psychology, 66, 688–701.
88.
RubinD. B. (1978). Bayesian inference for causal effects: The role of randomization. Annals of Statistics, 6, 34–58.
89.
RubinD. B. (1980). Discussion of “Randomization analysis of experimental data in the Fisher randomization test” by Basu. Journal of the American Statistical Association, 75, 591–593.
90.
SchneiderB.CarnoyM.KilpatrickJ.SchmidtW. H.ShavelsonR. J. (2007). Estimating causal effects using experimental and observational designs (report from the Governing Board of the American Educational Research Association Grants Program). Washington, DC: American Educational Research Association.
91.
ScribnerS. (1997). Mind and social practice. New York: Cambridge University Press.
92.
ScribnerS.ColeM. (1981). Psychology of literacy. Cambridge, MA: Harvard University Press.
93.
ShulmanL. S.GoldeC. M.BueschelA. C.GarabedianK. J. (2006). Reclaiming education’s doctorates: A critique and a proposal. Educational Researcher, 35(3), 25–32.
94.
SiegelH. (2006). Epistemological diversity and education research: Much ado about nothing much?Educational Researcher, 35(2), 3–12.
95.
SlavinR. E. (2002). Evidence-based education policies: Transforming educational practice and research. Educational Researcher, 31(7), 15–21.
96.
SlavinR. E. (2004). Education research can and must address “what works” questions. Educational Researcher, 33(1), 27–28.
97.
TashakkoriA.CreswellJ. W. (2008). Editorial: Envisioning the future stewards of the social-behavioral research enterprise. Journal of Mixed Methods Research, 2, 291.
98.
ThompsonB. (1999). Statistical significance tests, effect size reporting and the vain pursuit of pseudo-objectivity. Theory and Psychology, 9, 191–196.
99.
ThompsonB. (2002). What future quantitative social science research could look like: Confidence intervals for effect sizes. Educational Researcher, 31(3), 25–32.
100.
ThorndikeR. M. (2005). Measurement and evaluation in psychology and education (7th ed.). Upper Saddle River, NJ: Pearson.
101.
Vacha-HaaseT.HensonR. K.CarusoJ. (2002). Reliability generalization: Moving toward improved understanding and use of score reliability. Educational and Psychological Measurement, 62, 562–569.
102.
VygotskyL. S. (1978). Mind in society: Development of higher psychological processes (ColeM.John-SteinerV.ScribnerS.SoubermanE., Eds.). Cambridge, MA: Harvard University Press.
103.
VygotskyL. S. (1986). Thought and language (KozulinA., Ed.). Cambridge, MA: Massachusetts Institute of Technology Press.
104.
WertschJ. V. (1990). The voice of rationality in a sociocultural approach to mind. In MollL. C. (Ed.) Vygotsky and education: Instructional implications and applications of sociohistorical psychology (pp. 111–126). Cambridge, UK: Cambridge University Press.
105.
WertschJ. V. (1991). Voices of the mind: A sociocultural approach to mediated action. Boston: Harvard University Press.
106.
WestC. K.CarmodyC.StallingsW. M. (1983). The quality of research articles in the Journal of Educational Research, 1970 and 1980. Journal of Educational Research, 77, 70–76.
107.
WilkinsonL., & APA Task Force on Statistical Inference. (1999). Statistical methods in psychology journals: Guidelines and explanation. American Psychologist, 54, 594–604.
108.
YoungL. J. (2001). Border crossings and other journeys: Re-envisioning the doctoral preparation of education researchers. Educational Researcher, 30(5), 3–5.
109.
ZientekL. R.CapraroM. M.CapraroR. M. (2008). Reporting practices in quantitative teacher education research: One look at the evidence cited in the AERA panel report. Educational Researcher, 37, 208–216.
110.
ZuckermanM.HodginsH. S.ZuckermanA.RosenthalR. (1993). Contemporary issues in the analysis of data: A survey of 551 psychologists. Psychological Science, 4, 49–53.
111.
ZwickW. R.VelicerW. F. (1986). Factors influencing five rules for determining the number of components to retain. Psychological Bulletin, 99, 432–442.
