Sage Journals: Discover world-class research

Abstract

Toward the goal of more rapid knowledge accumulation via better meta-analyses, this article explores statistical approaches intended to increase the precision and comparability of effect sizes from education research. The featured estimate of the proposed approach is a standardized mean difference effect size whose numerator is a mean difference that has been adjusted for baseline differences in the outcome measure, at a minimum, and whose denominator is the total variance. The article describes the utility and efficiency of covariate adjustment through baseline measures and the need to standardize effects on a total variance that accounts for variation at multiple levels. As computation of the total variance can be complex in multilevel studies, a shiny application is provided to assist with computation of the total variance and subsequent effect size. Examples are provided for how to interpret and input the required calculator inputs.

Keywords

educational policy effect size evaluation experimental design hierarchical linear modeling meta-analysis program evaluation quasi-experimental analysis regression analyses validity/reliability

Get full access to this article

View all access options for this article.

References

Ades

A. E.

Dias

Mayo-Wilson

Kounali

(2015). Simultaneous synthesis of treatment effects and mapping to a common scale: An alternative to standardisation. Research Synthesis Methods, 6(1), 96–107. https://doi.org/10.1002/jrsm.1130

American Educational Research Association. (2006). Standards for reporting on empirical social science research in AERA publications. Educational Researcher, 35(6), 33–40. https://doi.org/10.3102/0013189X035006033

Appelbaum

Cooper

Kline

R. B.

Mayo-Wilson

Nezu

A. M.

Rao

S. M.

(2018). Journal article reporting standards for quantitative research in psychology: The APA Publications and Communications Board Task Force Report. American Psychologist, 73(1), 3–25. https://doi.org/10.1037/amp0000191

Baird

M. D.

Pane

J. F.

(2019). Translating standardized effects of education programs into more interpretable metrics. Educational Researcher, 48(4), 217–228. https://doi.org/10.3102/0013189X19848729

Bond

C. F.

Jr. Witala

W. L.

Richard

F. D.

(2003). Meta-analysis of raw mean differences. Psychological Methods, 8(4), 406–418. https://doi.org/10.1037/1082-989X.8.4.406

Borenstein

Hedges

L. V. H.

(2019). Effect sizes for meta-analysis. In Cooper

Hedges

L. V.

Valentine

J. C.

(Eds.), The handbook of research synthesis and meta-analysis (3rd ed., pp. 207–244). Russell Sage Foundation. https://doi.org/10.7758/9781610448864.14

Cohen

(1970). Approximate power and sample size determination for common one-sample and two-sample hypothesis tests. Educational and Psychological Measurement, 30(4), 811–831. https://doi.org/10.1177/001316447003000404

Doabler

C. T.

Clarke

Kosty

D. B.

Baker

S. K.

Smolkowski

Fien

(2016). Effects of a core kindergarten mathematics curriculum on the mathematics achievement of Spanish-speaking English learners. School Psychology Review, 45(3), 343–361. https://doi.org/10.17105/SPR45-3.343-361

Dong

Reinke

W. M.

Herman

K. C.

Bradshaw

C. P.

Murray

D. W.

(2016). Meaningful effect sizes, intraclass correlations, and proportions of variance explained by covariates for planning two- and three-level cluster randomized trials of social and behavioral outcomes. Evaluation Review, 40(4), 334–377. https://doi.org/10.1177/0193841X16671283

10.

Fuhrman

(2001). The policy influence of education R&D centers. Testimony to the U.S. House Committee on Education and the Workforce, 107th Congress, 1st Session.

11.

Grissom

R. J.

Kim

J. J.

(2012). Effect sizes for research: Univariate and multivariate applications. Routledge. https://doi.org/10.4324/9780203803233

12.

Hedberg

E. C.

Hedges

L. V.

(2014). Reference values of within-district intraclass correlations of academic achievement by district characteristics: Results from a meta-analysis of district-specific values. Evaluation Review, 38(6), 546–582. https://doi.org/10.1177/0193841X14554212

13.

Hedges

L. V.

(2007). Effect sizes in cluster-randomized designs. Journal of Educational and Behavioral Statistics, 32(4), 341–370. https://doi.org/10.3102/1076998606298043

14.

Hedges

L. V.

(2011). Effect sizes in three-level cluster-randomized experiments. Journal of Educational and Behavioral Statistics, 36(3), 346–380. https://doi.org/10.3102/1076998610376617

15.

Hedges

L. V.

Hedberg

E. C.

(2007). Intraclass correlation values for planning group-randomized trials in education. Educational Evaluation and Policy Analysis, 29(1), 60–87. https://doi.org/10.3102/0162373707299706

16.

Hedges

L. V.

Hedberg

E. C.

(2013). Intraclass correlations and covariate outcome correlations for planning two- and three-level cluster-randomized experiments in education. Evaluation Review, 37(6), 445–489. https://doi.org/10.1177/0193841X14529126

17.

Hedges

L. V.

Hedberg

E. C.

(2021). Online intraclass correlation database. http://stateva.ci.northwestern.edu/

18.

Institute of Education Sciences. (2021a). Institute of education sciences. https://ies.ed.gov/

19.

Institute of Education Sciences. (2021b). What works clearinghouse. https://ies.ed.gov/ncee/wwc/

20.

Institute of Education Sciences. (2021c). What works clearinghouse procedures handbook v. 4.1. https://ies.ed.gov/ncee/wwc/Docs/referenceresources/WWC-Procedures-Handbook-v4-1-508.pdf

21.

Institute of Education Sciences. (2021d). Making common measures more common. https://ies.ed.gov/director/remarks/5-05-2020.asp

22.

Institute of Education Sciences. (2021e). Reporting guide for study authors: Group design studies. https://ies.ed.gov/ncee/wwc/Docs/ReferenceResources/wwc_gd_guide_022218.pdf

23.

Institute of Education Sciences. (2021f). Supplement to the What Works Clearinghouse procedures handbook, Version 4.1. https://ies.ed.gov/ncee/wwc/Docs/referenceresources/WWC-41-Supplement-508_09212020.pdf

24.

Lipsey

M. W.

Wilson

D. B.

(2001). Practical meta-analysis. Sage.

25.

Maassen

van Assen

M. A. L. M.

Nuijten

M. B.

Olsson-Collentine

Wicherts

J. M.

(2020). Reproducibility of individual effect sizes in meta-analyses in psychology. PLoS One, 15(5), e0233107. https://doi.org/10.1371/journal.pone.0233107

26.

Methods Group of the Campbell Collaboration. (2019). Methodological expectations of Campbell Collaboration intervention reviews: Reporting standards (Campbell Policies and Guidelines Series No. 4). https://doi.org/10.4073/cpg.2016.4

27.

National Research Council. (1992). Research and education reform: Roles for the Office of Educational Research and Improvement. National Academies Press.

28.

National Research Council. (2002). Scientific research in education. National Academies Press. https://doi.org/10.17226/10236

29.

Polanin

J. R.

Snilstveit

(2016). Converting between effect sizes. Campbell Systematic Reviews, 12(1), 1–13. https://doi.org/10.4073/cmpn.2016.3

30.

Reich

S. M.

Yau

J. C.

Muskat

Uvalle

Cannata

(2019). Digital or print? A comparison of preschoolers’ comprehension, vocabulary, and engagement from a print book and an e-Book. AERA Open, 5(3). https://doi.org/10.1177/2332858419878389

31.

Schochet

P. A.

(2005). Statistical power for random assignment evaluations of education programs. Mathematica Policy Research.

32.

Spybrook

Westine

C. D.

Taylor

J. A.

(2016). Design parameters for impact research in science education: A multistate analysis. AERA Open, 2(1), 2332858415625975. https://doi.org/10.1177/2332858415625975

33.

SREE. (2020). Registry of efficacy and effectiveness studies. https://sreereg.icpsr.umich.edu/sreereg/

34.

Sun

R. W.

Cheung

S. F.

(2020). The influence of nonnormality from primary studies on the standardized mean difference in meta-analysis. Behavior Research Methods, 52, 1552-1567. https://doi.org/10.3758/s13428-019-01334-x

35.

Tipton

Pustejovsky

J. E.

Ahmadi

(2019). Current practices in meta-regression in psychology, education, and medicine. Research Synthesis Methods, 10(2), 180-194. https://doi.org/10.1002/jrsm.1339

36.

Wilson

(2020). Practical meta-analysis effect size calculator. https://campbellcollaboration.org/escalc/html/EffectSizeCalculator-Home.php

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.20 MB

Promoting Knowledge Accumulation About Intervention Effects: Exploring Strategies for Standardizing Statistical Approaches and Effect Size Reporting

Abstract

Keywords

Get full access to this article

References

Supplementary Material