Intraclass Correlations for Evaluating the Effects of Teacher Empowerment Programs on Student Educational Outcomes

Abstract

Programs that improve teaching effectiveness represent a core strategy to improve student educational outcomes and close student achievement gaps. This article compiles empirical values of intraclass correlations for designing effective and efficient experimental studies evaluating the effects of these programs. The Early Childhood Longitudinal Study, Kindergarten Class of 1998–1999 (ECLS-K) data are used to estimate empirical design parameter values in the setting of students nested within teachers nested within schools. The results show that intraclass correlations at the school level are relatively stable, and those at the teacher level increase along with grade level. We illustrate the utility of the results by demonstrating how to leverage this information to design more efficient and effective experimental studies.

Keywords

evaluation professional development program evaluation school/teacher effectiveness hierarchical linear modeling experimental design research methodology

Get full access to this article

View all access options for this article.

References

Babinski

L. M.

Amendum

S. J.

Knotek

S. E.

Sánchez

Malone

(2018). Improving young English learners’ language and literacy skills through teacher professional development: A randomized controlled trial. American Educational Research Journal, 55(1), 117–143. https://doi.org/10.3102%2F0002831217732335

Bates

Mächler

Bolker

Walker

(2015). Fitting linear mixed-effects models using lme4. Journal of Statistical Software, 67(1), 1–48. https://doi.org/10.18637/jss.v067.i01

Bloom

H. S.

(2005). Randomizing groups to evaluate place-based programs. In Bloom

H. S.

(Ed.), Learning more from social experiments: Evolving analytic approaches (pp. 115–172). Russell Sage Foundation.

Bloom

H. S.

Richburg-Hayes

Black

A. R.

(2007). Using covariates to improve precision for studies that randomize schools to evaluate educational interventions. Educational Evaluation and Policy Analysis, 29(1), 30–59. https://doi.org/10.3102/0162373707299550

Chan

Hedges

L. V.

Hedberg

E. C.

(2020). The effects of downstream clustering in longitudinal studies. The Journal of Experimental Education, 90(3), 731–759. https://doi.org/10.1080/00220973.2020.1783501

Didion

Toste

J. R.

Filderman

M. J.

(2020). Teacher professional development and student reading achievement: A meta-analytic review of the effects. Journal of Research on Educational Effectiveness, 13(1), 29–66. https://doi.org/10.1080/19345747.2019.1670884

Dong

Reinke

W. M.

Herman

K. C.

Bradshaw

C. P.

Murray

D. W.

(2016). Meaningful effect sizes, intraclass correlations, and proportions of variance explained by covariates for planning two-and three-level cluster randomized trials of social and behavioral outcomes. Evaluation Review, 40(4), 334–377. https://doi.org/10.1177/0193841X16671283

Fahle

E. M.

Reardon

S. F.

(2018). How much do test scores vary among school districts? New estimates using population data, 2009–2015. Educational Researcher, 47(4), 221–234. https://doi.org/10.3102/0013189X18759524

Griffin

C. C.

Dana

N. F.

Pape

S. J.

Algina

Bae

Prosser

S. K.

League

M. B.

(2018). Prime online: Exploring teacher professional development for creating inclusive elementary mathematics classrooms. Teacher Education and Special Education, 41(2), 121–139. https://doi.org/10.1177/0888406417740702

10.

Hedberg

E. C.

(2016). Academic and behavioral design parameters for cluster randomized trials in kindergarten: An analysis of the Early Childhood Longitudinal Study 2011 Kindergarten Cohort (ECLS-K 2011). Evaluation Review, 40(4), 279–313. https://doi.org/10.1177%2F0193841X16655657

11.

Hedges

L. V.

Borenstein

(2014). Conditional optimal design in three-and four-level experiments. Journal of Educational and Behavioral Statistics, 39(4), 257–281. https://doi.org/10.3102/1076998614534897

12.

Hedges

L. V.

Hedberg

E. C.

(2007). Intraclass correlation values for planning group-randomized trials in education. Educational Evaluation and Policy Analysis, 29(1), 60–87. https://doi.org/10.3102/0162373707299706

13.

Hedges

L. V.

Hedberg

E. C.

(2013). Intraclass correlations and covariate outcome correlations for planning two-and three-level cluster-randomized experiments in education. Evaluation Review, 37(6), 445–489. https://doi.org/10.1177/0193841X14529126

14.

Hill

H. C.

Lynch

Gonzalez

K. E.

Pollard

(2020). Professional development that improves STEM outcomes. Phi Delta Kappan, 101(5), 50–56. https://doi.org/10.1177/0031721720903829

15.

Jacob

Goddard

Kim

Miller

Goddard

(2015). Exploring the causal impact of the McREL Balanced Leadership Program on leadership, principal efficacy, instructional climate, educator turnover, and student achievement. Educational Evaluation and Policy Analysis, 37(3), 314–332. https://doi.org/10.3102/0162373714549620

16.

Jacob

Zhu

Bloom

(2010). New empirical evidence for the design of group randomized trials in education. Journal of Research on Educational Effectiveness, 3(2), 157–198. https://doi.org/10.1080/19345741003592428

17.

Jayanthi

Gersten

Taylor

M. J.

Smolkowski

Dimino

(2017). Impact of the developing mathematical ideas professional development program on grade 4 students’ and teachers’ understanding of fractions (REL 2017–256). U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance, Regional Educational Laboratory Southeast. http://ies.ed.gov/ncee/edlabs

18.

Jennings

P. A.

Brown

J. L.

Frank

J. L.

Doyle

Davis

Rasheed

DeWeese

DeMauro

A. A.

Cham

Greenberg

M. T.

(2017). Impacts of the CARE for teachers program on teachers’ social and emotional competence and classroom interactions. Journal of Educational Psychology, 109(7), 1010–1028. https://doi.org/10.1037/edu0000187

19.

Kalogrides

Loeb

Béteille

(2013). Systematic sorting: Teacher characteristics and class assignments. Sociology of Education, 86(2), 103–123. https://doi.org/10.1177/0038040712456555

20.

Kelcey

Hill

H. C.

Chin

M. J.

(2019). Teacher mathematical knowledge, instructional quality, and student outcomes: A multilevel quantile mediation analysis. School Effectiveness and School Improvement, 30(4), 398–431. https://doi.org/10.1080/09243453.2019.1570944

21.

Kelcey

Phelps

(2013). Considerations for designing group randomized trials of professional development with teacher knowledge outcomes. Educational Evaluation and Policy Analysis, 35(3), 370–390. https://doi.org/10.3102/0162373713482766

22.

Kelcey

Shen

(2016). Multilevel design of school effectiveness studies in sub-Saharan Africa. School Effectiveness and School Improvement, 27(4), 492–510. https://doi.org/10.1080/09243453.2016.1168855

23.

Kelcey

Shen

(2017a). Experimental evaluation of school-based HIV programs in sub-Saharan Africa. Studies in Educational Evaluation, 55, 27–34. https://doi.org/10.1016/j.stueduc.2017.04.004

24.

Kelcey

Shen

(2017b). Planning community-based assessments of HIV educational intervention programs in sub-Saharan Africa. Health Education Research, 32(4), 353–363. https://doi.org/10.1093/her/cyx046

25.

Kelcey

Shen

Spybrook

(2016). Intraclass correlation coefficients for designing cluster-randomized trials in sub-Saharan Africa education. Evaluation Review, 40(6), 500–525. https://doi.org/10.1177/0193841X16660246

26.

Kivlighan

D. M.

III Aloe

A. M.

Adams

M. C.

Garrison

Y. L.

Obrecht

Y. C. S.

Kim

J. Y. C.

Hooley

I. W.

Chan

Deng

(2020). Does the group in group psychotherapy matter? A meta-analysis of the intraclass correlation coefficient in group treatment research. Journal of Consulting and Clinical Psychology, 88(4), Article 322. https://psycnet.apa.org/doi/10.1037/ccp0000474

27.

Konstantopoulos

(2008a). The power of the test for treatment effects in three-level block randomized designs. Journal of Research on Educational Effectiveness, 1(4), 265–288. https://doi.org/10.1080/19345740802328216

28.

Konstantopoulos

(2008b). The power of the test for treatment effects in three-level cluster randomized designs. Journal of Research on Educational Effectiveness, 1(1), 66–88. https://doi.org/10.1080/19345740701692522

29.

Konstantopoulos

(2009). Incorporating cost in power analysis for three-level cluster-randomized designs. Evaluation Review, 33(4), 335–357. https://doi.org/10.1177/0193841X09337991

30.

Konstantopoulos

(2011). Optimal sampling of units in three-level cluster randomized designs: An ANCOVA framework. Educational and Psychological Measurement, 71(5), 798–813. https://doi.org/10.1177/0013164410397186

31.

Levin

H. M.

McEwan

P. J.

Belfield

Bowden

A. B.

Shand

(2017). Economic evaluation in education: Cost-effectiveness and benefit-cost analysis. SAGE.

32.

Liu

(2003). Statistical power and optimum sample allocation ratio for treatment and control having unequal costs per unit of randomization. Journal of Educational and Behavioral Statistics, 28(3), 231–248. https://doi.org/10.3102/10769986028003231

33.

Lynch

Hill

H. C.

Gonzalez

K. E.

Pollard

(2019). Strengthening the research base that informs STEM instructional improvement efforts: A meta-analysis. Educational Evaluation and Policy Analysis, 41(3), 260–293. https://doi.org/10.3102/0162373719849044

34.

McCoach

D. B.

Gubbins

E. J.

Foreman

Rubenstein

L. D.

Rambo-Hernandez

K. E.

(2014). Evaluating the efficacy of using predifferentiated and enriched mathematics curricula for grade 3 students: A multisite cluster-randomized trial. Gifted Child Quarterly, 58(4), 272–286. https://doi.org/10.1177/0016986214547631

35.

Moerbeek

(2004). The consequence of ignoring a level of nesting in multilevel analysis. Multivariate Behavioral Research, 39(1), 129–149. https://doi.org/10.1207/s15327906mbr3901_5

36.

Moerbeek

(2005). Randomization of clusters versus randomization of persons within clusters: Which is preferable? The American Statistician, 59(2), 173–179. https://www.jstor.org/stable/27643621

37.

Nye

Hedges

L. V.

Konstantopoulos

(2004). How large are teacher effects? Educational Evaluation and Policy Analysis, 26, 237–257. https://doi.org/10.3102/01623737026003237

38.

Opdenakker

M. C.

Van Damme

(2000). The importance of identifying levels in multilevel analysis: An illustration of the effects of ignoring the top or intermediate levels in school effectiveness research. School Effectiveness and School Improvement, 11(1), 103–130.

39.

Philipsen

Tondeur

Roblin

N. P.

Vanslambrouck

Zhu

(2019). Improving teacher professional development for online and blended learning: A systematic meta-aggregative review. Educational Technology Research and Development, 67(5), 1145–1174. https://doi.org/10.1007/s11423-019-09645-8

40.

Raudenbush

S. W.

(1997). Statistical analysis and optimal design for cluster randomized trials. Psychological Methods, 2(2), 173–185. https://doi.org/10.1037/1082-989X.2.2.173

41.

Raudenbush

S. W.

Bryk

A. S.

(2002). Hierarchical linear models: Applications and data analysis methods (2nd ed.). SAGE.

42.

Raudenbush

S. W.

Liu

(2000). Statistical power and optimal design for multisite randomized trials. Psychological Methods, 5(2), 199–213. https://doi.org/10.1037/1082-989X.5.2.199

43.

Raudenbush

S. W.

Martinez

Spybrook

(2007). Strategies for improving precision in group-randomized experiments. Educational Evaluation and Policy Analysis, 29(1), 5–29. https://doi.org/10.3102/0162373707299460

44.

Shen

Kelcey

(2020). Optimal sample allocation under unequal costs in cluster-randomized trials. Journal of Educational and Behavioral Statistics, 45(4), 446–474. https://doi.org/10.3102/1076998620912418

45.

Shen

Kelcey

(2022a). odr: Optimal design and statistical power for multilevel experiments (Version 1.3.1) [Software]. https://cran.r-project.org/web/packages/odr.

46.

Shen

Kelcey

(2022b). Optimal sample allocation in multisite randomized trials. The Journal of Experimental Education, 90(3), 693–711. https://doi.org/10.1080/00220973.2020.1830361

47.

Shen

Kelcey

(2022c). Optimal sampling ratios in three-level multisite experiments. Journal of Research on Educational Effectiveness, 15(1), 130–150. https://doi.org/10.1080/19345747.2021.1953200

48.

Springer

M. G.

Ballou

Hamilton

V. N.

Lockwood

J. R.

McCaffrey

D. F.

Pepper

Stecher

B. M.

(2011). Teacher pay for performance: Experimental evidence from the Project on Incentives in Teaching (POINT). Society for Research on Educational Effectiveness. https://files.eric.ed.gov/fulltext/ED518378.pdf

49.

Spybrook

Shi

Kelcey

(2016). Progress in the past decade: An examination of the precision of cluster randomized trials funded by the US Institute of Education Sciences. International Journal of Research & Method in Education, 39(3), 255–267. https://doi.org/10.1080/1743727X.2016.1150454

50.

Tourangeau

Nord

C. L. T.

Pollack

Atkins-Burnett

(2006). Early childhood longitudinal study, kindergarten class of 1998–99 (ECLS-K), combined user’s manual for the ECLS-K fifth-grade data files and electronic codebooks (NCES 2006–032). U.S. Department of Education. National Center for Education Statistics. https://nces.ed.gov/pubs2006/2006032.pdf

51.

Van den Noortgate

Opdenakker

M. C.

Onghena

. (2005). The effects of ignoring a level in multilevel analysis. School Effectiveness and School Improvement, 16, 281–230. https://doi.org/10.1080/09243450500114850

52.

Westine

C. D.

Spybrook

Taylor

J. A.

(2013). An empirical investigation of variance design parameters for planning cluster-randomized trials of science achievement. Evaluation Review, 37(6), 490–519. https://doi.org/10.1177/0193841X14531584

53.

Westine

C. D.

Unlu

Taylor

Spybrook

Zhang

Anderson

(2020). Design parameter values for impact evaluations of science and mathematics interventions involving teacher outcomes. Journal of Research on Educational Effectiveness, 13(4), 816–839. https://doi.org/10.1080/19345747.2020.1821849

54.

Nichols

(2010). New estimates of design parameters for clustered randomization studies: Findings from North Carolina and Florida (Working Paper 43). National Center for Analysis of Longitudinal Data in Education Research.

55.

Zhang

Spybrook

Unlu

(2020). Examining design and statistical power for planning cluster randomized trials aimed at improving student science achievement and science teacher outcomes. AERA Open, 6(3), 1–12. https://doi.org/10.1177%2F2332858420939526

56.

Zhu

Jacob

Bloom

(2012). Designing and analyzing studies that randomize schools to estimate intervention effects on student academic outcomes without classroom-level information. Educational Evaluation and Policy Analysis, 34(1), 45–68. https://doi.org/10.3102/0162373711423786