Toward a System of Evidence for All: Current Practices and Future Opportunities in 37 Randomized Trials

Abstract

As a result of the evidence-based decision-making movement, the number of randomized trials evaluating educational programs and curricula has increased dramatically over the past 20 years. Policy makers and practitioners are encouraged to use the results of these trials to inform their decision making in schools and school districts. At the same time, however, little is known about the schools taking part in these randomized trials, both regarding how and why they were recruited and how they compare to populations in need of research. In this article, we report on a study of 37 cluster randomized trials funded by the Institute of Education Sciences between 2011 and 2015. Principal investigators of these grants were interviewed regarding the recruitment process and practices. Additionally, data on the schools included in 34 of these studies were analyzed to determine the general demographics of schools included in funded research, as well as how these samples compare to important policy relevant populations. We show that the types of schools included in research differ in a variety of ways from these populations. Large schools from large school districts in urban areas were overrepresented, whereas schools from small school districts in rural areas and towns are underrepresented. The article concludes with a discussion of how recruitment practices might be improved in order to meet the goals of the evidence-based decision-making movement.

Keywords

decision making descriptive analysis educational policy evaluation mixed methods program evaluation research methodology research utilization survey research

Get full access to this article

View all access options for this article.

References

Chhin

C. S.

Taylor

K. A.

Wei

W. S.

(2018). Supporting a culture of replication: An examination of education and special education research grants funded by the Institute of Education Sciences. Educational Researcher, 47(9), 594–605.

Cook

T. D.

(2002). Randomized experiments in educational policy research: A critical examination of the reasons the educational evaluation community has offered for not doing them. Educational Evaluation and Policy Analysis, 24(3), 175–199.

Curtin

Presser

Singer

(2005). Changes in telephone survey nonresponse over the past quarter century. Public Opinion Quarterly, 69(1), 87–98.

Dong

Maynard

(2013). PowerUp!: A tool for calculating minimum detectable effect sizes and minimum required sample sizes for experimental and quasi-experimental design studies. Journal of Research on Educational Effectiveness, 6(1), 24–67. Retrieved from https://doi.org/10.1080/19345747.2012.673143.

Institute of Education Sciences. (2004). Request for applications: Cognition and student learning research grants (NCER-05-07).

Institute of Education Sciences. (2016). Request for applications: Education research grants (CFDA Number: 84.305A).

Lortie-Forgues

Inglis

(2019). Rigorous large-scale educational RCTs are often uninformative: Should we be concerned? Educational Researcher, 48(3), 158–166.

Raudenbush

S. W.

Martinez

Spybrook

(2007). Strategies for improving precision in group-randomized experiments. Educational Evaluation and Policy Analysis, 29(1), 5–29. Retrieved from https://doi.org/10.3102/0162373707299460

Saldana

(2016). The coding manual for qualitative researchers. SAGE Publications.

10.

Shadish

Cook

T. D.

Campbell

D. T.

(2002). Experimental and quasi-experimental designs for generalized causal inference. Houghton Mifflin.

11.

Singer

(2002). The use of incentives to reduce nonresponse in household surveys. Survey Nonresponse, 51, 163–177.

12.

Spybrook

(2008). Are power analyses reported with adequate detail? Evidence from the first wave of group randomized trials funded by the Institute of Education Sciences. Journal of Research on Educational Effectiveness, 1(3), 215–235.

13.

Spybrook

Kelcey

Dong

(2016). Power for detecting treatment by moderator effects in two-and three-level cluster randomized trials. Journal of Educational and Behavioral Statistics, 41(6), 605–627.

14.

Spybrook

Lininger

Cullen

(2013). From planning to implementation: An examination of changes in the research design, sample size, and statistical power of group randomized trials launched by the Institute of Education Sciences. Journal of Research on Educational Effectiveness, 6(4), 396–420.

15.

Spybrook

Raudenbush

S. W.

(2009). An examination of the precision and technical accuracy of the first wave of group-randomized trials funded by the Institute of Education Sciences. Educational Evaluation and Policy Analysis, 31(3), 298–318.

16.

Spybrook

Shi

Kelcey

(2016). Progress in the past decade: An examination of the precision of cluster randomized trials funded by the US Institute of Education Sciences. International Journal of Research & Method in Education, 39(3), 255–267.

17.

Spybrook

Zhang

Kelcey

Dong

(2020). Learning from cluster randomized trials in education: An assessment of the capacity of studies to determine what works, for whom, and under what conditions. Educational Evaluation and Policy Analysis, 0162373720929018.

18.

Stuart

E. A.

Bell

S. H.

Ebnesajjad

Olsen

R. B.

Orr

L. L.

(2017). Characteristics of school districts that participate in rigorous national educational evaluations. Journal of Research on Educational Effectiveness, 10(1), 168–206.

19.

Stuart

E. A.

Cole

S. R.

Bradshaw

C. P.

Leaf

P. J.

(2011). The use of propensity scores to assess the generalizability of results from randomized trials. Journal of the Royal Statistical Society: Series A (Statistics in Society), 174(2), 369–386.

20.

Tipton

(2013). Improving generalizations from experiments using propensity score subclassification: Assumptions, properties, and contexts. Journal of Educational and Behavioral Statistics, 38(3), 239–266.

21.

Tipton

(2014). How generalizable is your experiment? An index for comparing experimental samples and populations. Journal of Educational and Behavioral Statistics, 39(6), 478–501.

22.

Tipton

Fellers

Caverly

Vaden-Kiernan

Borman

Sullivan

Ruiz

Castilla

(2016). Site selection in experiments: An assessment of site recruitment and generalizability in two scale-up studies. Journal of Research on Educational Effectiveness, 9(Suppl. 1), 209–228.

23.

Tipton

Hallberg

Hedges

L. V.

Chan

(2017). Implications of small samples for generalization: Adjustments and rules of thumb. Evaluation Review, 41(5), 472–505.

24.

Tipton

Miller

(2015). The generalizer. Retrieved from www.thegeneralizer.org

25.

Tipton

Olsen

R. B.

(2018). A review of statistical methods for generalizing from evaluations of educational interventions. Educational Researcher, 0013189X18781522.

26.

U.S. Department of Education. (2016). Non-regulatory guidance: Using evidence to strengthen education investments. Retrieved from https://ed.gov/policy/elsec/leg/essa/guidanceuseseinvestment.pdf

27.

U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics. (2016). Common Core of Data.

28.

Wagner

J. R.

(2008). Adaptive survey design to reduce nonresponse bias. Doctoral diss., University of Michigan.

29.

Wickham

(2016). ggplot2: Elegant graphics for data analysis. Springer-Verlag.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.45 MB