Using Ecometric Data to Explore Sources of Cross-Site Impact Variance in Multi-Site Trials

Abstract

In 2003, Bloom, Hill, and Riccio (BHR) published an influential paper introducing novel methods for explaining the variation in local impacts observed in multi-site randomized control trials of socio-economic interventions in terms of site-level mediators. This paper seeks to improve upon this previous work by using student-level data to measure site-level mediators and confounders. Development of asymptotic behavior backed up with simulations and an empirical example. Students and training providers. Two simulations and an empirical application to data from an evaluation of the Health Professions Opportunity Grants (HPOG) Program. This empirical analysis involved roughly 6600 participants across 37 local sites. We examine bias and mean square error of estimates of mediation coefficients as well as the true coverage of nominal 95-percent confidence intervals on the mediation coefficients. Simulations suggest that the new methods generally improve the quality of inferences even when there is no confounding. Applying this methodology to the HPOG study shows that program-average FTE months of study by month six was a significant mediator of both career progress and long-term degree/credential receipt. Evaluators can robustify their BHR-style analyses by the use of the methods proposed here.

Keywords

ecometrics classroom climate attenuation bias multilevel data RCTS evaluation

Get full access to this article

View all access options for this article.

References

Aguinis

Culpepper

S. A.

(2015). An expanded decision-making procedure for examining cross-level interaction effects with multilevel modeling. Organizational Research Methods, 18(2), 155–176. https://doi.org/10.1177/1094428114563618

Ammann

C. M.

Genton

M. G.

(2010). Technical Note: Correcting for signal attenuation from noisy proxy data in climate reconstructions. Climate of the Past, 6(2), 273–279. https://doi.org/10.5194/cp-6-273-2010

Anderson

G. J.

(1970). Effects of classroom social climate on individual learning. American Educational Research Journal, 7(2), 135–152. https://doi.org/10.3102/00028312007002135

Asparouhov

Muthen

(2007). Constructing Covariates in Multilevel Regression. Mplus Web Notes, (11). https://www.statmodel.com/download/webnotes/webnote11.pdf

Bell

Associates (2009). Evaluation brief: Measuring implementation fidelity. James Bell and Associates. http://www.acf.hhs.gov/sites/default/files/cb/measuring_implementation_fidelity.pdf

Bloom

H. S.

(2005). Learning more from social experiments: Evolving analytic approaches. Russell Sage Foundation.

Bloom

H. S.

Hill

C. J.

Riccio

J. A.

(2003). Linking program implementation and effectiveness: Lessons from a pooled sample of welfare-to-work experiments. Journal of Policy Analysis and Management, 22(4), 551–575. https://doi.org/10.1002/pam.10154

Braga

Paccagnella

Pellizzari

(2014). Evaluating students’ evaluations of professors. Economics of Education Review, 41, 71–88. https://doi.org/10.1016/j.econedurev.2014.04.002

Carrell

S. E.

West

J. E.

(2010). Does professor quality matter? Evidence from random assignment of students to professors. Journal of Political Economy, 118(3), 409–432. https://doi.org/10.1086/653808

10.

Cook

J. R.

Stefanski

L. A.

(1994). Simulation-extrapolation estimation in parametric measurement error models. Journal of the American Statistical Association, 89(428), 1314–1328. https://doi.org/10.1080/01621459.1994.10476871

11.

Davis

T. J.

(2015). Teacher beliefs and perceptions about preschool bullying. PhD Dissertation, University of Alabama.

12.

Efron

Stein

, & (1981). The jackknife estimate of variance. The Annals of Statistics, 9(3), 586–596. https://doi.org/10.1214/aos/1176345462

13.

Goldstein

(2011). Multilevel statistical models. John Wiley.

14.

Hedges

L. V.

Hedberg

E. C.

(2007). Intraclass correlation values for planning group-randomized trials in education. Educational Evaluation and Policy Analysis, 29(1), 60–87. https://doi.org/10.3102/0162373707299706

15.

Hox

J. J.

(2010). Multilevel analysis: Techniques and applications. Routledge.

16.

Johnson

A. D.

Markowitz

A. J.

Hill

C. J.

Phillips

D. A.

(2016). Variation in impacts of tulsa pre-K on cognitive development in kindergarten: The role of instructional support. Developmental Psychology, 52(12), 2145–2158. https://doi.org/10.1037/dev0000226

17.

Gao

X. S.

Xiong

Zhao

Zhang

Zhou

D. M.

(2010). Identification of differentially expressed genes related to radioresistance of human esophageal cancer cells. Chinese Journal of Cancer, 29(10), 882–888. https://doi.org/10.5732/cjc.010.10148

18.

Lipsky

(1980). Street-level bureaucracy: Dilemmas of the individual in public services. Russel Sage Foundation.

19.

Mackenbach

J. D.

Lakerveld

van Lenthe

F. J.

Kawachi

McKee

Rutter

Glonti

Compernolle

De Bourdeaudhuij

Feuillet

Oppert

J. M.

Nijpels

Brug

(2016). Neighbourhood social capital: Measurement issues and associations with health outcomes. Obesity Reviews: An Official Journal of the International Association for the Study of Obesity, 17(Suppl 1), 96–107. https://doi.org/10.1111/obr.12373

20.

Malone

(2015). My existence didn't make No difference to them: Perceptions of teacher expectations among african-American students and their Families. (Electronic Thesis or Dissertation). Retrieved from https://etd.ohiolink.edu/

21.

Marsh

H. W.

Lüdtke

Nagengast

Trautwein

Morin

A. J. S.

Abduljabbar

A. S.

Köller

(2012). Classroom climate and contextual effects: Conceptual and methodological issues in the evaluation of group-level effects. Educational Psychologist, 47(2), 106–124. https://doi.org/10.1080/00461520.2012.670488

22.

Martin

Inchley

Humphris

Currie

"Assessing the psychometric and ecometric properties of neighborhood scales using adolescent survey data from urban and rural Scotland population health metrics Moerbeek

Teerenstra

In Power Analysis of Trials with Multilevel Data. 2017, 15(1), 11. CRC Press.

23.

Morris

P. A.

Connors

Friedman-Krauss

McCoy

D. C.

Weiland

Feller

Page

Bloom

Yoshikawa

(2018). New findings on impact variation from the head start impact study: Informing the scale-up of early childhood programs. AERA Open, 4(2). https://doi.org/10.1177/2332858418769287

24.

Peck

L R.

Werner

Harvill

Litwok

Moulton

Fountain

A. R.

Locke

(2018). Health profession opportunity grants (HPOG 1.0) impact study interim report: Program Implementation and Short-term impacts. In OPRE report 2018-16a Office of planning, research, and evaluation, administration for children and Families, U.S. Department of Health and Human Services.

25.

Raudenbush

S. W.

Sampson

R. J.

(1999). 1. Ecometrics: Toward a science of assessing ecological settings, with application to the systematic social observation of neighborhoods. Sociological Methodology, 29(1), 1–41. https://journals.sagepub.com/doi/10.1111/0081-1750.00059

26.

Reardon

S. F.

Raudenbush

S. W.

(2013). Under what assumptions do site-by-treatment instruments identify average causal effects? Sociological Methods and Research, 42(2), 143–163. https://doi.org/10.1177/0049124113494575

27.

Reardon

S. F.

Unlu

Zhu

Bloom

H. S.

(2014). Bias and bias correction in multisite instrumental variables analysis of heterogeneous mediator effects. Journal of Educational and Behavioral Statistics, 39(1), 53–86. https://doi.org/10.3102/1076998613512525

28.

Robinson

G. K.

(1991). That BLUP is a good thing: The estimation of random effects. Statistical Science, 6(1), 15–32. https://doi.org/10.1214/ss/1177011926

29.

Rothstein

Von Wachter

(2017). Social experiments in the labor market. In Handbook Of Economic Field Experiments, (2, pp. 555–637). Elsevier.

30.

Schunck

(2016). Cluster size and aggregated level 2 variables in multilevel models: A cautionary note. Methods, Data, Analyses, 10(1), 97–108. https://doi.org/10.12758/mda.2016.005

31.

Smith

J. A.

Whalley

Wilcox

(2020), Are program participants good Evaluators? IZA Discussion Paper No. 13584, Available at SSRN:https://ssrn.com/abstract=3674305

32.

Stefanski

L. A.

Cook

J. R.

(1995). Simulation-extrapolation: The Measurement error jackknife. Journal of the American Statistical Association, 90(432), 1247–1256. https://doi.org/10.1080/01621459.1995.10476629

33.

Tukey

J. W.

(1958). Bias and confidence in not quite large samples (abstract). The Annals of Mathematical Statistics, 29(2), 614. https://doi.org/10.1214/aoms/1177706647

34.

Waksberg

Sperry

Judkins

Smith

(1993). The National Survey of Family Growth, Cycle IV, evaluation of linked design . Vital Health Stat 2, (117).

35.

Walton

Harvill

E. L.

Peck

Laura R.

(2019). Which program characteristics are linked to program impacts? Lessons from the HPOG 1.0 evaluation. OPRE report 2019- 51. Office of Planning, Research, and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services.

36.

Weiss

M. J.

Bloom

H. S.

Brock

(2014). A conceptual framework for studying the sources of variation in program effects. Journal of Policy Analysis and Management, 33(3), 778–808. https://doi.org/10.1002/pam.21760

37.

Weiss

M. J.

BloomVerbitsky-Savitz

H. S. N.

Gupta

Vigil

A. E.

Cullinan

D. N.

(2017). How much do the effects of education and training programs vary across sites? Evidence from past multisite randomized trials. Journal of Research on Educational Effectiveness, 10(4), 843–876. https://doi.org/10.1080/19345747.2017.1300719

38.

Wolter

K. M.

(2007). Introduction to variance estimation (2nd ed.). Springer.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.40 MB