Sage Journals: Discover world-class research

Abstract

Determining which organizations are more effective in implementing an intervention program is essential for theoretically and empirically characterizing exemplary practice and for intervening to enhance the capacity of ineffective ones. However, sites differ in their local ecological conditions including client composition, availability of alternative programs, and broader community context. Applying the causal inference framework, this study proposes a formal mathematical definition of an organization's local relative effectiveness attributable solely to malleable organizational practice. Capitalizing on multisite randomized trials, the identification leverages observed control group outcomes that capture some of the confounding impacts of otherwise unmeasured contextual variation. We propose a two-step mixed-effects modeling procedure that adjusts for pre-existing between-site variation. Monte Carlo simulations demonstrate its superior performance compared to conventional methods. We apply the new strategy to an evaluation of Job Corps centers nationwide serving disadvantaged youths.

Keywords

causal inference heterogeneous treatment effect multisite trials organizational effectiveness program implementation

Get full access to this article

View all access options for this article.

References

Aldrich

H. E.

Pfeffer

(1976). Environments of organizations. Annual Review of Sociology, 2(1), 79–105. https://doi.org/10.1146/annurev.so.02.080176.000455

Allcott

(2015). Site selection bias in program evaluation. The Quarterly Journal of Economics, 130(3), 1117–1165. https://doi.org/10.1093/qje/qjv015

Anderson

K. H.

Burkhauser

R. V.

Raymond

J. E.

(1993). The effect of creaming on placement rates under the Job Training Partnership Act. ILR Review, 46(4), 613–624. https://doi.org/10.1177/001979399304600402

Angrist

J. D.

Imbens

G. W.

Rubin

D. B.

(1996). Identification of causal effects using instrumental variables. Journal of the American Statistical Association, 91(434), 444–455. https://doi.org/10.1080/01621459.1996.10476902

Armstrong

T. B.

Kolesár

Plagborg-Møller

(2022). Robust empirical Bayes confidence intervals. Econometrica, 90(6), 2567–2602. https://doi.org/10.3982/ECTA18597

Barnow

(1992). The effects of performance standards on state and local programs. In Manski

Garfinkel

(Eds.), Evaluating welfare and training programs (pp. 277–309). Cambridge, MA, USA: Harvard University Press.

Barnow

B. S.

Heinrich

(2010). One standard fits all? The pros and cons of performance standard adjustments. Public Administration Review, 70(1), 60–71. https://doi.org/10.1111/j.1540-6210.2009.02111.x

Bass

B. M.

Avolio

B. J.

(1994). Improving organizational effectiveness through transformational leadership. Thousand Oaks, CA, US: Sage.

Benjamin

L. M.

(2012). Nonprofit organizations and outcome measurement: From tracking program activities to focusing on frontline work. American Journal of Evaluation, 33(3), 431–447. https://doi.org/10.1177/1098214012440496

10.

Besharov

D. J.

Baehler

K. J.

Klerman

J. A.

(Eds.). (2016). Improving public services: International experiences in using evaluation tools to measure program performance. New York, NY, USA: Oxford University Press.

11.

Blalock

A. B.

Barnow

B. S.

(2001). Is the new obsession with performance management masking the truth about social programs? In Forsythe

D. W.

(Ed.), Quicker, better, cheaper? Managing performance in American government (pp. 485–517). Albany, NY, USA: The Rockefeller Institute Press.

12.

Blanco

Flores

C. A.

Flores-Lagunes

(2013). Bounds on average and quantile treatment effects of Job Corps training on wages. Journal of Human Resources, 48(3), 659–701. https://doi.org/10.1353/jhr.2013.0017

13.

Bloom

H. S.

(1984). Accounting for no-shows in experimental evaluation designs. Evaluation Review, 8(2), 225–246. https://doi.org/10.1177/0193841X8400800205

14.

Bloom

H. S.

(2005). Learning more from social experiments: Evolving analytic approaches. New York, NY, USA: Russell Sage Foundation.

15.

Bloom

H. S.

Hill

C. J.

Riccio

J. A.

(2003). Linking program implementation and effectiveness: Lessons from a pooled sample of welfare-to-work experiments. Journal of Policy Analysis and Management, 22(4), 551–575. https://doi.org/10.1002/pam.10154

16.

Bloom

H. S.

Raudenbush

Weiss

M. J.

Porter

(2017). Using multisite experiments to study cross-site variation in treatment effects: A hybrid approach with fixed intercepts and a random treatment coefficient. Journal of Research on Educational Effectiveness, 10(4), 817–842. https://doi.org/10.1080/19345747.2016.1264518

17.

Bryk

Schneider

(2002). Trust in schools: A core resource for improvement. New York, NY, USA: Russell Sage Foundation.

18.

Burghardt

Schochet

P. Z.

(2001). National Job Corps study: Impacts by center characteristics. Washington, DC, USA: US Department of Labor, Employment and Training Administration, Office of Policy and Research.

19.

Cameron

Whetten

(1983). Organizational effectiveness: A comparison of multiple models. New York, NY, USA: Academic.

20.

Century

Cassata

(2016). Implementation research: Finding common ground on what, how, why, where, and who. Review of Research in Education, 40(1), 169–215. https://doi.org/10.3102/0091732X16665332

21.

Chernozhukov

Chetverikov

Demirer

Duflo

Hansen

Newey

Robins

(2018). Double/debiased machine learning for treatment and structural parameters. The Econometrics Journal, 21(1), C1–C68. https://doi.org/10.1111/ectj.12097

22.

Chipman

H. A.

George

E. I.

McCulloch

R. E.

(2010). BART: Bayesian additive regression trees. The Annals of Applied Statistics, 4(1), 266–298. https://doi.org/10.1214/09-AOAS285

23.

Coburn

C. E.

(2001). Collective sensemaking about reading: How teachers mediate reading policy in their professional communities. Educational Evaluation and Policy Analysis, 23(2), 145–170. https://doi.org/10.3102/01623737023002145

24.

Cordray

D. S.

Pion

G. M.

(1993). Psychosocial rehabilitation assessment: A broader perspective. In Glueckauf

Bond

Sechrest

McDonel

(Eds.), Improving assessment in rehabilitation and health (pp. 215–240). Thousand Oaks, CA, USA: Sage.

25.

Cox

Kelcey

(2023). Statistical power for detecting moderation in partially nested designs. American Journal of Evaluation, 44(1), 133–152. https://doi.org/10.1177/1098214020977692

26.

Dahabreh

I. J.

Matthews

Steingrimsson

J. A.

Scharfstein

D. O.

Stuart

E. A.

(2024). Using trial and observational data to assess effectiveness: Trial emulation, transportability, benchmarking, and joint analysis. Epidemiologic Reviews, 46(1), 1–16. https://doi.org/10.1093/epirev/mxac011

27.

Dahabreh

I. J.

Robertson

S. E.

Steingrimsson

J. A.

Stuart

E. A.

Hernan

M. A.

(2020). Extending inferences from a randomized trial to a new target population. Statistics in Medicine, 39(14), 1999–2014. https://doi.org/10.1002/sim.8426

28.

Dahabreh

I. J.

Robertson

S. E.

Tchetgen

E. J.

Stuart

E. A.

Hernan

M. A.

(2019). Generalizing causal inferences from individuals in randomized trials to all trial-eligible individuals. Biometrics, 75(2), 685–694. https://doi.org/10.1111/biom.13009

29.

Degtiar

Rose

(2023). A review of generalizability and transportability. Annual Review of Statistics and Its Application, 10(1), 501–524. https://doi.org/10.1146/annurev-statistics-042522-103837

30.

Dorie

Harada

Carnegie

N. B.

Hill

(2016). A flexible, interpretable framework for assessing sensitivity to unmeasured confounding. Statistics in Medicine, 35(20), 3453–3470. https://doi.org/10.1002/sim.6973

31.

Efron

Hastie

(2016). Computer age statistical inference (Vol. 5). Cambridge, UK/New York, NY, USA: Cambridge University Press.

32.

Flores

C A

Mitnik

O A

(2013). Comparing treatments across labor markets: An assessment of nonexperimental multiple-treatment strategies. The Review of Economics and Statistics, 95(5), 1691–1707. https://doi.org/10.1162/REST_a_00373

33.

Flores-Lagunes

Gonzalez

Neumann

(2010). Learning but not earning? The impact of Job Corps training on Hispanic youth. Economic Inquiry, 48(3), 651–667. https://doi.org/10.1111/j.1465-7295.2009.00211.x

34.

Fortson

Rotz

Burkander

Mastri

Schochet

Rosenberg

McConnell

D’amico

(2017). Providing public workforce services to job seekers: 30-month impact findings on the WIA Adult and Dislocated Worker programs. Princeton, NJ, USA: Mathematica Policy Research.

35.

Frumento

Mealli

Pacini

Rubin

D. B.

(2012). Evaluating the effect of training on wages in the presence of noncompliance, nonemployment, and missing outcome data. Journal of the American Statistical Association, 107(498), 450–466. https://doi.org/10.1080/01621459.2011.643719

36.

Hahn

P. R.

Murray

J. S.

Carvalho

C. M.

(2020). Bayesian regression tree models for causal inference: Regularization, confounding, and heterogeneous effects (with discussion). Bayesian Analysis, 15(3), 965–1056. https://doi.org/10.1214/19-BA1195

37.

Hall

R. H.

(1980). Effectiveness theory and organizational effectiveness. The Journal of Applied Behavioral Science, 16(4), 536–545. https://doi.org/10.1177/002188638001600408

38.

Hastie

Tibshirani

R. J.

(2017). Extended comparisons of best subset selection, forward stepwise selection, and the lasso. arXiv preprint arXiv:1707.08692.

39.

Hastie

Tibshirani

Wainwright

(2015). Statistical learning with sparsity: The Lasso and generalizations. New York, NY, USA: Chapman & Hall/CRC.

40.

Heckman

Heinrich

Smith

(2002). The performance of performance standards. Journal of Human Resources, 37(4), 778–811. https://doi.org/10.2307/3069617

41.

Heckman

Heinrich

C. J.

Courty

Marschke

Smith

(2011). The performance of performance standards. Kalamazoo, MI, USA: W.E. Upjohn Institute for Employment Research.

42.

Heckman

Smith

Taber

(1998). Accounting for dropouts in evaluations of social programs. Review of Economics and Statistics, 80(1), 1–14. https://doi.org/10.1162/003465398557203

43.

Holland

P. W.

(1986). Statistics and causal inference. Journal of the American Statistical Association, 81(396), 945–960. https://doi.org/10.1080/01621459.1986.10478354

44.

Hong

(2004). Causal inference for multi-level observational data with application to kindergarten retention [Unpublished doctoral dissertation]. University of Michigan, School of Education.

45.

Hong

(2015). Causality in a social world: Moderation, mediation and spill-over. Chichester, West Sussex, UK: John Wiley & Sons.

46.

Hong

Raudenbush

S. W.

(2013). Heterogeneous agents, social interactions, and causal inference. In Morgan

S. L.

(Ed.), Handbook of causal analysis for social research (pp. 331–352). Dordrecht, Netherlands: Springer Netherlands.

47.

Hong

Yang

Qin

(2023). Posttreatment confounding in causal mediation studies: A cutting-edge problem and a novel solution via sensitivity analysis. Biometrics, 79(2), 1042–1056. https://doi.org/10.1111/biom.13705

48.

Hotz

V. J.

Imbens

G. W.

Mortimer

J. H.

(2005). Predicting the efficacy of future training programs using past experiences at other locations. Journal of Econometrics, 125(1–2), 241–270. https://doi.org/10.1016/j.jeconom.2004.04.009

49.

Imai

King

Nall

(2009). The essential role of pair matching in cluster-randomized experiments, with application to the Mexican universal health insurance evaluation. Statistical Science, 24(1), 29–53. https://doi.org/10.1214/08-STS274

50.

Johnson

Gritz

Jackson

Burghardt

Boussy

Leonard

Orians

(1999). National Job Corps Study: report on the process analysis. Research and Evaluation Report. Mathematica Policy Research.

51.

Lee

Nowell

(2015). A framework for assessing the performance of nonprofit organizations. American Journal of Evaluation, 36(3), 299–319. https://doi.org/10.1177/1098214014545828

52.

Leonard-Barton

(1988). Implementation as mutual adaptation of technology and organization. Research Policy, 17(1), 251–267. https://doi.org/10.1016/0048-7333(88)90006-6

53.

Levitt

March

J. G.

(1988). Organizational learning. Annual Review of Sociology, 14(1), 319–338. https://doi.org/10.1146/annurev.so.14.080188.001535

54.

Lipsey

M. W.

Cordray

D. S.

(2000). Evaluation methods for social intervention. Annual Review of Psychology, 51(1), 345–375. https://doi.org/10.1146/annurev.psych.51.1.345

55.

Ben-Michael

Feller

Miratrix

(2023). Is it who you are or where you are? Accounting for compositional differences in cross-site treatment effect variation. Journal of Educational and Behavioral Statistics, 48(4), 420–453. https://doi.org/10.3102/10769986231155427

56.

McConnell

S. M.

Glazerman

(2001). National Job Corps study: The benefits and costs of Job Corps. Washington, DC, USA: U.S. Department of Labor, Employment and Training Administration.

57.

McLaughlin

M. W.

(1976). Implementation as mutual adaptation: Change in classroom organization. Teachers College Record, 77(3), 1–9. https://doi.org/10.1177/016146817607700304

58.

Morris

P. A.

Connors

Friedman-Krauss

McCoy

D. C.

Weiland

Feller

Yoshikawa

(2018). New findings on impact variation from the Head Start Impact Study: Informing the scale-up of early childhood programs. AERA Open, 4(2), 1–16. https://doi.org/10.1177/2332858418769287

59.

Olsen

R. B.

Orr

L. L.

Bell

S. H.

Petraglia

Badillo-Goicoechea

Miyaoka

Stuart

E. A.

(2024). Using a multi-site RCT to predict impacts for a single site: Do better data and methods yield more accurate predictions? Journal of Research on Educational Effectiveness, 17(1), 184–210. https://doi.org/10.1080/19345747.2023.2180464

60.

Patton

M. Q.

(2001). Evaluation, knowledge management, best practices, and high quality lessons learned. American Journal of Evaluation, 22(3), 329–336. https://doi.org/10.1177/109821400102200307

61.

Qin

Deutsch

Hong

(2021). Unpacking complex mediation mechanisms and their heterogeneity between sites in a Job Corps evaluation. Journal of Policy Analysis and Management, 40(1), 158–190. https://doi.org/10.1002/pam.22268

62.

Qin

Hong

(2017). A weighting method for assessing between-site heterogeneity in causal mediation mechanism. Journal of Educational and Behavioral Statistics, 42(3), 308–340. https://doi.org/10.3102/1076998617694879

63.

Qin

Hong

Deutsch

Bein

(2019). Multisite causal mediation analysis in the presence of complex sample and survey designs and non-random nonresponse. Journal of the Royal Statistical Society Series A: Statistics in Society, 182(4), 1343–1370. https://doi.org/10.1111/rssa.12446

64.

Qin

Yang

(2022). Simulation-based sensitivity analysis for causal mediation studies. Psychological Methods, 27(6), 1000–1013.

65.

Raudenbush

S. W.

Willms

(1995). The estimation of school effects. Journal of Educational and Behavioral Statistics, 20(4), 307–335. https://doi.org/10.3102/10769986020004307

66.

Raudenbush

S. W.

(2004). What are value-added models estimating and what does this imply for statistical practice? Journal of Educational and Behavioral Statistics, 29(1), 121–129. https://doi.org/10.3102/10769986029001121

67.

Raudenbush

S. W.

Bloom

H. S.

(2015). Learning about and from a distribution of program impacts using multisite trials. American Journal of Evaluation, 36(4), 475–499. https://doi.org/10.1177/1098214015600515

68.

Raudenbush

S. W.

Bryk

A. S.

(2002). Hierarchical linear models: Applications and data analysis methods. Thousand, CA, USA: Sage.

69.

Reardon

S. F.

Raudenbush

S. W.

(2009). Assumptions of value-added models for estimating school effects. Education Finance and Policy, 4(4), 492–519. https://doi.org/10.1162/edfp.2009.4.4.492

70.

Reynolds

A. J.

(1998). Confirmatory program evaluation: A method for strengthening causal inference. American Journal of Evaluation, 19(2), 203–221. https://doi.org/10.1177/109821409801900204

71.

Rogers

(2016). Understanding and supporting equity: Implications of methodological and procedural choices in equity-focused evaluations. In Donaldson

S. I.

Picciotto

(Eds.), Evaluation for an equitable society (pp. 199–215). Charlotte, NC, USA: Information Age Publishing, Inc.

72.

Rubin

D. B.

(1978). Bayesian Inference for causal effects: The role of randomization. The Annals of Statistics, 6(1), 34–58. https://doi.org/10.1214/aos/1176344064

73.

Rubin

D. B.

(1986). Comment: Which ifs have causal answers. Journal of the American Statistical Association, 81(396), 961–962. https://doi.org/10.1080/01621459.1986.10478355

74.

Rubin

D. B.

Stuart

E. A.

Zanutto

E. L.

(2004). A potential outcomes view of value-added assessment in education. Journal of Educational and Behavioral Statistics, 29(1), 103–116. https://doi.org/10.3102/10769986029001103

75.

Russ-Eft

Preskill

Jordan

J. B.

(2024). Evaluation in organizations: A systematic approach to enhancing learning, performance, and change. New York, NY, USA: Basic Books.

76.

Schochet

P. Z.

(2021). Long-run labor market effects of the Job Corps program: Evidence from a nationally representative experiment. Journal of Policy Analysis and Management, 40(1), 128–157. https://doi.org/10.1002/pam.22233

77.

Schochet

P. Z.

Burghardt

McConnell

(2008). Does Job Corps work? Impact findings from the National Job Corps Study. American Economic Review, 98(5), 1864–1886. https://doi.org/10.1257/aer.98.5.1864

78.

Schochet

P. Z.

Burghardt

J. A.

(2008). Do Job Corps performance measures track program impacts? Journal of Policy Analysis and Management, 27(3), 556–576. https://doi.org/10.1002/pam.20356

79.

Schochet

P. Z.

Fortson

(2014). When do regression-adjusted performance measures track longer-term program impacts? A case study for Job Corps. Journal of Policy Analysis and Management, 33(2), 495–525. https://doi.org/10.1002/pam.21746

80.

Seltzer

M. H.

(1994). Studying variation in program success: A multilevel modeling approach. Evaluation Review, 18(3), 342–361. https://doi.org/10.1177/0193841X9401800304

81.

Sharma

Singh

R. K.

(2019). A unified model of organizational effectiveness. Journal of Organizational Effectiveness: People and Performance, 6(2), 114–128. https://doi.org/10.1108/JOEPP-10-2018-0084

82.

Shulman

(1986). Those who understand: Knowledge growth in teaching. Educational Researcher, 15(2), 4–14. https://doi.org/10.3102/0013189X015002004

83.

Shulman

(2011). Knowledge and teaching: Foundations of the new reform. Harvard Educational Review, 57(1), 1–23. https://doi.org/10.17763/haer.57.1.j463w79r56455411

84.

Spillane

J. P.

Reiser

B. J.

Reimer

(2002). Policy implementation and cognition: Reframing and refocusing implementation research. Review of Educational Research, 72(3), 387–431. https://doi.org/10.3102/00346543072003387

85.

Staiger

D. O.

Kane

T. J.

(2015). Making decisions with imprecise performance measures: The relationship between annual student achievement gains and a teacher's career value added. In Kane

T. J.

Keer

K. A.

Pianta

R. C.

(Eds.), Designing teacher evaluation systems: New guidance from the measures of effective teaching project (pp. 144–169). San Francisco, CA, USA: Jossey-Bass.

86.

Steers

R. M.

(1975). Problems in the measurement of organizational effectiveness. Administrative Science Quarterly, 20(4), 546–558. https://doi.org/10.2307/2392022

87.

Stuart

E. A.

Cole

S. R.

Bradshaw

C. P.

Leaf

P. J.

(2011). The use of propensity scores to assess the generalizability of results from randomized trials. Journal of the Royal Statistical Society Series A: Statistics in Society, 174(2), 369–386. https://doi.org/10.1111/j.1467-985X.2010.00673.x

88.

Thrush

(2018, August 26). $1.7 billion federal job training program is “failing the students.” The New York Times. https://www.nytimes.com/2018/08/26/us/politics/job-corps-training-program.html

89.

Tipton

(2014). How generalizable is your experiment? An index for comparing experimental samples and populations. Journal of Educational and Behavioral Statistics, 39(6), 478–501. https://doi.org/10.3102/1076998614558486

90.

Tipton

Hallberg

Hedges

L. V.

Chan

(2017). Implications of small samples for generalization: Adjustments and rules of thumb. Evaluation Review, 41(5), 472–505. https://doi.org/10.1177/0193841X16655665

91.

Tipton

Olsen

R. B.

(2018). A review of statistical methods for generalizing from evaluations of educational interventions. Educational Researcher, 47(8), 516–524. https://doi.org/10.3102/0013189X18781522

92.

Turner

E. L.

Gallis

J. A.

Prague

Murray

D. M.

(2017). Review of recent methodological developments in group-randomized trials: Part 1—Design. American Journal of Public Health, 107(6), 907–915. https://doi.org/10.2105/AJPH.2017.303706

93.

Walton

Harvill

E. L.

Peck

L. R.

(2019). Which program characteristics are linked to program impacts? Lessons from the HPOG 1.0 evaluation. OPRE report 2019-51. Washington, DC: Office of Planning, Research, and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services.

94.

Weick

K. E.

(1987). Organizational culture as a source of high reliability. California Management Review, 29(2), 112–127. https://doi.org/10.2307/41165243

95.

Weiss

M. J.

Bloom

H. S.

Brock

(2014). A conceptual framework for studying the sources of variation in program effects. Journal of Policy Analysis and Management, 33(3), 778–808. https://doi.org/10.1002/pam.21760

96.

Weiss

M. J.

Bloom

H. S.

Verbitsky-Savitz

Gupta

Vigil

A. E.

Cullinan

D. N.

(2017). How much do the effects of education and training programs vary across sites? Evidence from past multisite randomized trials. Journal of Research on Educational Effectiveness, 10(4), 843–876. https://doi.org/10.1080/19345747.2017.1300719

97.

Westine

C. D.

(2016). Finding efficiency in the design of large multisite evaluations: Estimating variances for science achievement studies. American Journal of Evaluation, 37(3), 311–325. https://doi.org/10.1177/1098214015624014

98.

Yukl

(2008). How leaders influence organizational effectiveness. The Leadership Quarterly, 19(6), 708–722. https://doi.org/10.1016/j.leaqua.2008.09.008

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

1.02 MB

Evaluating Organizational Effectiveness: A New Strategy to Leverage Multisite Randomized Trials for Valid Assessment

Abstract

Keywords

Get full access to this article

References

Supplementary Material