An Empirical Validation of the Regression Point Displacement Design Using Within-Study Comparison Logic: Emerging Possibilities and Cautions

Abstract

Background

The regression point displacement (RPD) design is a quasi-experiment (QE) that aims to control many threats to internal validity. Though it has existed for several decades, RPD has only recently begun to answer applied research questions in lieu of stronger QEs.

Objectives

Our primary objective was to implement within-study comparison (WSC) logic to create RPD replicates and to determine conditions under which RPD might provide estimates comparable to those found in validating experiments.

Research Design

We utilize three randomized controlled trials (two cluster-level, one individual-level), artificially decomposing or creating cluster structures, to create multiple RPDs. We compare results in each RPD treatment group to a fixed set of control groups to gauge the congruence of these repeated RPD realizations with results found in these three RCTs.

Results

RPD’s performance was uneven. Using multiple criteria, we found that RPDs successfully predicted the direction of the RCT’s intervention effect but inconsistently fell within the .10 SD threshold. A scant 13% of RPD results were statistically significant at either the .05 or .01 alpha-level. RPD results were within the 95% confidence interval of RCTs around half the time, and false negative rates were substantially higher than false positive rates.

Conclusions

RPD consistently underestimates treatment effects in validating RCTs. We analyze reasons for this insensitivity and offer practical suggestions to improve the chances RPD will correctly identify favorable results. We note that the synthetic, “decomposition of cluster RCTs,” WSC design represents a prototype for evaluating other QEs.

Keywords

regression point displacement validation within-study comparison cluster randomized controlled trial decomposition of cluster randomized controlled trials

Get full access to this article

View all access options for this article.

References

Amrhein

Greenland

McShane

(2019). Scientists rise up against statistical significance, Nature, 567, 305–307. https://doi.org/10.1038/d41586-019-00857-9

Bacon

(2004). The contributions of reliability and pretests to effective assessment. Practical Assessment, Research & Evaluation, 9, 1–8. Retrieved from https://pareonline.net/getvn.asp?v=9&n=3

Bothwell

L. E.

Podolsky

S. H.

(2016). The emergence of the randomized, controlled trail. New England Journal of Medicine, 375, 501–504. https://doi.org/10.1056/NEJp1604635

Campbell

D. T.

Stanley

J. C

(1963). Experimental and quasi-experimental designs for research. Chicago: Rand McNally.

Chaplin

D. D.

Cook

T. D.

Zurovac

Coopersmith

J. S.

Finucane

M. M.

Vollmer

L. N.

Morris

R. E.

, (2018). The internal and external validity of the regression discontinuity design: A meta-analysis of 15 within-study comparisons. Journal of Policy Analysis and Management, 37, 403–429. https://doi.org/10.1002/pam.22051

Cochran

W. G.

(1957). Analysis of covariance: Its nature and uses. Biometrics, 13(3), 261–281. https://doi.org/10.2307/2527916

Cook

T. D.

(2008). “Waiting for life to arrive”: A history of the regression-discontinuity design in psychology, statistics, and economics. Journal of Econometrics, 142, 636–654. https://doi.org/10.1016/j.jeconom.2007.05.002

Cook

T. D.

Campbell

D. T.

(1979). Quasi-experimentation: Design and analysis for field settings. Chicago, IL: Rand McNally.

Cook

T. D.

Shadish

W. R.

Wong

V. C.

(2008). Three conditions under which experiments and observational studies often produce comparable causal estimates: New findings from within-study comparisons. Journal of Policy Analysis and Management, 27, 724–750. https://doi.org/10.1002/pam.20375

10.

Cook

T. D.

Zhu

Klein

Starkey

Thomas

(2020). How much bias results if a quasi-experimental design compares local comparison groups, a pretest outcome measure, and other covariates?: A within study comparison pf preschool effects. Psychological Methods, 25, 626–646. https://doi.org/10.1037/met0000260

11.

Fretheim

Soumerai

S. B.

Zhang

Oxman

A. D.

Ross-Degnan

, (2013). Interrupted time series analysis yielded an effect estimate concordant with the cluster-randomized control trial result. Journal of Clinical Epidemiology, 66, 883–887. https://doi.org/10.1016/j.jclinepi.2013.03.016

12.

Gelman

Hill

(2006). Data analysis using regression and multilevel/hierarchical models. Cambridge University Press.

13.

Gleason

Resch

Berk

(2018). RD or not RD: Using experimental studies to assess the performance of the regression discontinuity approach. Evaluation Review, 42, 3–33. https://doi.org/10.1177/0193841X18787267.

14.

Kang

J. H.

(2014). Designing studies to evaluate an orientation program using mobile augmented reality for net generation students. In iConference 2014 Proceedings, 901–904. https://doi.org/10.9776/14305.

15.

Kowalski

Yeaton

W. H.

Kuhr

Holger

(2016). Helping hospitals improve patient centeredness: Assessing the impact of feedback following a best practices workshop. Evaluation & the Health Professions, 40(2), 180–202.

16.

Kruschke

J. K.

(2018). Rejecting or accepting parameter values in Bayesian estimation. Advances in Methods and Practices in Psychological Science, 1, 270–280. https://doi.org/10.1177/2515245918771304.

17.

Lalonde

(1986). Evaluating the econometric evaluations of training with experimental data. The American Economic Review, 76, 604–620. Retrieved from: http://links.jstor.org/sici?sici=0002-8282%2819860 ... O%3B2-P&origin=repec

18.

Linden

Trochim

W. M. K.

Adams

J. L.

(2006). Evaluating program effectiveness using the regression point displacement design. Evaluation & the Health Professions, 29, 407–423.

19.

Lipsey

(1990). Design sensitivity: Statistical power for experimental research. Newbury Park, CA: Sage.

20.

Little

T. D.

Gorrall

B. K.

Panko

Curtis

J. D.

(2017). Modern practices to improve human development research. Research in Human Development, 14, 338–349. https://doi.org/10.1080/15427609.2017.1370967

21.

Lundberg

Johnson

Stewart

B. M.

(2021). What is your estimand? Defining the target quantity connects statistical evidence to theory. American Sociological Review, 86, 532–565. https://doi.org/10.1177/00031224211004187

22.

Open Science Collaboration (2015). Estimating the reproducibility of psychological science. Science, 349, aac4716. https://doi.org/10.1126/science.aac4716

23.

Shadish

W. R.

(2011). Randomized controlled studies and alternative designs in outcome studies: Challenges and opportunities. Research on Social Work Practice, 21, 636–643. https://doi.org/10.1177/1049731511403324

24.

Shadish

W. R.

Cook

T. D.

Campbell

D. T.

(2002). Experimental and quasi-experimental design for generalized causal inference. Boston: Houghton Mifflin.

25.

Shadish

W. R.

Galindo

Wong

V. C.

Steiner

P. M.

Cook

T. D.

(2011). A randomized experiment comparing random and cutoff-based assignment. Psychological Methods, 16, 179–191. https://doi.org/10.1037/a0023345

26.

StClair

Cook

T. D.

Hallberg

(2014). Examining the internal validity and statistical precision of the comparative interrupted time series design by comparison with a randomized experiment. American Journal of Evaluation, 35, 311–327. https://doi.org/10.1177/1098214014527337

27.

Steiner

P. M.

Cook

T. D.

Shadish

W. R.

Clark

M. H.

(2010). The importance of covariate selection in controlling for selection bias in observational studies. Psychological Methods, 15, 250–267. https://doi.org/10.1037/a0018719

28.

Steiner

P. M.

Wong

V. C.

(2016). Assessing correspondence between experimental and non-experimental results in within-study comparisons. Retrieved from: http://curry.virginia.edu/uploads/resourceLibrary/46Analysis

29.

Steiner

P. M.

Wong

V. C.

(2018). Assessing correspondence between experimental and nonexperimental estimates in within-study comparisons. Evaluation Review, 42, 214–247. https://doi.org/10.1177/0193841X18773807

30.

Steiner

P. M.

Wong

V. C.

Anglin

(2019). A causal replication framework for designing and assessing replication efforts. Zeitschrift für Psychologie, 227, 280–292. https://doi.org/10.1027/2151-2604/a000385

31.

Sundt

Boppre

(2020) Did Oregon’s tough mandatory sentencing law “measure 11” improve public safety? New evidence about an old debate from a multiple-design, experimental strategy, Justice Quarterly, 1, 22. https://doi.org/10.1080/07418825.2020.1792964

32.

Sundt

Salisbury

E. J.

Harmon

M. G.

(2016). Is downsizing prisons dangerous? The effect of California’s realignment act on public safety. Criminology & Public Policy. 15, 315–341. https://doi.org/10.1111/1745-9133.12199.

33.

Thistlethwaite

D. L.

Campbell

D. T.

(1960). Regression-discontinuity analysis: An alternative to the ex post facto experiment. Journal of Educational Psychology, 51: 309–317. https://doi.org/10.1037/h0044319

34.

Trochim

W. M. K.

Campbell

D. T.

(1996). The regression point displacement design for evaluating community-based pilot programs and demonstration projects. Unpublished manuscript. Retrieved from http://www.socialresearchmethods.net/research/RPD.RPD.pdf.

35.

Wong

V. C.

Steiner

P. M.

(2018) Designs of empirical evaluations of nonexperimental methods in field settings. Evaluation Review, 42, 176–213. https://doi.org/10.117/0193841X18773807

36.

Wyman

P. A.

Henry

Knoblauch

Brown

C. H.

(2015). Designs for testing group-based interventions with limited numbers of social units: The dynamic wait-listed and regression point displacement designs. Prevention Science, 16, 956–966. https://doi.org/10.1007/s11121-014-0535-6

37.

Yeaton

W. H.

(2019). Experimental and quasi-experimental designs. In Brough

(Ed.), Advanced Research Methods for Applied Psychology: Design, Analysis and Reporting (pp. 107–123). New York: Routledge.

38.

Yeaton

W. H.

Moss

B. G.

(2020). A multiple-design, experimental strategy: Academic probation warning letter’s impact on student achievement. The Journal of Experimental Education, 123–144. https://doi.org/10.1080/00220973.2018.1469110

39.

Yeaton

W. H.

Sechrest

(1981). Critical dimensions in the choice and maintenance of successful treatments: strength, integrity, and effectiveness. Journal of Consulting and Clinical Psychology, 49(2), 157–167. https://doi.org/10.1037//0022-006x.49.2.156

40.

Lombard

Harrison

Kozica

Zoungas

Ranasinha

Teedee

(2016.) Preventing weight gain in women in rural communities: A cluster randomized trial. PLoS Medicine, 13, e1001941. https://doi.org/10.1371/journal.pmed.1001941

41.

Menant

J. C.

Migliacco

A. A.

Sturnieks

D. L.

Hicks

Ratanapongleka

Turner

Delbaere

Titov

Meinrath

McVeigh

J. C. T

Lord

S. R

(2018). Reducing the burden of dizziness in middle-aged and older people: A multifactorial, tailored, single-blind, randomized controlled trial. PLoS Medicine, 15, e1002620. https://doi.org/10.1371/journal.pmed.1002620

42.

Ogedegbe

Plange-Rhule

Gyamfi

Chaplin

Ntim

Apusiga

Iwelunmor

Awudzi

K. Y.

Quakyi

K. N.

Mogaverro

Khurshid

Tayo

Cooper

(2018). Health insurance coverage with or without a nurse-led task shifting strategy for hypertension control: a pragmatic cluster randomized trial in Ghana. PLoS Medicine, 15. e1002561, https://doi.org/10.1371/journal.pmed.1002561

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.34 MB