The regression point displacement (RPD) design is a quasi-experiment (QE) that aims to control many threats to internal validity. Though it has existed for several decades, RPD has only recently begun to answer applied research questions in lieu of stronger QEs.
Objectives
Our primary objective was to implement within-study comparison (WSC) logic to create RPD replicates and to determine conditions under which RPD might provide estimates comparable to those found in validating experiments.
Research Design
We utilize three randomized controlled trials (two cluster-level, one individual-level), artificially decomposing or creating cluster structures, to create multiple RPDs. We compare results in each RPD treatment group to a fixed set of control groups to gauge the congruence of these repeated RPD realizations with results found in these three RCTs.
Results
RPD’s performance was uneven. Using multiple criteria, we found that RPDs successfully predicted the direction of the RCT’s intervention effect but inconsistently fell within the .10 SD threshold. A scant 13% of RPD results were statistically significant at either the .05 or .01 alpha-level. RPD results were within the 95% confidence interval of RCTs around half the time, and false negative rates were substantially higher than false positive rates.
Conclusions
RPD consistently underestimates treatment effects in validating RCTs. We analyze reasons for this insensitivity and offer practical suggestions to improve the chances RPD will correctly identify favorable results. We note that the synthetic, “decomposition of cluster RCTs,” WSC design represents a prototype for evaluating other QEs.
BaconD. (2004). The contributions of reliability and pretests to effective assessment. Practical Assessment, Research & Evaluation, 9, 1–8. Retrieved fromhttps://pareonline.net/getvn.asp?v=9&n=3
3.
BothwellL. E.PodolskyS. H. (2016). The emergence of the randomized, controlled trail. New England Journal of Medicine, 375, 501–504. https://doi.org/10.1056/NEJp1604635
4.
CampbellD. T.StanleyJ. C (1963). Experimental and quasi-experimental designs for research. Chicago: Rand McNally.
5.
ChaplinD. D.CookT. D.ZurovacJ.CoopersmithJ. S.FinucaneM. M.VollmerL. N.MorrisR. E., (2018). The internal and external validity of the regression discontinuity design: A meta-analysis of 15 within-study comparisons. Journal of Policy Analysis and Management, 37, 403–429. https://doi.org/10.1002/pam.22051
6.
CochranW. G. (1957). Analysis of covariance: Its nature and uses. Biometrics, 13(3), 261–281. https://doi.org/10.2307/2527916
7.
CookT. D. (2008). “Waiting for life to arrive”: A history of the regression-discontinuity design in psychology, statistics, and economics. Journal of Econometrics, 142, 636–654. https://doi.org/10.1016/j.jeconom.2007.05.002
8.
CookT. D.CampbellD. T. (1979). Quasi-experimentation: Design and analysis for field settings. Chicago, IL: Rand McNally.
9.
CookT. D.ShadishW. R.WongV. C. (2008). Three conditions under which experiments and observational studies often produce comparable causal estimates: New findings from within-study comparisons. Journal of Policy Analysis and Management, 27, 724–750. https://doi.org/10.1002/pam.20375
10.
CookT. D.ZhuN.KleinA.StarkeyP.ThomasJ. (2020). How much bias results if a quasi-experimental design compares local comparison groups, a pretest outcome measure, and other covariates?: A within study comparison pf preschool effects. Psychological Methods, 25, 626–646. https://doi.org/10.1037/met0000260
11.
FretheimA.SoumeraiS. B.ZhangF.OxmanA. D.Ross-DegnanD., (2013). Interrupted time series analysis yielded an effect estimate concordant with the cluster-randomized control trial result. Journal of Clinical Epidemiology, 66, 883–887. https://doi.org/10.1016/j.jclinepi.2013.03.016
12.
GelmanA.HillJ. (2006). Data analysis using regression and multilevel/hierarchical models. Cambridge University Press.
13.
GleasonP.ReschA.BerkJ. (2018). RD or not RD: Using experimental studies to assess the performance of the regression discontinuity approach. Evaluation Review, 42, 3–33. https://doi.org/10.1177/0193841X18787267.
14.
KangJ. H. (2014). Designing studies to evaluate an orientation program using mobile augmented reality for net generation students. In iConference 2014 Proceedings, 901–904. https://doi.org/10.9776/14305.
15.
KowalskiC.YeatonW. H.KuhrK.HolgerP. (2016). Helping hospitals improve patient centeredness: Assessing the impact of feedback following a best practices workshop. Evaluation & the Health Professions, 40(2), 180–202.
16.
KruschkeJ. K. (2018). Rejecting or accepting parameter values in Bayesian estimation. Advances in Methods and Practices in Psychological Science, 1, 270–280. https://doi.org/10.1177/2515245918771304.
LindenA.TrochimW. M. K.AdamsJ. L. (2006). Evaluating program effectiveness using the regression point displacement design. Evaluation & the Health Professions, 29, 407–423.
19.
LipseyM. (1990). Design sensitivity: Statistical power for experimental research. Newbury Park, CA: Sage.
20.
LittleT. D.GorrallB. K.PankoP.CurtisJ. D. (2017). Modern practices to improve human development research. Research in Human Development, 14, 338–349. https://doi.org/10.1080/15427609.2017.1370967
21.
LundbergI.JohnsonR.StewartB. M. (2021). What is your estimand? Defining the target quantity connects statistical evidence to theory. American Sociological Review, 86, 532–565. https://doi.org/10.1177/00031224211004187
ShadishW. R. (2011). Randomized controlled studies and alternative designs in outcome studies: Challenges and opportunities. Research on Social Work Practice, 21, 636–643. https://doi.org/10.1177/1049731511403324
24.
ShadishW. R.CookT. D.CampbellD. T. (2002). Experimental and quasi-experimental design for generalized causal inference. Boston: Houghton Mifflin.
25.
ShadishW. R.GalindoR.WongV. C.SteinerP. M.CookT. D. (2011). A randomized experiment comparing random and cutoff-based assignment. Psychological Methods, 16, 179–191. https://doi.org/10.1037/a0023345
26.
StClairT.CookT. D.HallbergK. (2014). Examining the internal validity and statistical precision of the comparative interrupted time series design by comparison with a randomized experiment. American Journal of Evaluation, 35, 311–327. https://doi.org/10.1177/1098214014527337
27.
SteinerP. M.CookT. D.ShadishW. R.ClarkM. H. (2010). The importance of covariate selection in controlling for selection bias in observational studies. Psychological Methods, 15, 250–267. https://doi.org/10.1037/a0018719
SteinerP. M.WongV. C. (2018). Assessing correspondence between experimental and nonexperimental estimates in within-study comparisons. Evaluation Review, 42, 214–247. https://doi.org/10.1177/0193841X18773807
30.
SteinerP. M.WongV. C.AnglinK. (2019). A causal replication framework for designing and assessing replication efforts. Zeitschrift für Psychologie, 227, 280–292. https://doi.org/10.1027/2151-2604/a000385
31.
SundtJ.BoppreB. (2020) Did Oregon’s tough mandatory sentencing law “measure 11” improve public safety? New evidence about an old debate from a multiple-design, experimental strategy, Justice Quarterly, 1, 22. https://doi.org/10.1080/07418825.2020.1792964
32.
SundtJ.SalisburyE. J.HarmonM. G. (2016). Is downsizing prisons dangerous? The effect of California’s realignment act on public safety. Criminology & Public Policy. 15, 315–341. https://doi.org/10.1111/1745-9133.12199.
33.
ThistlethwaiteD. L.CampbellD. T. (1960). Regression-discontinuity analysis: An alternative to the ex post facto experiment. Journal of Educational Psychology, 51: 309–317. https://doi.org/10.1037/h0044319
34.
TrochimW. M. K.CampbellD. T. (1996). The regression point displacement design for evaluating community-based pilot programs and demonstration projects. Unpublished manuscript. Retrieved fromhttp://www.socialresearchmethods.net/research/RPD.RPD.pdf.
35.
WongV. C.SteinerP. M. (2018) Designs of empirical evaluations of nonexperimental methods in field settings. Evaluation Review, 42, 176–213. https://doi.org/10.117/0193841X18773807
36.
WymanP. A.HenryD.KnoblauchS.BrownC. H. (2015). Designs for testing group-based interventions with limited numbers of social units: The dynamic wait-listed and regression point displacement designs. Prevention Science, 16, 956–966. https://doi.org/10.1007/s11121-014-0535-6
37.
YeatonW. H. (2019). Experimental and quasi-experimental designs. In BroughP. (Ed.), Advanced Research Methods for Applied Psychology: Design, Analysis and Reporting (pp. 107–123). New York: Routledge.
38.
YeatonW. H.MossB. G. (2020). A multiple-design, experimental strategy: Academic probation warning letter’s impact on student achievement. The Journal of Experimental Education, 123–144. https://doi.org/10.1080/00220973.2018.1469110
39.
YeatonW. H.SechrestL. (1981). Critical dimensions in the choice and maintenance of successful treatments: strength, integrity, and effectiveness. Journal of Consulting and Clinical Psychology, 49(2), 157–167. https://doi.org/10.1037//0022-006x.49.2.156
40.
LombardC.HarrisonCKozicaS.ZoungasS.RanasinhaS.TeedeeH. (2016.) Preventing weight gain in women in rural communities: A cluster randomized trial. PLoS Medicine, 13, e1001941. https://doi.org/10.1371/journal.pmed.1001941
41.
MenantJ. C.MigliaccoA. A.SturnieksD. L.HicksC.LoJ.RatanaponglekaM.TurnerJ.DelbaereK.TitovN.MeinrathD.McVeighC.CloseJ. C. TLordS. R (2018). Reducing the burden of dizziness in middle-aged and older people: A multifactorial, tailored, single-blind, randomized controlled trial. PLoS Medicine, 15, e1002620. https://doi.org/10.1371/journal.pmed.1002620
42.
OgedegbeG.Plange-RhuleJ.GyamfiJ.ChaplinW.NtimM.ApusigaK.IwelunmorJ.AwudziK. Y.QuakyiK. N.MogaverroJ.KhurshidK.TayoB.CooperR. (2018). Health insurance coverage with or without a nurse-led task shifting strategy for hypertension control: a pragmatic cluster randomized trial in Ghana. PLoS Medicine, 15. e1002561, https://doi.org/10.1371/journal.pmed.1002561
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
